OPINION PAPER

Large Search Model: Redefining Search Stackin the Era of LLMs
Liang Wang

Nan Yang

Xiaolong Huang

Linjun Yang

MicrosoftUSA

MicrosoftUSA

MicrosoftUSA

MicrosoftUSAwangliang@microsoft.com

nanya@microsoft.com

Rangan Majumder

Furu Wei

MicrosoftUSA

Microsoft
USA

Abstract
Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These
components are often optimized and deployed independently. In this paper, we introduce
a novel conceptual framework called large search model, which redefines the conventional
search stack by unifying search tasks with one large language model (LLM). All tasks areformulated as autoregressive text generation problems, allowing for the customization oftasks through the use of natural language prompts. This proposed framework capitalizes on
the strong language understanding and reasoning capabilities of LLMs, offering the potential
to enhance search result quality while simultaneously simplifying the existing cumbersome
search stack. To substantiate the feasibility of this framework, we present a series of proofof-concept experiments and discuss the potential challenges associated with implementing
this approach within real-world search systems.

1

Introduction

With the ever-increasing amount of information on the web, search engines have emerged as an
indispensable tool for finding and accessing information. Several decades of research efforts inthe fields of information retrieval (IR), machine learning, and computational advertising haveculminated in the continuous evolution and commercial success of search engines like Google andBing. Although current systems still provide the traditional ten blue links as the primary search
results, the Search Engine Result Page (SERP) often contains various other valuable information,
including direct answers [Potthast et al., 2021], featured snippets, knowledge panels, related
queries, and multimedia content, etc. Recently, the popularization of ChatGPT 1 has led to theemergence of conversational search interfaces, where LLMs generate responses to user queries withretrieval augmentation.
1

https://openai.com/blog/chatgpt

ACM SIGIR Forum

1

Vol. 57 No. 2 – December 2023

To support the aforementioned user information needs, modern search engines are built on astack of different components, including query intent understanding, embedding-based and lexicalmatching-based retrieval, multi-stage ranking, question answering, summarization, etc. Typically,
these components are designed and optimized independently, with the standard practice of finetuning pre-trained language models like BERT [Devlin et al., 2019] or T5 [Raffel et al., 2020]
on task-specific datasets. To handle hundreds of thousands of queries per second, knowledge
distillation is also widely used to reduce the computational cost of inference. However, this results
in a cumbersome search stack that is difficult to maintain. Furthermore, the quality of search
results for long-tailed and complex information needs are still far from satisfactory. A unified
modeling framework that offers flexible interfaces and improved generalization would be a moredesirable solution.
Large language models (LLMs) that leverage self-supervised pre-training and scaling laws [Kaplan et al., 2020] have emerged as general-purpose interfaces for natural language understanding
and generation. Cutting-edge LLMs like GPT-4 [OpenAI, 2023] and LLaMA [Touvron et al.,2023] are trained on web-scale corpora and demonstrate remarkable zero-shot and few-shot learning capabilities. In fact, they even surpass human performance on a variety of professional exams[OpenAI, 2023]. These desirable features position LLMs as a promising option for the unified
modeling of search tasks.
In this paper, we propose a conceptual framework denoted as large search model that reimagines
the conventional search stack from an LLM perspective. The large search model is an LLM thatis customized to the search domain. All IR tasks, except the first-stage retrieval, are formulated
as text generation problems and are handled by a single large search model. Given a user queryand potentially thousands of retrieved documents, the large search model generates the various
elements that constitute the SERP, including the ranked document list, document snippets, direct
answers, etc. Natural language prompts serve as the interface to customize the behavior of themodel. Different tasks are specified by different prompt templates, and the adoption of LLM also
allows for performing new tasks that are not explicitly trained. Additionally, ongoing researchon multi-modal LLMs [Alayrac et al., 2022; Huang et al., 2023] also allows the modeling of fulldocument contents (text, images, videos, layout, etc.), rather than just the textual portion of the
documents.
To make our envisioned large search model ready for production systems, several new challenges
must be addressed. For instance, the inference cost of LLMs remains prohibitively high for realtime applications due to the autoregressive nature of text generation. Additionally, efficient longcontext modeling without compromising quality is still an open problem. Finally, ensuring thatthe generated content conforms to responsible AI principles is crucial for the deployment of thisframework. Many of these challenges are not unique to the search domain, but are shared byother applications and have received extensive attention from the research community.
To empirically validate our approach, we instantiate a simplified version of the large searchmodel with the open-source LLaMA model [Touvron et al., 2023], and conduct some preliminary
experiments on joint listwise ranking and answer generation tasks. The results show that our
trained model is capable of achieving competitive performance compared to strong baselines. To
fully unleash the potential of large search models, we call for further research to establish abenchmark setting and develop new methods to tackle the challenges from the aspects of modelarchitecture, training, inference, etc.
ACM SIGIR Forum2

Vol. 57 No. 2 – December 2023

2

Related Work

2.1

Neural Information Retrieval

The field of information retrieval has undergone a paradigm shift in recent years, transitioning fromtraditional lexical term-based models to neural models. Neural information retrieval models [Mitra
et al., 2018] aim to overcome the limitations of traditional models, such as the vocabulary mismatch
problem and the lack of deep semantic understanding. These models have demonstrated promising
results in various applications, including document retrieval [Xiong et al., 2021], document reranking [Nogueira et al., 2019a], query generation [Nogueira et al., 2019b], and question answering[Karpukhin et al., 2020], etc. Pre-trained Transformer-based language models [Devlin et al.,2019; Reimers and Gurevych, 2019] are widely used as the backbone of neural IR models, andhave been shown to be effective on several benchmark datasets [Nguyen et al., 2016; Kwiatkowski
et al., 2019]. Despite their success, Thakur et al. [2021] pointed out that neural IR models still
underperformed the BM25 algorithm in out-of-domain scenarios. Studies by Wang et al. [2022]show that large-scale contrastive pre-training can help to improve the generalization of neuralmodels. Combining the expressive power of neural models with the lexical matching ability of termbased models [Lin, 2021; Gao et al., 2021] can further improve the robustness and interpretabilityof search systems.

2.2

Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) endows language models with the capability to retrieve
relevant information from external knowledge sources [Ram et al., 2023; Lewis et al., 2020; Shiet al., 2023], and thus can improve the factuality and informativeness of the generated texts. RAG
also offers a natural approach to cite information sources [Nakano et al., 2021; Liu et al., 2023b],
which enhances the verifiability of the information presented. The retrieved information can beincorporated into the generation process in different ways, such as input concatenation [Ramet al., 2023], attention-based fusion [Borgeaud et al., 2022], or output probability interpolation[Khandelwal et al., 2020]. Furthermore, Lewis et al. [2020]; Zhong et al. [2022] proposed to jointly
train the retrieval and generation components to promote their co-adaptation. Nevertheless, it isstill unclear what is the optimal way to train RAG models, and the degree to which the retrieved
information is utilized in the generation process requires further investigation.

2.3

Large Language Models

Large language models (LLMs), particularly decoder-only models such as LLaMA [Touvron et al.,2023] and GPT-4 [OpenAI, 2023], have shown great potential as general-purpose interfaces for natural language understanding and generation. LLMs exhibit several attractive emergent behaviors,
including few-shot in-context learning, instruction following [Ouyang et al., 2022], and chainof-thought reasoning [Wei et al., 2022] etc. In the context of information retrieval, LLMs havebeen utilized for pseudo-query generation [Dai et al., 2022], pseudo-document generation [Gaoet al., 2022; Wang et al., 2023b], long-form question answering [Nakano et al., 2021], and ranking
through prompting [Sun et al., 2023]. While existing work mainly focuses on the application of

ACM SIGIR Forum

3

Vol. 57 No. 2 – December 2023

LLMs to specific search tasks, we propose a conceptual framework to unify various search tasks
with one LLM.

3

Large Search Model

10 documents

Existing Search Stack

Ranking

Query Understanding
Models~100 documents

Snippets

Answers

Relevance Model

Snippet Model

Pre-ranking

Ranking list

Large Search Model(w/ (M)LLMs)

Answer Model~1000 documents…

First-stage Retrieval
Query & Prompts

Billions of documents

Figure 1. Comparison of the conventional search stack and our proposed large search model. The
conventional search stack comprises a cascading retrieval and ranking pipeline, along with many othercomponents to generate the Search Engine Result Page (SERP). In contrast, our proposed framework
employs a unified modeling approach, where prompts are utilized to customize the large search model fordiverse search tasks. It is worth mentioning that the figure presented herein is for illustrative purposes
only and does not correspond to any specific implementation of modern search engines. MLLM stands
for Multi-modal Large Language Models.

3.1

Unified Modeling of Information Retrieval Tasks
Fine-tuned Enc-Dec / Enc
Prompting GPT-4
Large Search Model (ours)# Models
Many1
1

Fine-tuning✔
✗✔

Latency
Cost
Low
Low
High
High
Medium Medium

Task Generalization✗✔
✔

Table 1. Comparison between three paradigms from different dimensions. “Fine-tuned Enc-Dec /Enc” refers to fine-tuning small encoder-decoder models or encoder-only models, which is the prevalentapproach at present. Our proposed framework can be fine-tuned on IR tasks, necessitating a smaller
model size to achieve comparable performance to GPT-4. This leads to a reduction in inference latency
and cost. However, we acknowledge that this argument requires further empirical validation.

We define the concept of large search model as a customized LLM (which may be multi-modal)that can perform various IR tasks robustly through natural language prompting, as depicted in
ACM SIGIR Forum

4

Vol. 57 No. 2 – December 2023

Figure 1. The technical implementation is not confined to any particular architecture or training
objective, and is not the central theme of this paper.
Compared to the current mainstream production systems that fine-tune separate small-size
models such as T5 or BERT for each task, the primary advantage of large search model lies in
its ability to enhance task performance through a unified modeling approach. This anticipatedimprovement in generalization capability comes from the increased model capacity of LLMs andthe possibility of leveraging the knowledge from other related tasks. Of course, this comes atthe expense of increased latency and inference costs, which is a major challenge for large-scale
deployment of LLMs. In contrast to prompting state-of-the-art proprietary models like GPT-4,
our proposed framework offers the flexibility to customize the model to desired search scenarios
by fine-tuning on domain-specific data, which are often abundant in commercial search engines.
A comprehensive comparison is presented in Table 1.
Modern search systems rely on collaborative efforts from many components to generate theresult page. Large search model can potentially replace many of these components, and may evenmake some of them obsolete. Here we provide a non-exhaustive list of search tasks in different
phases of search systems that can benefit from our method:• Online Serving Responding to real-time user queries requires ranking over first-stage
retrieval results, answer generation, snippet generation and query suggestion etc.
• Data Augmentation Model-based query generation and relevance labeling can be used
to augment the training data for ranking models.
• Indexing It is an offline procedure to build a queryable document corpus, which requires
content extraction, term weighting, and document expansion etc.
• Human Evaluation Many queries are long-tailed or ambiguous, thus are difficult toevaluate for human raters. Automatic query intent generation can help lower the cognitive
burden of human raters and improve evaluation quality and efficiency.
One notable exception is the first-stage retrieval component, which is currently implemented
with embedding-based or lexical-based retrieval methods in our framework. In contrast, Metzler
et al. [2021]; Tay et al. [2022] propose a paradigm where the retrieval corpus is implicitly encodedin the model parameters, thereby eliminating the need for a separate index. However, we argue
that this paradigm is still in its infancy and has serious limitations to be addressed, such as how
to scale to large corpus [Pradeep et al., 2023] and how to dynamically update the corpus [Tayet al., 2022].

3.2

Customization through Prompting

By formulating various IR tasks as text-to-text generation problems, we can build a unified large
search model and specify which task to perform through natural language prompting. This idea
was initially popularized by T5 [Raffel et al., 2020] and subsequently developed by decoder-only
LLMs such as GPT-3. LLMs can be directed to generate text in two primary ways:• In-context Learning The task is implicitly specified through a set of in-context examples,with each example comprising an input-output text pair. Language models need to learn
the pattern of the input-output pairs and generalize to new unseen examples. This approachrequires access to some labeled data and typically results in lengthy prompts.
ACM SIGIR Forum

5

Vol. 57 No. 2 – December 2023

• Instruction The task is specified by natural language instructions without any labeled data.
This paradigm is more flexible and user-friendly. LLMs after instruction tuning [Ouyang
et al., 2022] can often generalize to new instructions that are not seen during training.
In Table 2, we give some examples to demonstrate how to instantiate different tasks within
one model by customizing the prompt. Different from the traditional multitask learning approach,this paradigm does not add any task-specific parameters and enables zero-shot generalization to
new tasks during inference.

3.3

Long Context Modeling

One of the shortcomings of current LLMs is that they can only process a limited amount ofcontext at a time. During training, the input texts are typically split into chunks of fixed length
to mitigate the quadratic computational complexity of self-attention. Extending the effective
context length requires either continual fine-tuning on longer texts or designing special positioninterpolation strategies [Chen et al., 2023]. Another concern over inference with long context is
the increased memory consumption, due to the need to cache key-value states in autoregressive
decoding. Techniques like multi-query attention [Shazeer, 2019] and grouped-query attention[Ainslie et al., 2023] are commonly employed to reduce the inference memory footprint at the costof model quality. Although substantial progress has been made in extending context length, fromthe 512 tokens for BERT to 4k for LLaMA-2, and 32k for GPT-4 [OpenAI, 2023], they are still
often inadequate for real-world application scenarios. Furthermore, recent research [Liu et al.,2023a] has raised doubts about whether LLMs can effectively utilize long contexts as they claim.
Long context modeling is an essential capability requirement for our envisioned large searchmodel. Here we list several search-related tasks that can greatly benefit from such capability:
Long Document Retrieval and Ranking A significant portion of web documents are longform texts, such as news articles, legal documents, scientific papers, and code repositories, among
others. However, most of the existing ranking and retrieval models only support short text inputs.
For example, both the Sentence-BERT [Reimers and Gurevych, 2019] and E5 [Wang et al., 2022]
family of models only support inputs of up to 512 tokens. Although some heuristic methods, suchas FirstP and MaxP [Xiong et al., 2021], are proposed to bypass the difficulty of long document
modeling, their limitations are also obvious.
Retrieval-Augmented Generation (RAG) LLMs are known to hallucinate facts especiallywhen the generation requires long-tailed knowledge, and they are not aware of the latest eventsor information that are simply not publicly available. Retrieval augmentation is a widely adopted
technique to mitigate this problem. However, when generation requires conditioning on many
retrieved documents, the context length can easily exceed the maximum input length of LLMs,
even if each document can individually fit within the length limit. Take the task of open-domainquestion answering as an example, Izacard and Grave [2021] found that aggregating informationfrom 100 passages achieves the best performance. If concatenating all passages into a single input,
the resulting length can be well beyond the maximum context length of many LLMs.

ACM SIGIR Forum

6

Vol. 57 No. 2 – December 2023

Task

ExampleTask

ExampleTask

Document Ranking
Query: what is wifi vs Bluetooth### Documents[1] Takeaway: Bluetooth and Wi-Fi: Find out what separates these two wireless technologies.
Source: Flickr/Dana Spiegel. Bluetooth and Wi-Fi are both methods that provide wirelesscommunication, but the difference between the two mainly stems from what they are. . .
. . . (truncated)### End of documents
Rank the documents according to their relevance to the query.
A: 2>4>1>3>6 . . .
Question Answering
Q: what is hearing loss### Documents[1] Hearing loss, also known as hard of hearing, anacusis or hearing impairment, is a partial or
total inability to hear. It may occur in one or both ears. . .
[2] Hearing loss, or hearing impairment, is a partial or total loss of the ability to hear and canaffect one or both ears. . .
. . . (truncated)### End of documents
Write an answer to the query based on the given documents.
A: Hearing loss, also known as hearing impairment, is a partial or total inability tohear in one or both ears. It can be caused by factors such as noise, genetics, aging,
infections, or trauma. . . (truncated)
Snippet Generation
Q: yoga therapy benefits
D: Mindful Yoga for Trauma Training for Yoga Teachers — 15-Hour Program For many of us,
the practices of yoga have changed our lives. That statement may seem dramatic. . . (truncated)
Generate a short snippet based on the given document to answer the given query.

ExampleTask
ExampleTask

A: Yoga therapy benefits individuals coping with trauma-related difficulties, particularly veterans with Post Traumatic Stress. Mindful Yoga Therapy comprises fivepractices that provide symptom relief and support daily life skills. Although not acure, yoga aids the healing journey and stimulates the overall healing process.
Query Intent Generation
If a user searches for “what county is aurora mo in”, what is this user’s intent?
The user’s intent is to find out which county the city of Aurora, Missouri is locatedin. They are looking for the name of the county that encompasses the city.
Query Suggestion
A user searches for the query “what glass made of”, what other related queries that might be ofinterest to this user?

ExampleHow is glass made? / Different types of glass and their uses / The history of glassmaking / The science behind glass transparency

Table 2. Instantiation of different information retrieval tasks through prompting. The texts in bold
are the expected outputs of the corresponding tasks. This list is not exhaustive and only serves as ademonstration of the flexibility of our proposed framework. Long texts are truncated for brevity.

ACM SIGIR Forum

7

Vol. 57 No. 2 – December 2023

Conversational Search Conversational interfaces, popularized by ChatGPT, have been integrated into several mainstream search engines, such as New Bing 2 and Google Bard 3 . They
provide a more natural way for users to interact with search engines through multi-turn conversations. As the conversation progresses, the conversation history can grow to a considerable length.
Ideally, a conversational search system should be able to utilize the entire conversation history
to model the user’s intent and generate the most relevant responses. This requires LLMs to becapable of processing long contexts.

3.4

Multi-modal Large Search Model

Multi-modal contents other than plain texts are ubiquitous on the web, including images, videos,
audio, and other rich media formats. Incorporating such information into the search model cansignificantly improve the quality of search results and enable new search experiences. Users can
submit queries in mixed modalities, and the search engine will render the results in the mostappropriate format. Existing commercial systems have some but limited support for multi-modal
search in the form of independent modules, such as stand-alone image search, video search, andmusic search tabs.
Here we argue that the development of multi-modal foundation models [Radford et al., 2021;
Alayrac et al., 2022; Huang et al., 2023] could bring tremendous opportunities to the next generation of search engines. Large multi-modal foundation models can provide a deep understandingof web content, regardless of the modalities they exist in, contrasting to the current systems that
almost exclusively rely on texts. They can also be used to synthesize multi-modal contents that
better serve the user’s information needs and provide a more immersive experience. Products like
Bing Image Creator 4 powered by DALL-E 2 [Ramesh et al., 2022] are already taking the firststep in this direction. Developing larger and more robust multi-modal foundation models is afast-evolving research area, and we expect that they will unlock numerous possibilities for web
content understanding and generation.

3.5

Practical Considerations for Deployment

Deploying LLMs in a real-world search system is a challenging task that requires careful planningand optimization. They need to be scaled up to handle tens of thousands of queries per second
from users around the world under strict latency constraints while providing more accurate anduseful information compared to the existing search stack. Some of the practical considerations forour envisioned system are as follows:
Inference Efficiency LLMs typically have billions of parameters and require huge amountsof computational and memory resources to run. Besides relying on the development of more
powerful hardware, common techniques that can be used to improve the efficiency include modelcompression, quantization, pruning, and kernel fusion [Han et al., 2015; Dao et al., 2022]. Sparse2

https://www.bing.com/new
https://bard.google.com/
4https://www.bing.com/create
3

ACM SIGIR Forum

8

Vol. 57 No. 2 – December 2023

Mixture-of-Experts (MoE) architecture [Fedus et al., 2022] is also a promising direction to increase the model capacity while keeping the inference cost manageable. However, most open-source
LLMs still follow the dense Transformer architecture. Another line of research seeks to speed up
the autoregressive decoding process by attempting to generate multiple tokens in each inference
step. Speculative decoding [Leviathan et al., 2023] adopts a small language model to generate a few candidate tokens, which are then verified with the target LLM, while the inference withreference [Yang et al., 2023] method guesses the future tokens based on the available context. Additionally, caching mechanisms can be utilized to avoid online decoding for frequently seen queries.
Hallucination LLMs may hallucinate texts that are incorrect, nonsensical, or inconsistent withthe given information. For example, LLMs may generate a factually inaccurate statement abouta historical event, or a fictional entity that does not exist in the real world. Hallucinations canhave serious consequences for users who rely on LLMs for accurate information, education, ordecision making. Studies [Nakano et al., 2021; OpenAI, 2023] suggest that hallucinations can be
reduced by retrieval augmentation and further scaling up of LLMs, but it remains an unsolved
open problem.
Alignment Language model alignment [Kenton et al., 2021; Ouyang et al., 2022] has multifaceted meanings. Here, we mainly focus on the alignment between LLMs and human values.
As the training data of LLMs often contains undesirable content, LLMs without alignment cangenerate outputs that are offensive, biased, or even harmful to users. Even for models after alignment, carefully designed adversarial prompts can still mislead the model to generate inappropriate
outputs. To deploy LLMs in a useful and responsible way, comprehensive measures must be taken
as safeguards, including data filtering, content moderation, and red-team testing [Ganguli et al.,2022], etc.

4

Proof-of-Concept Experiments

In this section, we present some proof-of-concept experiments to showcase the potential of our
proposed framework. Specifically, we fine-tune the LLaMA-7B model [Touvron et al., 2023] on
the MS MARCO passage ranking dataset [Nguyen et al., 2016] for two tasks: listwise ranking
and retrieval-augmented answer generation. Both tasks are framed as text generation problems,
and we only compute the cross entropy loss on the target tokens. The prompt templates aretaken from Table 2. Since the original dataset does not provide annotations for listwise ranking,we use the re-ranker score from Wang et al. [2023a] as ranking labels while always placing thehuman-annotated positive passage in first place during training. For the answer generation task,
we collect the outputs for 40k queries from gpt-35-turbo 5 as the ground truth answers. The input
passages are the top-100 retrieval results from an off-the-shelf dense retriever E5large-v2 [Wanget al., 2022]. We utilize the linear positional interpolation method [Chen et al., 2023] and skip
encodings [Zhu et al., 2023] to extend the context length from 2k to 16k. The model is trainedfor one epoch with a batch size of 128 and a learning rate of 10−5 .
5https://oai.azure.com/portal

ACM SIGIR Forum

9

Vol. 57 No. 2 – December 2023

BM25
ANCE [Xiong et al., 2021]
E5large-v2 [Wang et al., 2022]
Ours (Listwise rank + LLaMA7b )MS MARCO
18.4
33.0
38.4
41.7

TREC DL 19
51.2
64.5
70.9
72.9

TREC DL 20
47.7
64.6
72.1
74.0

Table 3. Listwise ranking results on the MS MARCO passage ranking and TREC Deep Learning tracks.
Our model takes the top-100 passages retrieved by E5large-v2 as input and outputs top-5 passages withgreedy decoding. Consistent with previous work, we report MRR@10 for MS MARCO and NDCG@10
for TREC DL 19/20.
Query

Retrieved contexts

Model generation

GPT-4 Score (1-10)

rsa definition key[1] RSA is a cryptosystem for public-key encryption, and is widely used for securing sensitive
data, particularly when being sent over an insecure network such as the Internet.
[2] RSA algorithm RSA (algorithm) RSA is an algorithm used by modern computers to encryptand decrypt messages. It is an asymmetric cryptographic algorithm. Asymmetric means that
there are two different keys. This is also called public key cryptography, because one of themcan be given to everyone. The other key must be kept private.
[3] RSA is a cryptosystem for public-key encryption, and is widely used for securing sensitive
data, particularly when being sent over an insecure network such as the Internet. Download thisguide to become a Web App Security superhero.
. . . (97 more, truncated)
RSA is a public-key cryptosystem that uses two different keys, one public and one private. The
public key is used for encryption, while the private key is used for decryption. The asymmetryin RSA is based on the difficulty of factoring the product of two large prime numbers, which isthe factoring problem. The public key is used to encrypt messages, while the private key is used
to decrypt them. The RSA algorithm is based on the difficulty in factoring very large numbers,
and deducing an RSA key takes a huge amount of time and processing power. RSA is widelyused for securing sensitive data, particularly when being sent over an insecure network such as
the Internet.
9

Table 4. An example for answer generation. Our model generates an answer for the given query “rsadefinition key” conditioned on the top-100 retrieved contexts.

Table 3 indicates that our model outperforms BM25 sparse retrieval and multiple strong dense
retrievers, thus showcasing its potential for listwise ranking. For answer generation, we assess thequality of the generated answers with GPT-4 on a scale of 1 to 10, where higher scores denotebetter quality. On the TREC DL 2019 queries, our model achieves an average score of 8.4, while
its teacher model gpt-35-turbo gets 8.9. Upon manual inspection, we find that most generated
answers are fluent and relevant to the query, but some of them are not as comprehensive as theground truth answers and could contain repetitions. Table 4 presents an example of the generated
answers. Further experiments on additional tasks and datasets will be left for future endeavors.

5

Conclusion

This paper introduces the large search model framework to redefine the technique stack of searchsystems in the era of LLMs. We argue that the unique characteristics of LLMs permit the adoptionof a unified modeling approach for various IR tasks and offer improved generalization ability,
ACM SIGIR Forum

10

Vol. 57 No. 2 – December 2023

instead of fine-tuning and deploying numerous task-specific small encoder-decoder or encoderonly models. Along with the great potential of this framework, we also discuss several emerging
challenges that necessitate further research, such as high inference cost, long context modeling,
and the potential risks of misalignment, etc. To demonstrate the feasibility of our framework,
proof-of-concept experiments are conducted, although a larger-scale evaluation is required for amore comprehensive assessment.
While modern search engines have been instrumental in democratizing access to information,building a robust search system demands an enormous amount of engineering effort spanning multiple components, and the search results are still not satisfactory in many cases. We believe thatthe ongoing development of LLMs will bring a new wave of innovation to the field of information
retrieval, and we hope that our work can inspire further research in this direction.

Acknowledgments
We would like to thank many members from the Microsoft Bing team for their valuable feedback
and suggestions. We also thank the SIGIR Forum for providing the platform to share opinions
and ideas.

References
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit
Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. ArXiv preprint, abs/2305.13245, 2023. URL https://arxiv.org/abs/2305.13245.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, KarelLenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language
model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–
23736, 2022.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diegode Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren
Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol
Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International
Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA,volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR, 2022. URL
https://proceedings.mlr.press/v162/borgeaud22a.html.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context windowof large language models via positional interpolation. ArXiv preprint, abs/2306.15595, 2023.
URL https://arxiv.org/abs/2306.15595.

ACM SIGIR Forum

11

Vol. 57 No. 2 – December 2023

Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu,
Keith Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. In
The Eleventh International Conference on Learning Representations, 2022.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast andmemory-efficient exact attention with io-awareness. Advances in Neural Information Processing
Systems, 35:16344–16359, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis,
Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL
https://aclanthology.org/N19-1423.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232–5270, 2022.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben
Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to
reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv preprint, abs/2209.07858,
2022. URL https://arxiv.org/abs/2209.07858.
Luyu Gao, Zhuyun Dai, and Jamie Callan. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 3030–3042, Online, 2021. Association for Computational Linguistics. doi:
10.18653/v1/2021.naacl-main.241. URL https://aclanthology.org/2021.naacl-main.241.
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without
relevance labels. ArXiv preprint, abs/2212.10496, 2022. URL https://arxiv.org/abs/2212.10496.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks
with pruning, trained quantization and huffman coding. ArXiv preprint, abs/1510.00149, 2015.
URL https://arxiv.org/abs/1510.00149.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao
Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligningperception with language models. ArXiv preprint, abs/2302.14045, 2023. URL https://arxi
v.org/abs/2302.14045.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open
domain question answering. In Proceedings of the 16th Conference of the European Chapter ofthe Association for Computational Linguistics: Main Volume, pages 874–880, Online, 2021.
Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.74. URL https://aclanthology.org/2021.eacl-main.74.
ACM SIGIR Forum

12

Vol. 57 No. 2 – December 2023

Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child,
Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models.
ArXiv preprint, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi
Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP), pages 6769–6781, Online, 2020. Association for Computational Linguistics. doi:
10.18653/v1/2020.emnlp-main.550. URL https://aclanthology.org/2020.emnlp-main.55
0.
Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey
Irving. Alignment of language agents. ArXiv preprint, abs/2103.14659, 2021. URL https://arxiv.org/abs/2103.14659.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In 8th International Conference
on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=HklBjCEKvH.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris
Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion
Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and SlavPetrov. Natural questions: A benchmark for question answering research. Transactions of the
Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl a 00276. URL
https://aclanthology.org/Q19-1026.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR,
2023.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman
Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, andDouwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Hugo
Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin,
editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural
Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL
https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e
5-Abstract.html.
Jimmy J. Lin. A proposed conceptual framework for a representational approach to information
retrieval. ACM SIGIR Forum, 55:1 – 29, 2021.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni,
and Percy Liang. Lost in the middle: How language models use long contexts. ArXiv preprint,
abs/2307.03172, 2023a. URL https://arxiv.org/abs/2307.03172.

ACM SIGIR Forum

13

Vol. 57 No. 2 – December 2023

Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, P. Zhang, Yuxiao Dong, and
Jie Tang. Webglm: Towards an efficient web-enhanced question answering system with humanpreferences. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery andData Mining, 2023b.
Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. Rethinking search: making domain experts
out of dilettantes. In Acm sigir forum, volume 55, pages 1–27. ACM New York, NY, USA, 2021.
Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations
and Trends® in Information Retrieval, 13(1):1–126, 2018.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher
Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted
question-answering with human feedback. ArXiv preprint, abs/2112.09332, 2021. URL https://arxiv.org/abs/2112.09332.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and
Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016.
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking
with bert. ArXiv preprint, abs/1910.14424, 2019a. URL https://arxiv.org/abs/1910.14424.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query
prediction. ArXiv preprint, abs/1904.08375, 2019b. URL https://arxiv.org/abs/1904.083
75.
OpenAI. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL https://arxiv.
org/abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong
Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
instructions with human feedback. Advances in Neural Information Processing Systems, 35:
27730–27744, 2022.
Martin Potthast, Matthias Hagen, and Benno Stein. The dilemma of the direct answer. ACM
SIGIR Forum, 54:1 – 12, 2021.
Ronak Pradeep, Kai Hui, Jai Gupta, Adam D Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler,
and Vinh Q Tran. How does generative retrieval scale to millions of passages? ArXiv preprint,
abs/2305.11841, 2023. URL https://arxiv.org/abs/2305.11841.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya
Sutskever. Learning transferable visual models from natural language supervision. In Marina
Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine
Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine
Learning Research, pages 8748–8763. PMLR, 2021. URL http://proceedings.mlr.press/v1
39/radford21a.html.
ACM SIGIR Forum

14

Vol. 57 No. 2 – December 2023

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified
text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL http://jmlr.org/papers/v21/20-074.html.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin LeytonBrown, and Yoav Shoham. In-context retrieval-augmented language models. ArXiv preprint,
abs/2302.00083, 2023. URL https://arxiv.org/abs/2302.00083.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical textconditional image generation with clip latents. ArXiv preprint, abs/2204.06125, 2022. URL
https://arxiv.org/abs/2204.06125.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), pages 3982–3992, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. ArXiv preprint,
abs/1911.02150, 2019. URL https://arxiv.org/abs/1911.02150.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmented black-box language models. ArXiv
preprint, abs/2301.12652, 2023. URL https://arxiv.org/abs/2301.12652.
Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt
good at search? investigating large language models as re-ranking agent. ArXiv preprint,
abs/2304.09542, 2023. URL https://arxiv.org/abs/2304.09542.
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui,Zhe Zhao, Jai Gupta, et al. Transformer memory as a differentiable search index. Advances in
Neural Information Processing Systems, 35:21831–21843, 2022.
Nandan Thakur, Nils Reimers, Andreas Ruckl’e, Abhishek Srivastava, and Iryna Gurevych. Beir:
A heterogenous benchmark for zero-shot evaluation of information retrieval models. ArXiv
preprint, abs/2104.08663, 2021. URL https://arxiv.org/abs/2104.08663.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, TimothéeLacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez,
Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation
language models. ArXiv preprint, abs/2302.13971, 2023. URL https://arxiv.org/abs/2302.13971.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. ArXiv
preprint, abs/2212.03533, 2022. URL https://arxiv.org/abs/2212.03533.
ACM SIGIR Forum

15

Vol. 57 No. 2 – December 2023

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. SimLM: Pre-training with representation bottleneck for dense passage retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2244–2258, Toronto, Canada, July 2023a.
Association for Computational Linguistics. doi: 10.18653/v1/2023.acl- long.125. URL
https://aclanthology.org/2023.acl-long.125.
Liang Wang, Nan Yang, and Furu Wei. Query2doc: Query expansion with large language models.
ArXiv preprint, abs/2303.07678, 2023b. URL https://arxiv.org/abs/2303.07678.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advancesin Neural Information Processing Systems, 35:24824–24837, 2022.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed,
and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense
text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual
Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/for
um?id=zeFrfgyZln.
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder,and Furu Wei. Inference with reference: Lossless acceleration of large language models. ArXiv
preprint, abs/2304.04487, 2023. URL https://arxiv.org/abs/2304.04487.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation.
In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pages 5657–5673, Abu Dhabi, United Arab Emirates, 2022. Association for Computational
Linguistics. URL https://aclanthology.org/2022.emnlp-main.382.
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose:
Efficient context window extension of llms via positional skip-wise training. ArXiv preprint,
abs/2309.10400, 2023. URL https://arxiv.org/abs/2309.10400.

ACM SIGIR Forum

16

Vol. 57 No. 2 – December 2023