Boosting Search Engines with Interactive Agents

1 INTRODUCTION . Can machines learn to use a search engine as an interactive tool for finding information ? Web search is the portal to a vast ecosystem of general and specialized knowledge , designed to support humans in their effort to seek relevant information and make well-informed decisions . Utilizing search as a tool is intuitive , and most users quickly learn interactive search strategies characterized by sequential reasoning , exploration , and synthesis ( Hearst , 2009 ; Rutter et al. , 2015 ; Russell , 2019 ) . The success of web search relies on machines learning human notions of relevance , but also on the users ’ ability to ( re- ) formulate appropriate queries , grounded in a tacit understanding of strengths and limitations of search engines . Given recent breakthroughs in language models ( LM ) ( Vaswani et al. , 2017 ; Devlin et al. , 2019 ; Brown et al. , 2020 ) as well as in reinforcement learning ( RL ) ( Mnih et al. , 2013 ; Silver et al. , 2016 ; Berner et al. , 2019 ) , it seems timely to ask whether , and how , agents can be trained to interactively use search engines . However , the lack of expert search sessions puts supervised learning out of reach , and RL is often ineffective in complex natural language understanding ( NLU ) tasks . The feasibility of autonomous search agents hence remains an open question , which inspires our research . We pursue a design philosophy in which search agents operate in structured action spaces defined as generative grammars , resulting in compositional , productive , and semantically transparent policies . Further domain knowledge is included through the use of well-known models and algorithms from NLU and information retrieval ( IR ) . Most notably , we develop a self-supervised learning scheme for generating high-quality search session data , by exploiting insights from relevance feedback ( Rocchio , 1971 ) , used to train a supervised LM search agent based on T5 ( Raffel et al. , 2020 ) . We also build an RL search agent based on MuZero ( Schrittwieser et al. , 2020 ) and BERT ( Devlin et al. , 2019 ) , which performs planning via rule-constrained Monte Carlo tree search and a learned dynamics model . We run experiments on an open-domain question answering task , OpenQA ( Lee et al. , 2019 ) . Search agents learn diverse policies leading to deep , effective explorations of the search results . The MuZero agent outperforms a BM25 ( Robertson & Zaragoza , 2009 ) search function running over a Wikipedia index , on both retrieval and answer quality metrics . Thus , providing novel evidence for the potential of knowledge-infused RL in hard NLU tasks . The T5 agent can more easily leverage large pre-trained encoder-decoders and proves superior to MuZero . Furthermore , a straightforward ensemble of agents is comparable in performance to a state of the art neural retrieval system , DPR ( Karpukhin et al. , 2020 ) , while relying solely on interpretable , symbolic retrieval operations . This suggests new challenges for future work ; e.g. , involving hybrid architectures and policy synthesis.1 1We open-source the code and trained checkpoints for both agents : anonymized during review . 2 SELF-SUPERVISED LEARNING FOR INTERACTIVE SEARCH . It has been a powerful vision for more than 20 years to design search engines that are intuitive and simple to use . Despite their remarkable success , search engines are not perfect and may not yield the most relevant result ( s ) in one shot . This is particularly true for rare and intrinsically difficult queries , which may require interactive exploration by the user to be answered correctly and exhaustively . Contextual query refinement is a common technique ( Jansen et al. , 2009 ) , even among children ( Rutter et al. , 2015 ) , used to improve search by combining evidence from previous results and background knowledge ( Huang & Efthimiadis , 2009 ) . Such refinements often rely on inspecting result snippets and titles or on skimming the content of top-ranked documents . This process is iterative and may be repeated to produce a sequence of queries q0 , q1 , . . . , qT until ( optimistically ) a satisfactory answer is found . It seems natural to mimic this interactive process by a search agent , which learns the basic step of generating a follow-up query from previous queries and their search results . Furthermore , it is noteworthy how power users apply dedicated search operators and sophisticated investigative strategies to solve deep search puzzles ( Russell , 2019 ) . In particular , unary operators offer a great deal of fine-grained control and transparency and as such are highly effective in expert hands . We concentrate on three operators : ‘ + ’ , which limits results to documents that contain a specific term , ‘ - ’ which excludes results that contain the term , and ‘ ∧i ’ which boosts a term weight in the BM25 score computation by a factor i ∈ R. For instance – see also Figure 1 – the query ’ who won the us open ’ , may lead to both tennis and golf-related results . An expert searcher could zero-in on the tennis intent by excluding golf-related terms and boosting tennis-related ones . As we show in this paper , these operators are also pivotal in designing interactive search agents . 2.1 RESULT AGGREGATION FOR CONTEXTUAL REFINEMENT . Web searchers expect the best answer to be among the top two hits on the first results page ( Hearst , 2009 , §5 ) and pay marginal attention to the bottom half of the 10 blue links ( Granka et al. , 2004 ; Joachims et al. , 2005 ; Nielsen & Pernice , 2009 ; Strzelecki , 2020 ) . Likewise , a search agent considers only the top k documents returned by the search engine at every step , where k = 5 . During a search session the agent maintains a list of the top-k documents overall , which is returned at the end . To aggregate results we use a machine reader ( MR , cf . ( Rajpurkar et al. , 2016 ) ) . Specifically , we use a DPR-like reader/passage scorer ( Karpukhin et al. , 2020 ) , which builds upon a pre-trained BERT model . Beyond identifying the most promising answer span within each result document d , the system also estimates the probability of d containing the ( unspecified ) answer P ( d 3 ANSWER | q ) ∈ [ 0 ; 1 ] . This probability can be viewed as a Passage Score ( PS ) that induces a calibrated ranking across all result documents within a session . An observation representing the session at any given step is built by extracting a fixed-length token window centered at the answer span predicted by the reader for each document . In addition , we include the document titles . Finally , the query tokens and refinements describing qt are also included . This leads to a segmented observation token sequence ot which is truncated to length ≤ 512 , a common input length for pre-trained transformer-based LMs ( cf . Appendix B for details , examples ) . We then use BERT or T5 to produce an embedding st from which the search agent will generate the next query . If we denote the result set for qt by Dt , then we get diagrammatically q0 , . . . , qtsearch ↓ engine D0 , . . . , Dt MR/PS7−→ ot︸ ︷︷ ︸ observation LM7−→ st︸ ︷︷ ︸ encoding agent7−→ qt+1︸ ︷︷ ︸ generation ( 1 ) We focus on the case where qt+1 is obtained from qt through augmentation . This may add a keyword w ∈ Σidx , where Σidx is the search index vocabulary , with the usual disjunctive search engine semantics or a structured search term formed by the use of unary operators ( ‘ + ’ , ‘ - ’ , ‘ ∧ ’ ) and fields . 2.2 ROCCHIO QUERY EXPANSIONS . In the absence of training sessions from human expert users , we propose to generate synthetic search sessions in a self-supervised manner , making use of a set of question-answer pairs ( q , a ) . We initialize q0=q and aim to find a sequence of refinements that make progress towards identifying documents containing the answer a , based on a reward function qt 7→ Dt 7→ rt ∈ [ 0 ; 1 ] ( cf . §4 ) . A query is not further refined , if either t=20 ( maximal length ) or if no score increasing refinement can be found . To create candidate refinements , we make use of the idea of relevance feedback as suggested in Rocchio ( 1971 ) . An elementary refinement – called a Rocchio expansion – then takes the form qt+1 : = qt ∆qt , ∆qt : = [ +| − | ∧i TITLE | CONTENT ] wt , wt ∈ Σt : = Σqt ∪ Στt ∪ Σαt ∪ Σβt ( 2 ) where i is the boosting coefficient and Σt refers to a set of terms accessible to the agent . By that we mean terms that occur in the top PS-ranked session documents . We use superscripts to refer to the vocabulary of the question ( q ) , titles ( τ ) , answers ( α ) or bodies ( β ) of documents in ot . Note that adding terms 6∈ Σt would make refinements difficult to reproduce for an agent and thus would provide supervision of low utility . Another aspect of creating sessions as described above has to do with the search complexity of finding optimal sequences of Rocchio expansions . We consider q∗ = q + a as the “ ideal ” query , whose results define the vocabulary Σ∗ . For efficiency reasons , we further constrain the terms to be added via exact matches , term boosting or term exclusions by defining respective constrained dictionaries Σ↑t = Σt ∩ Σ∗ , Σ↓t = Σt − Σ∗ . ( 3 ) This means it is possible to upgrade accessible terms , wt , to exact matches , or weight boosting , if they also occur in the ideal result set ( wt ∈ Σ↑t ) ; and to exclude accessible terms if they are not present in the ideal results ( wt ∈ Σ↓t ) . We have found experimentally that this leads to a good trade-off between the quality of Rocchio expansions and the search effort to find them . The search for sequences of Rocchio expansions is done heuristically . More details , pseudo-code illustrating the procedure and examples can be found in §5 , Appendix A Appendix G . 2.3 SELF-SUPERVISED T5 AGENT . We suggest to train a generative search agent in a supervised manner by making use of synthetic search sessions generated by Rocchio expansions . We use T5 , a pretrained transformer encoder-decoder model which achieves state-of-the-art results on multiple NLU tasks . As a search agent , T5 predicts a single new search expansion from an observed state . In the spirit of everything-is-string-prediction , both state and expansions are represented as plain strings . See Appendix B for a full example . Our T5 agent is trained via Behavioral Cloning ( BC ) ( Michie , 1990 ) . We treat each step in a Rocchio session as a single training example . As is common in sequence prediction tasks , we use the crossentropy loss for optimization . BC is perhaps the simplest form of Imitation Learning ( IL ) , and has been proven effective in a variety of application domains ( Sharma et al. , 2018 ; Rodrı́guez-Hernandez et al. , 2019 ) . In our query refinement task , it allows to inherit the expressive power of the Rocchio query expansions and , differently from other IL approaches ( Ross et al. , 2011 ; Ho & Ermon , 2016 ; Ding , 2020 ) , requires only offline interactions with the search engine . Crucially , this enables scaling to the large action spaces and model sizes typical of recent LMs . Our T5 agent can also be described as a Decision Transformer with fixed max return ( Chen et al. , 2021 ) . At test time , we start with the initial query and incrementally add new expansions , querying the trained T5 model in every step . We then use the refined query to retrieve new documents and continue until either the set of new documents is empty or we reach the maximum number of steps . Throughout the session , we maintain the top-5 documents among all those retrieved .

Summary: This paper investigates a vital direction: How can RL agents learn to use search engines to find information. The authors propose a method that leverages pre-trained models conditioned on machine reading results to guide the selection of query refinement from aggregated search documents. The method performs comparably to neural retrievers on OpenQA-NQ but operates in a more interpretable way.