Boosting Search Engines with Interactive Agents

1 INTRODUCTION . Can machines learn to use a search engine as an interactive tool for finding information ? Web search is the portal to a vast ecosystem of general and specialized knowledge , designed to support humans in their effort to seek relevant information and make well-informed decisions . Utilizing search as a tool is intuitive , and most users quickly learn interactive search strategies characterized by sequential reasoning , exploration , and synthesis ( Hearst , 2009 ; Rutter et al. , 2015 ; Russell , 2019 ) . The success of web search relies on machines learning human notions of relevance , but also on the users ’ ability to ( re- ) formulate appropriate queries , grounded in a tacit understanding of strengths and limitations of search engines . Given recent breakthroughs in language models ( LM ) ( Vaswani et al. , 2017 ; Devlin et al. , 2019 ; Brown et al. , 2020 ) as well as in reinforcement learning ( RL ) ( Mnih et al. , 2013 ; Silver et al. , 2016 ; Berner et al. , 2019 ) , it seems timely to ask whether , and how , agents can be trained to interactively use search engines . However , the lack of expert search sessions puts supervised learning out of reach , and RL is often ineffective in complex natural language understanding ( NLU ) tasks . The feasibility of autonomous search agents hence remains an open question , which inspires our research . We pursue a design philosophy in which search agents operate in structured action spaces defined as generative grammars , resulting in compositional , productive , and semantically transparent policies . Further domain knowledge is included through the use of well-known models and algorithms from NLU and information retrieval ( IR ) . Most notably , we develop a self-supervised learning scheme for generating high-quality search session data , by exploiting insights from relevance feedback ( Rocchio , 1971 ) , used to train a supervised LM search agent based on T5 ( Raffel et al. , 2020 ) . We also build an RL search agent based on MuZero ( Schrittwieser et al. , 2020 ) and BERT ( Devlin et al. , 2019 ) , which performs planning via rule-constrained Monte Carlo tree search and a learned dynamics model . We run experiments on an open-domain question answering task , OpenQA ( Lee et al. , 2019 ) . Search agents learn diverse policies leading to deep , effective explorations of the search results . The MuZero agent outperforms a BM25 ( Robertson & Zaragoza , 2009 ) search function running over a Wikipedia index , on both retrieval and answer quality metrics . Thus , providing novel evidence for the potential of knowledge-infused RL in hard NLU tasks . The T5 agent can more easily leverage large pre-trained encoder-decoders and proves superior to MuZero . Furthermore , a straightforward ensemble of agents is comparable in performance to a state of the art neural retrieval system , DPR ( Karpukhin et al. , 2020 ) , while relying solely on interpretable , symbolic retrieval operations . This suggests new challenges for future work ; e.g. , involving hybrid architectures and policy synthesis.1 1We open-source the code and trained checkpoints for both agents : anonymized during review . 2 SELF-SUPERVISED LEARNING FOR INTERACTIVE SEARCH . It has been a powerful vision for more than 20 years to design search engines that are intuitive and simple to use . Despite their remarkable success , search engines are not perfect and may not yield the most relevant result ( s ) in one shot . This is particularly true for rare and intrinsically difficult queries , which may require interactive exploration by the user to be answered correctly and exhaustively . Contextual query refinement is a common technique ( Jansen et al. , 2009 ) , even among children ( Rutter et al. , 2015 ) , used to improve search by combining evidence from previous results and background knowledge ( Huang & Efthimiadis , 2009 ) . Such refinements often rely on inspecting result snippets and titles or on skimming the content of top-ranked documents . This process is iterative and may be repeated to produce a sequence of queries q0 , q1 , . . . , qT until ( optimistically ) a satisfactory answer is found . It seems natural to mimic this interactive process by a search agent , which learns the basic step of generating a follow-up query from previous queries and their search results . Furthermore , it is noteworthy how power users apply dedicated search operators and sophisticated investigative strategies to solve deep search puzzles ( Russell , 2019 ) . In particular , unary operators offer a great deal of fine-grained control and transparency and as such are highly effective in expert hands . We concentrate on three operators : ‘ + ’ , which limits results to documents that contain a specific term , ‘ - ’ which excludes results that contain the term , and ‘ ∧i ’ which boosts a term weight in the BM25 score computation by a factor i ∈ R. For instance – see also Figure 1 – the query ’ who won the us open ’ , may lead to both tennis and golf-related results . An expert searcher could zero-in on the tennis intent by excluding golf-related terms and boosting tennis-related ones . As we show in this paper , these operators are also pivotal in designing interactive search agents . 2.1 RESULT AGGREGATION FOR CONTEXTUAL REFINEMENT . Web searchers expect the best answer to be among the top two hits on the first results page ( Hearst , 2009 , §5 ) and pay marginal attention to the bottom half of the 10 blue links ( Granka et al. , 2004 ; Joachims et al. , 2005 ; Nielsen & Pernice , 2009 ; Strzelecki , 2020 ) . Likewise , a search agent considers only the top k documents returned by the search engine at every step , where k = 5 . During a search session the agent maintains a list of the top-k documents overall , which is returned at the end . To aggregate results we use a machine reader ( MR , cf . ( Rajpurkar et al. , 2016 ) ) . Specifically , we use a DPR-like reader/passage scorer ( Karpukhin et al. , 2020 ) , which builds upon a pre-trained BERT model . Beyond identifying the most promising answer span within each result document d , the system also estimates the probability of d containing the ( unspecified ) answer P ( d 3 ANSWER | q ) ∈ [ 0 ; 1 ] . This probability can be viewed as a Passage Score ( PS ) that induces a calibrated ranking across all result documents within a session . An observation representing the session at any given step is built by extracting a fixed-length token window centered at the answer span predicted by the reader for each document . In addition , we include the document titles . Finally , the query tokens and refinements describing qt are also included . This leads to a segmented observation token sequence ot which is truncated to length ≤ 512 , a common input length for pre-trained transformer-based LMs ( cf . Appendix B for details , examples ) . We then use BERT or T5 to produce an embedding st from which the search agent will generate the next query . If we denote the result set for qt by Dt , then we get diagrammatically q0 , . . . , qtsearch ↓ engine D0 , . . . , Dt MR/PS7−→ ot︸ ︷︷ ︸ observation LM7−→ st︸ ︷︷ ︸ encoding agent7−→ qt+1︸ ︷︷ ︸ generation ( 1 ) We focus on the case where qt+1 is obtained from qt through augmentation . This may add a keyword w ∈ Σidx , where Σidx is the search index vocabulary , with the usual disjunctive search engine semantics or a structured search term formed by the use of unary operators ( ‘ + ’ , ‘ - ’ , ‘ ∧ ’ ) and fields . 2.2 ROCCHIO QUERY EXPANSIONS . In the absence of training sessions from human expert users , we propose to generate synthetic search sessions in a self-supervised manner , making use of a set of question-answer pairs ( q , a ) . We initialize q0=q and aim to find a sequence of refinements that make progress towards identifying documents containing the answer a , based on a reward function qt 7→ Dt 7→ rt ∈ [ 0 ; 1 ] ( cf . §4 ) . A query is not further refined , if either t=20 ( maximal length ) or if no score increasing refinement can be found . To create candidate refinements , we make use of the idea of relevance feedback as suggested in Rocchio ( 1971 ) . An elementary refinement – called a Rocchio expansion – then takes the form qt+1 : = qt ∆qt , ∆qt : = [ +| − | ∧i TITLE | CONTENT ] wt , wt ∈ Σt : = Σqt ∪ Στt ∪ Σαt ∪ Σβt ( 2 ) where i is the boosting coefficient and Σt refers to a set of terms accessible to the agent . By that we mean terms that occur in the top PS-ranked session documents . We use superscripts to refer to the vocabulary of the question ( q ) , titles ( τ ) , answers ( α ) or bodies ( β ) of documents in ot . Note that adding terms 6∈ Σt would make refinements difficult to reproduce for an agent and thus would provide supervision of low utility . Another aspect of creating sessions as described above has to do with the search complexity of finding optimal sequences of Rocchio expansions . We consider q∗ = q + a as the “ ideal ” query , whose results define the vocabulary Σ∗ . For efficiency reasons , we further constrain the terms to be added via exact matches , term boosting or term exclusions by defining respective constrained dictionaries Σ↑t = Σt ∩ Σ∗ , Σ↓t = Σt − Σ∗ . ( 3 ) This means it is possible to upgrade accessible terms , wt , to exact matches , or weight boosting , if they also occur in the ideal result set ( wt ∈ Σ↑t ) ; and to exclude accessible terms if they are not present in the ideal results ( wt ∈ Σ↓t ) . We have found experimentally that this leads to a good trade-off between the quality of Rocchio expansions and the search effort to find them . The search for sequences of Rocchio expansions is done heuristically . More details , pseudo-code illustrating the procedure and examples can be found in §5 , Appendix A Appendix G . 2.3 SELF-SUPERVISED T5 AGENT . We suggest to train a generative search agent in a supervised manner by making use of synthetic search sessions generated by Rocchio expansions . We use T5 , a pretrained transformer encoder-decoder model which achieves state-of-the-art results on multiple NLU tasks . As a search agent , T5 predicts a single new search expansion from an observed state . In the spirit of everything-is-string-prediction , both state and expansions are represented as plain strings . See Appendix B for a full example . Our T5 agent is trained via Behavioral Cloning ( BC ) ( Michie , 1990 ) . We treat each step in a Rocchio session as a single training example . As is common in sequence prediction tasks , we use the crossentropy loss for optimization . BC is perhaps the simplest form of Imitation Learning ( IL ) , and has been proven effective in a variety of application domains ( Sharma et al. , 2018 ; Rodrı́guez-Hernandez et al. , 2019 ) . In our query refinement task , it allows to inherit the expressive power of the Rocchio query expansions and , differently from other IL approaches ( Ross et al. , 2011 ; Ho & Ermon , 2016 ; Ding , 2020 ) , requires only offline interactions with the search engine . Crucially , this enables scaling to the large action spaces and model sizes typical of recent LMs . Our T5 agent can also be described as a Decision Transformer with fixed max return ( Chen et al. , 2021 ) . At test time , we start with the initial query and incrementally add new expansions , querying the trained T5 model in every step . We then use the refined query to retrieve new documents and continue until either the set of new documents is empty or we reach the maximum number of steps . Throughout the session , we maintain the top-5 documents among all those retrieved .

Summary: This work designs an RL (customized BERT MuZero) agent for query reformulation, where the underlying observation is defined as an aggregation of windows of answer spans found with a machine reading module, document titles, query tokens, and refinements. The action space involves adding keywords to the query or form search terms with three operators that constrain the search space. Due to the lack of training data, synthetic search sessions are generated with a pretrained transformer model and imitation learning. The reward is a linear combination of NDCG (well-established IR ranking for non-binary relevance), a revised NDCG-like metric for exact matches (NDCEM) and a top-k Passage Score. Results on an OpenQA dataset are *close* to neural passage retrieval methods.