Decomposing Complex Queries for Tip-of-the-tongue Retrieval
Authors: Kevin Lin
Kyle Lo
Joseph E. Gonzalez
Dan Klein
Venue: EMNLP 2023 Findings (2023)
Keywords: information retrieval; large language models; query decomposition
TL;DR: Decomposing Complex Queries for Tip-of-the-tongue Retrieval

Abstract: When re-finding items, users who forget or are uncertain about identifying details often rely on creative strategies for expressing their information needs---complex queries that describe content elements (e.g., book characters or events), information beyond the document text (e.g., descriptions of book covers), or personal context (e.g., when they read a book). Standard retrieval models that rely on lexical or semantic overlap between query and document text are challenged in such retrieval settings, known as tip-of-the-tongue (TOT) retrieval.
We introduce a simple but effective framework for handling such complex queries by decomposing the query with an LLM into individual clues routing those as subqueries to specialized retrievers, and ensembling the results.
Our approach takes advantage of off-the-shelf retrievers (e.g., CLIP for retrieving images of book covers) or incorporate retriever-specific logic (e.g., date constraints). We show that our framework incorporating query decomposition into retrievers can improve gold book recall up to 6\% absolute gain for Recall@5 on a new collection of 14,441 real-world query-book pairs from an online community for resolving TOT inquiries.

Decomposing Complex Queries for Tip-of-the-tongue Retrieval
Kevin Lin
Kyle Lo
Joseph E. Gonzalez
Dan Klein
University of California Berkeley
< Allen Institute for AI
jegonaal klein)@berkeley.com
kylel@allenai.
Abstract
cover book was yellow with silhouette cat. pretty by a male author. think it was named something cat came back", but haven't to find under that title. This story told from the perspective of young boy. cat enters his becomes hypnotized and will do anything it.... read this in grade school in the middle 2000's and it really
When re-finding items, users who forget or are uncertain about identifying details often rely creative strategies for expressing their informa- tion needs complex queries that describe con- tent elements (e.g.. book characters events), information beyond the document text (e.g.- descriptions of book covers), or personal con- text (e.g., when they read a book). Standard retrieval models that rely on lexical or seman- tic overlap between query and document text are challenged in such retrieval settings, known as tip of the tongue (TOT) retrieval. We intro- duce simple but effective framework for han- dling such complex queries by decomposing the query with LLM into individual clues, routing those subqueries specialized re- trievers, and ensembling the results. Our ap- proach takes advantage of off-the-shelf retriev- ers (e.g., CLIP for retrieving images of book covers) or incorporate surriverr-specfff logic (e.g., date constraints). We show that frame- work incorporating query decomposition into retrievers can improve gold book recall up to 6% absolute gain for Recall @5 WhatsThat- Book, a new collection of 14,441 real-world query-book pairs from an online community for resolving TOT inquiries."
written
everyone
make sure I'm not I'm sure exists...
Query Decomposer
Retrieval Experts
Date Retriever
Title Retriever
Author Retriever
Cover Retriever
Plot Retriever
Genre Retriever
Semi- Structuredd Document
Sept 4,
The Improbable
2003
Abilberg
begins David's family takes
Fantasy, Young
stray kitton. At least the creature appears be...
Adult,
Childrans,
Animals,
Mystery
To resolve tip-of-the-tonge queries about books, our approach decomposes long, complex queries into subqueries routed to specific retrieval experts, each trained to handle particular aspect of book.
Figure 1:
1 Introduction
Tip of the tongue (TOT) refers to the retrieval set- ting in which user is unable to formulate precise query that identifies sought item, even if the user knows they's ve encountered this item before. For example, users searching for movies they watched or books they read long ago often resort to com- plex and creative queries that employ a diverse set of strategies to express information relevant to the sought item- high-leeell categories (e.g., topic, genre), content details from the movie or book (c.g., events, characters), references to personal context
(e.g., when they last read the book), descriptions of extratextual elements (e.g., movie promotional posters, book covers), and more. In fact, in an an- notation study of TOT queries for movics, Arguello et al. (2021) found over 30 types of informational facets that users may include when crafting queries. Figure shows a TOT query and its corresponding gold book.
A key challenge in TOT retrieval that queries are not just longer and more complex than those in popular retrieval datasets, but resolving them requires an enriched document collection since
We relcase code and data at https://ghhuu..com////2266hhaaas

---

query-document relevance can't necessarily be es- tablished from document content alone (see Ta- ble 1). For example, in Figure 1, the query's s de- scription of the book cover The cover of this book was yellow with a silhouette of a cat-can be highly useful for identifying the book, but necessitates the book's representation to contain that information.
relevant to the metadata field.
Predictive prompting aims to generate sub- queries that are similar the content of the meta- data field. Note that the subqueries need not be extractions from the original query and can be in- ferred metadata or even hallucinations from the query decomposer. Using LLMs to generate sub- queries affords us the ability to set the few-shot prompt generation targets to be predictions. This is important as the information in queries are rarely presented in a form amenable for matching with the corresponding document field. For example, books have publish dates, but queries will rarely mention these dates; instead, users may articulate personal context (e.g., "I read this book in high- school around 2002-2005"). Then to simplify the learning task for a late-focused retrieval expert, we might ask the LLM to predict a "latest possible publish date" (c.g.. 2005).
In this work, we present a simple yet effective technique for improving TOT retrieval: First, we decompose queries into individual subqueries or clues that each capture a single aspect of the tar- get document. Then, we route these subqueries to expert retrievers that can be trained individually. Finally, we combine their results with those from base retriever that receives the original query. Ex- periments show improvement in gold book recall over description-oni retrieval baselines on a set of 14,441 real-world query-book pairs collected from an online forum for resolving TOT inquiries, complete with cover images and metadata.
See Table 2 for examples of extractive and pre- dictive subqueries and Appendix A for examples of their prompts. In practice, we use GPT 3.5 few-shot prompting with up to 8 in-context examples. Each field has its own prompt template and set of examples. Subqueries for differ- ent fields can be generated in parallel, as the they are independent of each other.
2 Decomposing Queries
In this section, we describe our a simple but effective method for tackling long, complex TOMT queries. Given a collection of documents d1,..., dn and a textual query q, the TOT retrieval task aims to identify the sought document d'. The input (raw) documents are semi-structuure;; each document d contains metadata fields (d()).... (kk) (k) In our the WhatsThatBook dataset, the documents are books that have the fields that correspond to plot, its publication year, an image of its book cover, title, genre, and author etc. Missing elements take on a default value (e.g., blank image, earliest pub- lish date in overall book collection).
2.2 Retrieval Experts
We have retriever models, experts, that specialize to specific field types. Let R1,..., : Rg represent these retrievers. Retrievers can be implemented dense, sparse, or symbolic logic.
If retriever requires training, we run the query decomposer over all query- pairs (g. d) in the training set. This produces effectively train- ing datasets, where each dataset is comprised of a subquery and document-field pair. For example, field j would have training dataset of examples (d()).
2.1 Method
First, the query decomposer takes query & and outputs a set of subqueries ( (tt),, for k metadata fields. To do this, we in-context learn- ing with large language model (LLM) to generate a subquery q that is relevant to that field or op- tionally output the string "N/A" if the g does not contain any relevant information to the field; this is repeated for each field and can be run in indepen- dently in parallel for each field.
At indexing time, each document's 's field is in- dexed according to the specifications of its retriever expert. For example, if the retriever is implemented as an embedding model, then that specific field is converted into an embedding. On the other hand, if the retriever a sparse model, then a sparse index would be built using just that specific field's 's text.
We experiment with two prompting strategies, extractive and predictive. Extractive prompting aims to generate subqueries that are purely ex- tractions from the original query. The LLM is in- structed to output "N/A" if there is not information
At inference time, each retriever takes sub- query (نب) and retrieves documents from its associ- ated index of fields.

---

Dataset
Query Length
Lexical Overlap
MSMarco (Campos al., 2016)
7.68
0.55
Natural Questions (Kwiatkowski al., 2019)
10.35
0.52
BioASQ (Tsatsaronis et al., 2015)
14.82
0.58
TREC-COVID (Roberts al., 2020)
15.94
0.41
SciFact (Wadden al., 2022)
19.52
0,50
HotPotQA (Yang al., 2018h)
22.78
0.45
TOMT (Bhargav et al., 2022)
136.50
0.25
WhatsThatBook
156.20
0.19
Table Tip of the tongue (TOT) queries are significantly longer while also having less lexical overlap with the gold document, compared with queries in popular retrieval datasets. Query length is number of BPE (Sennrich et al., 2016) pieces, averaged across examples. Lexical overlap fraction of whole words in query that occur in gold passage(s), averaged across examples.
2.3 Implementation details
s(q, d) = Σ w (2(00) j=1
In practice, for titles, authors, plot, and genre, use Contriever (Izacard al., 2021), state-of-the- art dense retriever. For both models, we train for total of 10,000 steps with a batch size of 16, learn- ing rate Ic-4. For titles, we finetune with 3,327 extracted subqueries. For our base retriever, we use the full training set of original book deseriptions. For embedding search during inference, we use the faiss library and project all embeddings to 768 Johnson (2022).
All documents are scored in this manner, which induces document ranking for a given query q. We tune the weights W the validation set and select the weights that have the best Recall
Datasets
WhatsThatBook We introduce the WhatsThat- Book dataset consisting of query-book pairs col- lected from public online forum GoodReads for resolving TOT inquiries about books.* On this forum, users post inquiries describing their sought book and community members reply with links books on GoodReads as proposed answers." 5 If the searcher accepts a book as the correct answer, the post is manually tagged as SOLVED and a link to the found book is pinned to the thread. For these solved threads, we take the original inquiry as our query q and the linked book gold d' At the end, WhatsThatBook contains 14,441 query-book pairs. Each query corresponds to a unique book. Finally, these books are associated with pages on GoodReads, which we used to obtain publication year metadata and images of book covers. We fur- ther collect 172,422 negative books (books that do not correspond to gold query in our dataset) to make the setting more realistic, for a total of 186,863 books. To collect negative books, crawl books authored by crawling books by the authors of the positive books.
For cover images, we use CLIP (Radford et al., 2021). a state-of-the-at image-text model that can be used for image retrieval by scoring matches be- tween embedded images and their textual descrip- tions. Specifically, we finetune ViT-B/323 on 2,220 extracted subqueries using cross-entropy loss with batch size of 4, learning rate of 5e-5 and weight decay of 0.2 for 10 epochs with the Adam opti- mizer (Kingma and Ba, 2014). We select the model with the best top retrieval accuracy on validation set.
For publish dates useing the predictive prompt- ing, we use a symbolic function that heuristically scores 0 if a book published after the subquery date (i.e. predicted latest publish date) and oth- erwise. If necessary, we heuristically resolve the subquery a year.
2.4 Combining retrieved results
In this work, we restrict to simple strategy of using weighted sum of all & retrieval scores across the (g) dU)). That is, the final score is:
http:///www.goodraadss. com/grroppsshow115 wit-hhhwwwwSSw-h--s-- the-name-of of-thhf--oooooooo We scraped data from February 2022.
hhtps//aagginngeeee.wwoeeeboooou http://hngiingffcccccommmeeemmem B-32
'This is a simplification of community interactions. Threads also may include dialogue between original poster and members but this is heyond the scope of our work.

---

Query-Document
Clue-Field
Cluc-Field
Cluc-Field
a used store once and I remember saying my hushand "my daughter use read that book to little brother" and it's funny because cover is little girl reading a book outside little brother. called....my book, or my story, or something simple like that. It would about or more years old. The girl was blond and the boy brunct....I think!!!! Inside was cutest little sentences and my kids use to do what each page shink..thnnk you!!
Query: think
Extracted Tite: It's called.....my book, or my story, or some- thing simple that.
Extracted Date Clue: years old (2006 earlier)
Extructed Cover: The outside enver little girl reading a book her little brother.
Actual Cover:
f My First Book
Description: Glossy pictorial hardcover no dust jacket. 2001 7.75x9. 13x25, GUIDE FOR PAR- ENTS WITH PICTURES, HOW TO TEACHING CHILDREN READING.
Actual
Tille Actual Date First pub- lished September 1, 1984
My First Book
Query: read book in 2008 or 2009 that was part of series for young adults. It was fantasy. and about several families of witches and war- lockes. The main character named Holly.
Extracted Genre: for
Predicted Plot: Holly young witch who comes from a long line of magical families...
Predicted Young woman with long, curly hair holding wand and surouunded by swirling colors and symbols
Cover:
young adults, fantasy
Actual Cover:
WINK
Description: Holly Cathers's 's world shatters when her parents are killed in rafting accident. She is wrenched from her home in San Fran- cisco and sent to Scattle to live with her aunt, Marie-Claire, and her twin cousins, Amanda and Nicole...
Actual Genre:
Actual Plot: (see de-
Fantasy, Young Adult, Paranor- mal, Romance, Fiction
scription)
NANCY HOLDER
Query: Written in the 1990s, by [probably] one those romance novelists (not Nora Roberts though). The cover was just snowed-in cabin on a mountain side at night. In a nut shell An escaped prisoner takes woman captive and steals her car with her in An! idcas what this might be called? haven't forgotten it after all these years.
Extracted Author:
Predicted Cover: Dark, mysterious cabin ob- scured by falling snow with the silhouettc...
Extructed Cover: The cover was just a snowed- in cabin on a mountain side at night.
by [probably]
one of those romance novel- ists (not Nora Roberts though)
Actual Cover:
UDITH MINNIGHT
Description: A rootless foster child, Julie Math- ison had blossomed under the love showered upon her by her adoptive family. Now a lovely and vivacious young woman...
Actual Author:
Actual
Cover:
(see
Judith
Mc-
right)
Naught
Table 2: Query-document pairs, their generated sub-queries clues, and corresponding gold document fields. Clues can extracted directly from the query or predicted as a best-guess attempt to match actual document field. Clues can be about the book's 's title, author, date, cover, genre, general plot clements.

---

Extract
Plot
Dates
Cover
Title
Genre
Author
Num Posts
2657 (89.3%)
138 (4.6%)
43 (1.4%)
80 (2.7%)
39 (1.3%)
18 (0.6%)
2975
4630 (96.0%)
2985 (61.9%)
283 (5.9%)
641 (13.3%)
994 (20.6%)
117 (2.4%)
4825
3476 (97.6%)
2759 (77.4%)
856 (24.0%)
1415 (39.7%)
1841 (51.7%)
342 (9.6%)
3563
1821 (98.8%)
1627 (88.3%)
1092 (59.3%)
1458 (79.1%)
1064 (57.7%)
310 (16.8%)
1843
537 (99.4%)
511 (94.6%)
484 (89.6%)
502 (93.0%)
456 (84,4%)
210 (38.9%)
540
50 (100%)
50 (100%)
50 (100%)
(100%)
50 (100%)
50 (100%)
50
Total
13171 (95.5%)
8070 (58.5%)
2808 (20.4%)
4146 (30.1%)
4444 (32.2%)
1047 (7.6%)
13796
Table 3: Number of queries organized by results from running query decomposition with extractive prompting. Rows correspond to number of clues Extracted (between one and six). For instance, top row is queries with only single extracted clue and bottom row is queries with all clue types found. Num Posts counts number of queries in each bucket. Cell counts correspond to number of times given clue type was extracted in post, and percentages normalize against Num Posts. For example, 61.9% of queries with two extracted clues have one "Date" subquery.
Understanding WhatsThatBook queries. To ex- amine the information contained within queries, we analyze the results of query decomposition us- ing extractive prompting (see Table 3). First, we find only 645 (4.5%) posts have no clue extracted (and thus aren't t captured in the table total). Sec- ond, most posts have between one and three clues (μ=2.33). Third, nearly every query contains some description of the book's plot elements. Beyond that, over half of the queries provide some de- scription of temporal context surrounding the book. Queries containing descriptions of the author are rare, which is expected since knowing the author name would likely have allowed the user to find the book without seeking assistance. Given that, it's somewhat surprising that descriptions of titles occur almost third of the time. Manually exam- ining these, find title clues are often uncertain guesses ("I think the title might start with Nurse but I'm not sure") or details that models are likely to struggle with ("The title is made up of either two or four short words (maybe one syllable)").
train (n=11,5522, validation (n=1,444) and test (n=1,4445) sets. By the nature of our dataset con- struction, the number of queries and books is equal. We use all 14,441 books, which are gold targets with respect to some query, as our full docu- ment collection for indexing, As for Reddit-TOMP (Books), given its size we use the entire dataset additional test set.
4 Experiments
ThatsBhatBook
Model
Top 5
Top 10
Top 20
Top 100
BM25
8.5
12.5
16.2
22.5
CLIP
1.9
2.8
3.5
5.7
DPR
23.1
31.2
38.1
57.6
ColBERT
17.8
18.3
25.4
34.1
Contriever
26.5
33.5
40.3
61.3
Contriever (E)
26.7
33.4
41.8
60.5
Contriever (P)
29.3
35.5
42,1
61.7
Contriever (H)
25.0
34.1
40,2
60.4
Contriever (Q)
18.5
24.2
31.8
52.9
Ours (E)
26.6
40.2
60.4
Ours (P)
32.1
40.0
47.1
67.2
Reddit-TOMT (Books) We additionally use the books subset of the Reddit-TOMT (Bhargav et al., 2022) dataset, which includes 2,272 query- pairs.' Queries can refer to the same book, leaving 1.877 unique books in this dataset.
Table 4: Results on test set WhatsTharBook. (Top) Baselines operate directly over queries and descriptions of the books. (Middle) We use the top performing model, Contriever, on the concatenated representations of subqueries (E-extractive, P-predictive, Q-rewritten queries) or the hypothetical document (H). (Bottom) Subqueries routed individual retriever experts.
Experimental setup. For the experiments in the rest of this paper, we split WhatsThatBook into
4.1 Baseline models
"We note the full Reddit- Reddit-TOMT dataset also contains 13K queries matched with movies. To fully explore the ca- pabilities of our approach, we restricted to the books subset specifically due to feasibility of obtaining images of book covers, while doing the same with movic posters was more difficult. We leave this future investigations.
Models that use original queries. We evaluate our approach against several popular retrieval mod- els that have been used baselines for range of other retrieval datasets (see Table 1). For text- only models BM25 (Robertson and Walker, 1997; Robertson and Zaragoza, 2009), Dense Passage
"We removed 47 query-book pairs for which the gold book did not have a GoodReads link, which was necessary for obtaining cover images.

---

Medddi-TOM? Top
Without extracting clues about the book cover, us- ing CLIP on its own is not effective, likely due its limited context window. 8
Model Top
Top 20
Top 100
Contriever 42.8
51.1
58.6
77.7
Ours (P)
44.2
53.6
61.3
79.9
Query decomposition improves performance. When using the rewritten queries without training individual expert retrievers, we observe that con- catenating the predicted clues improve the model. However, both generating rewritten queries and en- tire hypothetical documents perform worse than just using the original document as input. Our ap- proach to decompose queries and route clues specialized retrievers improves performance (rang- ing from +6 to +7 Recall @ K across all cutoffs) over the next best baseline retriever. Table shows the results for each individual expert retriever on WhaosThatBook and the books subset of TOMT.
Table 5: Results on test set Reddit-TOMT (Books). We report the best performing baseline (Contriever) us- ing the original queries and the best performing model using our approach that uses the predicted subqueries (P).
Retrieval (DPR) (Karpukhin et al., 2020), Con- triever (Izacard et al., 2021), and CoIBERT (Khat- tab and Zaharia, 2020) the document representa- tion is simply the concatenation of all available doc- ument fields into single text field. For our image- only baseline- CLIP (Radford et al., 20210 the document representation is only the embedded book cover. All baselines receive the same input (full) query, and are finetuned with the same hy- perparameters ($2.2) except using the full training set (as opposed to just examples with a successful subquery extraction).
Performance degrades predictably with corpus size To see how well the performance scales as more documents are added, we also report the per- formance of the base performing baseline and our model as we add more documents to the corpus. We evaluate the performance for different corpus sizes by varying the number of negative books added to the corpus. Figure 2 shows that for both our decom- position and the baseline, the performance drops significantly as more negative books are added. There is a sharp decline of performance when neg- ative books are first introduced (between corpus size 1 and 2). Since the negative books are col- lected from the authors of the positive books, they may more difficult to distinguish than a random book. As more documents are added, there is a marginal decrease in performance smoothly. Fur- thermore, our decomposition method performance better than the baseline for different corpus sizes and has slightly less performance degradation more documents added to the corpus.
Models that use generated queries. To evaluate the effect of generating subqueries independently without training specialized retrievers, we also train the top performing text-only model from the set of models with queries enriched with LLMs. We experiment with using the concatentation of all sub- queries generated by the query decomposer as the query representation for both the extractive and predicted subqueries. To isolate the effect of gen- erating separate subqueries, we also use LLM to generate single rewritten query that is used as input to Contriever, by simply prompting LLMS to generate single new query that better retrieves the desired item (prompt in Appendix A. In ad- dition, we generate hypothetical documents and use these as the query, similar to Gao and Callan (2022) though we differ in that we train on these generations as queries while Gao and Callan (2022) restricted their use to the zero-shot setting.
Metadata fields exhibit long-tailed behavior. The query decomposer generates a plot subquery in at least 90% of the queries for both WhatsThat- Book and TOMT. Dates occur large proportion of the queries, but are not specific enough to be effective identifying the book. While subqueries for the author appear very infrequently, when they do appear, they are much more effective than more generic subqueries such as dates or genres. Images
4.2 Results
Table 4 shows the test results on Whats WhatsThatBook. We use Recall @ K metric as our primary metric since each query has exactly one correct item.
Contriever is the best-performnng baseline model. In this setting with low lexical overlap, we see that dense retrievers like DPR and Con- triever outperform sparse retrievers like BM25.
We pass the full query into CLIP and allow for truncation to happen naturally. This is big issue with CLIP, which sup- ports narrow query length; hence, motivating our approach to extract clues about book covers from the full query.

---

WhatsThatBook
Reddu-TOMT TOMT (Books)
Subquery
Top
Top
Top 20
Top 100
% not N/A
Top
Top
Top 20
Top 100
not N/A
plot (E)
29.0
36.4
44.9
64.2
90.4
45.5
56.2
61.0
76.0
94.3
plot (P)
27.9
34.7
42.4
61.7
100
43.2
50.5
58.8
77.1
100
dates (E)
1,3
2.1
3.4
6.0
54.8
0.0
1.0
2.2
2.8
59.8
dates (P)
0
0
1.1
2.3
100
0.0
1.8
2.3
4.4
100
cover (F)
5.3
6.7
7.7
14.1
21
8.6
11.6
15.5
17.7
35.8
cover (P)
3.1
4.4
5.6
7.2
100
5.8
8.4
10.0
13.4
100
title (F)
11.7
12.9
14.6
19.8
29.0
13.4
17.5
22.2
37.4
43.9
title (P)
4.3
5.7
13.4
100
14.3
17.2
21.0
35.5
100
genre
2.7
3.4
17.4
28.2
1.3
4.4
28.8
26.8
genre (P)
3.8
6.8
100
6.7
8.6
12.3
25.5
author (E)
6.0
8.0
8.0
9.1
6.9
7.1
8.2
9.6
9.6
9.9
author (P)
0
1
3.2
100
1.9
3.0
4.7
14.5
100
Table Results of individual retrieval experts on the test set of WhatsThatBook Reddia-TOMT (Books). indicate extractive and predictive prompts. Extractive prompts scores calculated only over the "N/A" queries.
on information that not explicitly present in the query. For example, the LLM is able to make ferences about the genre effectively based on the full query even when the user does not explicitly discuss the genre. Another benefit of the extractive clue is that the subqueries more grounded in the original query.
60
50
Recall
40
30-
20
Trade-off exists between generating subquery and individual retriever expert performance. Between the prompting strategies, we find that there is often a trade-off between how selective the query decomposer is with generating a subquery for a metadata field, and the effectiveness when gener- ated. For most of the metadata retrieval queries, the extractive prompting approach is slightly more ef- fective than predictive prompting on the examples that it does not predict "N/A" on.
Corpus size (multiples of original corpus)
Contriever k=5
Ours k=5
Contriever k=10
Ours k=10
Contriever k=20
Ours k=20
Contriever k=100
k=100
Figure 2: Recall at k of top performing baseline (Contriever) and our decomposition baseline for vary amounts of negatives documents added to the corpus.
5 Error Analysis
are much more effective when using decomposition compared to the image only CLIP baseline, as the image retriever model is able to access the visual part of the text
We sample 50 predictions where the top performing model where the model fails to get the book cor- reet the top and categorize the types of errors made by the model in Table 7. The most common kind of error is failure to model the relationship between the document and query, which happens in instances where there may be events that dense models fail to capture indicating that there is still headroom the task. Moreover, documents are sometimes brief or written in style that is not similar to other descriptions. Lastly, because users are recalling memories, some these can be false memories that lead to errors in retrieval.
Predictive prompting performs better than ex- tractive prompting. Overall, we find that pre- dicting subqueries is more effective than extract- ing. Subqueries thus, there more data for the specialized retriever experts to train on, compared extractive clues where some retrievers only have small fraction of the dataset for finetuning. Moreover, the predictive clues allows the LLM to make inferences based
generated for every query,

---

Error type
Query
Document
54%
Fail to Query-
think this was teen book. Don't remember Author or character names. remember is that the girl loses memory, it not Memoirs teenage amnessic"" It was fiction book and wasnčomic or manga. think the girl was involved a cur accident where someone her while she was walking? Apparently she was quite wild and broke instruments and was quite hated before she had the accident or something like that for things she done "to people, but she cant remember any of it.
Titl Kat Got Your Tongue' After a terrible accident, Kat wakes up with idea who she is, and no mem- ory of anything before
Doc Relation-
ship
crash. "doesnt even recognize her mum, much less her friends from school Poppy. Jade, and the mysterious Tina. Only af- ter she finds her old diary-written a voice no longer her own-docs Kat begin to discover the terrible secrets of her previous Thfs.This incredib After a terrible car accident,
28%
Document Rep- resentation In- sufficient
I read this book many years ago and it was an older book so I do not know the publication year. What remember is mother with two daughters that were young ladies possihly and their late teens carly twentics. The mother "favored the one daughter named Charlotte...
Tile The Daughters of Ardmore Hall Plot: An unbalanced woman seeks to destroy her daughters.
18%
Query Contains Error
Fiction, read in the 70s by mother who thinks it was probably written in that time period too. Book is about a dying woman remembering her child- hood during war in (probably South) Africa.Tite might have something do with dragon mosquito coil
Titl: Moon Tiger Plot: The elderly Claudia Hampton, a best-selling author of popular history: lies alone in a Lon- don hospital bed. Memories of her life still glow in her fading consciousness. but she imagines writing history...
Table Error types from the top performing model. Book representation simplified here without all metadata for space constraints.
6 Related Work
Wadden al., 2022). We direct readers Zhao et al. (2022) for a comprehensive, up-to-date survey of methods, tasks, and datasets.
Dense methods for document retrieval. Docu- ment retrieval has long history study in fields like machine learning, information retrieval, nat- ural language processing, library and information sciences, and others. Recent years has seen the rise in adoption of dense, neural network-based methods, such DPR (Karpukhin et al., 2020) and Contriever (Izacard al., 2022), which have been shown can outperform sparse methods like BM25 (Robertson and Walker, 1997; Robertson and Zaragoza, 2009) in retrieval settings in which query-document relevance cannot solely be deter- mined by lexical overlap. Researchers have stud- ied these models using large datasets of query- document pairs in web, Wikipedia, and scientific literature-scale retrieval settings (Campos et al., 2016; Sciavolino al., 2021: Roberts al., 2020). Many retrieval datasets have adopted particular task formats such as question answering (Kwiatkowski et al., 2019; Yang et al., 2018b; Tsatsaronis et al., 2015) or claim verification (Thorne et al., 2018;
Known-item and TOT retrieval. Tip of the tongue (TOT) is a form of known-item retrieval (Buckland, 1979; Lee al., 2006), long-studied area in the library and information sciences. Yet, lack of large-scale public datasets has made devel- opment of retrieval methods for this task difficult. Prior work on known-item retrieval focused on con- structing synthetic datasets (Azzopardi et al., 2007; Kim and Croft, 2009; Elsweiler et al., 201 1). For example, Hagen et al. (2015) released dataset of 2,755 query-item pairs from Yahoo! answers and injected query inaccuracies via hired annotators to simulate the phenomenon of false memories Hauff and Houben (2011); 1); Hauff et al. (2012), a common property of TOT settings.
The emergence of large, online communities for resolving TOT queries has enabled the curation of realistic datasets. Arguello et al. (2021) categorized the types of information referred to in TOT queries

---

from the website Remember This Movie. Most recently, Bhargav al. (2022) collected queries from the Tip Of The Tongue community on Red- dit and evaluated BM25 and DPR baselines. Our work expands on their work in a key way: We intro- duce a new method for retrieval inspired by long, complex TOT queries. In order to test our method on large dataset of TOT queries, we collected dataset of resolved TOT queries such that we also had access to metadata and book cover im- ages, which were not part of Bhargav et al. (2022)'s dataset.
Conclusion
We study tip of the tongue retrieval, a real-world information-sseeknng setting in which users issue long, complex queries for re-finding items despite being unable to articulate identifying details about those items. We introduce WhatsThaBookk, a large challenging dataset of real-world TOT queries for books. We also introduce simple but effective approach to handling these complex queries that
decomposes them into subqueries that are routed to expert retrievers for specialized scoring. Our simple framework allows for modular composition of different retrievers and leveraging of pretrained models for specific modalities such as CLIP for document images. We experiment with different subquery generation strategies and find that gen- crating predictions of document fields is more ef- fective. We observe improvements up to 6% abso- lute gain over sate-of-the-ar art dense retrievers for @ecall@5 when incorporating query decomposi- tion into existing retrievers.
Query Understanding and Decomposition. Our work understanding complex information- seeking queries by decomposition is related line of work breaking down language tasks into modular subtasks (Andreas et al., 2016). More recently, LLMs have been used for decomposing complex tasks such as multi-hop questions into a sequence of simpler subtasks (Khot et al., 2022) or smaller language steps handled by simpler models (Jhamtani et al., 2023).
Limitations
Related to decomposition of long, complex queries for retrieval is literature on document simi- larity (Mysore et al., 2022) or query-by -o-dcument (QBD) (Yang al., 2018a). In these works, common approach is decomposing documents into sub-passages (e.g. sentences) and performing re- trieval on those textual units. The key differentiator between these works and ours is that document sim- ilarity or QBD are inherently symmetric retrieval operations, whereas our setting requires designing approaches to handle asymmetry in available in- formation (and thus choice of modeling approach representation) between queries and documents. In this vein, one can also draw parallels to Lewis et al. (2021), which demonstrates that retrieving over model-generated question- -aniiionnn pairs in- stead of their originating documents can improve retrieval, likely due to improved query-document form alignment. In way, this is similar to our use of LLMs to generate clues that better align with extratextual document fields, though our work is focused on query-side decomposition rather than document-side enrichment. More recently, Wang et al. (2023) propose using LLMs for decomposing different facets of complex queries for scientific documents.
Given our proposed method is designed to handle TOT queries, there is an implicit assumption that the document collection covers the sought item with high probability. That is, a system tailored for TOT retrieval is unlikely to perform well if its in- dex is missing too many books. While our dataset is large-scale, one limitation that even a corpus of 187k documents does not cover the sought doc- ument sufficiently. Consider, for instance, the total number of books available on GoodReads (over 3.5B as of time of writing). Another limitation of our method is that the overhead for tackling an- other domain with this technique is non-trivial the prompts and few-shot examples may be directly transferable. We believe a promising av- enue for future work is reducing the effort needed to bootstrap the design and training of retrieval experts and incorporating them into a query decom- poser.
Acknowledgements
We thank the members of the Berkeley NLP group and the anonymous reviewers for their insightful feedback. This work was supported by a grant from DARPA SemaFFo, and gifts from sponsors of Sky Computing and BAIR. The content does not necessarily reflect the position of the sponsors, and no official endorsement should be inferred.
"ntns://eeememteettissoocc...o

---

References
Claudia Hauff, Matthias Hagen, Anna Beyer, and Benno Stein. 2012. Towards realistic known- moown-item topics for the clueweb. In Proceedings of the 4th Information Interaction in Context Symposium, IIIX '12, page 274-277, New York, NY, USA. Association for Com- puting Machinery.
Jacob Andreas. Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 39-48. IEEE Computer Society.
Claudia Hauff and Geert-Jan Houben. 2011. Cognitive processes in query generation. In Advances in In- formation Retrieval Theory, pages 176-187, Berlin, Heidelberg. Springer Berlin Heidelberg.
Jaime Arguello, Adam Ferguson, Emery Fine, Bhaskar Mitra, Hamed Zamani, and Fernando Diaz. 2021. Tip the tongue known-item retrieval: case study in movie identification. In Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, CHIIR 221, page 5-14, New York, NY,
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint wxxx:22122.01111
USA. Association for Computing Machinery.
Leif Azzopardi, Maarten de Rijke, and Krisztian Balog. 2007. Building simulated queries for known-item topics: An analysis using six european languages. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '07, page 455-462, New York, NY, USA. Association for Computing Machinery.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Samarth Bhargav, Georgios Sidiropoulos, and Evange- los Kanoulas. 2022. 'it's on the tip my tongue": new dataset for known-item retrieval. In Proceed- ings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22, page 48-56, New York, NY, USA. Association for Com- puting Machinery.
Harsh Jhamtani, Hao Fang, Patrick Xia. Eran Levy, Ja- cob Andreas, and Ben Van Durme. 2023. Natural language decomposition and interpretation of com- plex utterances. arXiv preprint 2.0Xx::23000867.
Christian Johnson. 2022. Binary encoded word mover's distance. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 167-172 Dublin, Ireland. Association for Computational Lin- guistics.
Michael K. Buckland. 1979. On types of search and the allocation of library resources. Journal of the American Society for Information Science, 333)):14333 147.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04900 04990.
Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading compre- hension dataset. ArXiv,
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39- 48.
David Elsweiler, David E. Losada, José C. Toucedo, and Ronald Fernandez, 201 I. Seeding simulated queries with user-study data for personal search eval- uation. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '1.. page 25-34 35-34, New York, NY, USA. Association for Computing Machin- ery.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 2843-2853, Dublin, Ireland. Association for Computational Lin- guistics.
Jinyoung Kim and W. Bruce Croft. 2009, Retrieval experiments using pseudo-desktop collections. In Proceedings of the 18th ACM Conference on Informa- rion and Knowledge Management, CIKM '09, page 1297-1306. New York, NY, USA. Association for Computing Machinery.
Matthias Hagen. Daniel Wägner, and Benno Stein. 2015, corpus of realistic known-item topics with associ- ated web pages in the clueweb09, In Advances in In- formation Retrieval, pages 513-525, Cham. Springer International Publishing.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.69800

---

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Epstein, Ilaa Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Trunsactions of the Association for Computational Linguisties, 7:453- 466.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Jin Ha Lee, Allen Renear, and Linda Smith. 2006. Known-item search: Variations on a concept. Pro- ceedings of the American Society for Information Science and Technology. 43(1):1-17.
James Thome. Andreas Vlachos. Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguisties.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Min- ervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sehastian Riedel. 2021. PAQ: 65 mil- lion probably- asked questions and what you can do with them. Transactions of the Association for Com- putational Linguistics, 9:1098-1115.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, loannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopou- los, Yannis Almirantis. John Pavlopoulos, Nico- las Baskiotis, Patrick Gallinari, Thierry Artières, Axel-Cyrille Ngonga Ngomo, Norman Heino, Éric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. 2015 20.5. An overview of the bioasq large-scale biomedical se- mantic indexing and question answering competition. BMC Bioinformatics, 16.
Sheshera Mysore, Arman Cohan, and Tom Hope. 2022. Multi-vector models with textual guidance for fine- grained scientific document similarity. In Procedd- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages Seattle, United States. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8763. PMLR.
David Wadden, Kyle Lo, Bailey Kuchl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022. Scifact-open: Towards open-domain scientific claim verification. In Findings of the Association for Computationall Linguistics: EMNLP 2022, pages 4719-4734, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics
Kirk Roberts. Tasmeer Alam, Steven Bedrick, Dina Demner-Fushmant Kyle Lo, Ian Soboroff, Ellen Voorhees, Lucy Lu Wang, and William R Hersh. 2020. TREC-COVID: rationale and structure of in- formation retrieval shared task for COVID-19. Jour- nal of the American Medical Informatics Association, 27(9):1431 1436-
Jianyou Wang, Kaicheng Wang, Xiaoyue Wang, Prud- hviraj Naidu, Leon Bergen, and Ramamohan Paturi. 2023. Doris-mae: Scientific document retrieval using multi-level aspect-based queries.
Eugene Yang, David D. Lewis, Ophir Frieder, David A. Grossman, and Roman Yurchak. 2018a. Retrieval and richness when querying by document. In Bien- nial Conference on Design of Experimental Search & Information Retrieval Systems.
S. E. Robertson and S. Walker. 1997. On relevance weights with little relevance information. In Proceed- ings of the 20th Annual International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, SIGIR '97, page 16-24, New York, NY, USA. Association for Computing Machinery.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018b. HotporQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Stephen Robertson and Hugo Zaragoza. 2009. The prob- abilistic relevance framework: BM25 and beyond. Now Publishers Inc.
Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lec, and Danqi Chen. 2021. Simple entity-centric ques- tions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 61 6138-6148, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji rong Wen. 2022. Dense text retrieval based on pretrained language models: A survey, ArXiv, abs/221 121221114445 1,14876.

---

A Prompts
Extractive Prompt
You are a utility built to take in forun quertes from users looking for books and output the aspact that is about the cover. If there not enough infermation, then output N/A, Do not guess and only output text that was in the query. Here are some examples:
Question: Hi there, read this book highscheol around 2802-2005. Fron what remenber, the main character is nicknamed "Mause" and she rides a big chestnut horse in junper shows. I think this book may have been Australian. The cover just showed chestnut horse and rider in nid Sump. think the title was one word--it may have been the nane of the horse.1 cannet remember the nane or the author of this book. I have googled everything I think of but I cannot to find this book and its driving crazy! I'd be grateful for any help on this! Thank you! Cover Clue: The cover just showed a chestnut harse and rider in nid jump.
Question: So read this book sonewhere between 2008-2018 when I wes in elementary school or niddle school.
It was about a girl who livad by the sea, and startad out with her having dinner at this boarding house sonething else that town) and she had to hide it. Then, the landlady of her house got nad when she found out, The NC went 't renember clearly but think there
lived in, And sonething happened (sha received fruit
banned in
down to the beach and saw this giant walrus thing (I gny who rode the walrus but naybe not) and the MC got nagical land. I don't 't renember much else, except the story had something to do with fruits for some reason,
also the walrus and they rode away to this
characters nother was likely from this nagical land. Also, in character brings color and happiness back to her seaside village and think there 's sonething else to do with strawbarries. Please help! I've been searching for this book for years! Thanks so much <3 Cover Clue: N/A
end, the
Question: remember reading this around 2008/2009. It was about girl being prettier than her mother, and when mother gets jealous she sends her daughter to a type of boarding school for people who are well known (Like nobles and princesses. She mey be reyelty). The cover was purple and ceptivating, which why picked
book to raad "*) with girl's face cover really had face on it, or if it was completely (viex spoiler][i believe that at the end, ninion of the girl's nather helped the girl get rid of her nother. Maybe it because the nather tried to kill her daughter for being beautiful, especislly since she was getting older. (hide spoiler)] Please help me find it. It was the first book that got ne to love reading
Houever,
the
that time. :) Caver clue: The cover was purple and captivating, (which was why I picked the book read "*) with girl 's face on it. However, I'm not sure if the cover really had a face on it, or if it was completely purple. question: probably read this book the late
early 2000s, and cannot find the title anywhere. The nain character is an outcast in her high school who lives in treiler park near a cenetery. From what I remember she
walking through the cenetery trying clear her mind when she wendered upon
funeral that had very few people at the funeral. After that she starts attending funerals on the weekend. She even has an cutfit she wears just to go to these funerals. tover Clue: N/A
Question: hi in looking
book there way two girls, the older one liked dumpster diving and had a duck shaped jar of nenories that she could go into. the younger one was blonde and liked buying shirts from thrift store and writing then, and she couldn renenber anyones faces. the younger to nail her non coconut and the asilman was the only face she could renember. there was also a wonan read couple
tried
in hospital who only said red shoes and the
green or blue.
ago sa probably 2017-20191sh? cover clus: the caver was green or blue.
Question: This book was blue (if that helps). And I think it had mailbox on the front of it. There were three girls
they hated their rival school. averyane send letters to another person in the schoal, one girl got a guy who wanted to impress a girl his school
to get
schools be friends they made
hin and the girl he liking each other :) The other girl likes playing "Ganes" with her letter guy... i dont really remember what happened with then. And the third girl had a jerk as her letter guy I dont really reneber what happened with hin either. If anyone knous the name of
sending letters went
*Practice Dates" And they ended
book I'n looking for would you please let know? :> Thanks su nuch :) :) :) Cover Clue: This book was blue (if that helps). And I think it had a mail the front of
Question: So do not renenber but the title, auther, or cover, but I renember a bit of the plot. believe it starts out with the nain characters set up on date. They dont know each others nanes, but their date consists of a sexy get to know you /photo shoot where theyre not allawed have sex but they do anyway. She gets pregnant but eant find hin after. Flash forward a bit, she works for her fathers large New York office and they nake deal to with with baby daddys office. He finds out she s pregnant and so the story goes. I hope this is enough to go on and soneone recognizes it. Thank you!
Clue: N/A
Now here is the example, remember not to add in additional information ts not in the question Question:: read this book back around 2088 (I think) can't renenber authar but read two of her works 1) is about this girl who falls in love with a guy who is sctuslly a dragon, he takes her back to his homeland and think he is injured, there are other dragons there and his reletives are also dragons author gives very vivid imagery I beliave there ware types of dragons like fire and ice ones... 2) the second book ( spoilers ahead) is about this girl who has sister named already engaged to some local village boy, but she falls in love with this cold mtan who is like "Ivy" and they escape together for a while but she returns and the girl also levas hin but he eludes har grasp, remember the end she talks about inagery on the wall roses ivy inter twining together... Any help nuch appreciated!!! :) Cover Clue:
(not toe
about nane) who is
Predictive Prompts
You are utility for guessing titles of bocks. Given the book described below, what is possible title for the book? Only return one predicted title without any extra text. Even if you 're unsure, try to cone with something, \nknDescription: ()

---

You are a utility for guessing author names. Guess possible author for the book described below. Don warry if you unsure. only return the name and a short explanation of why. wny.Annnnaseriptton.
You utility far categorizing books. Given the baok deseription below, generate several possible tags for the book. Try have diversity of coarse and five genres inlnDescription: ()
genres. Don't generate more than
You useful toal for generating ideas for art. Write sentences depicting what the book night look like based on the description below. Stick to only one Idea. Idea. wnw..nnneneeittoon.. (
You are a writing toal for ganorating 1daas. Write possible plat synopsis for book based on the description below. Don include the title author. AnVnDescriition:
Query Rewrite Prompt
are utility that user can helping users books. into book database to find the book. The query should forus on the These keywords book 's title, author, genre, year, ar distinguishing character or plot elements. Anknuser 's query: OX nüstabase Query:
user description below, generate query
will


Reviews:
[{"review_id": "KF6LspmmmN", "reviewer": "Reviewer_hexP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found the formulation of the problem to be interesting, and our approach of retrieval-by-decomposition to be novel. \n\nWe completely agree with the reviewer that a larger dataset of more books would be more realistic, and to this end, we have collected an additional 172,422 negative books (books that do not correspond to a gold query in our dataset) to make the setting more realistic. This has given us a total of 14,441 queries and 186,863 documents. With all documents added to the corpus, our contriever baseline achieves:\n\nRecall at 5 = 12.2 \n\nRecall at 10 = 16.1\n\nRecall at 20 = 20.1\n\nRecall at 100 = 33.4\n\nwhile our predicted clues and full decomposition gets:\n\nRecall at 5 = 18.6\n\nRecall at 10 = 23.4\n\nRecall at 20 = 28.6\n\nRecall at 100 = 43.5.\n\nIn comparing with our results in Table 4, observe: (1) All model performances are lower under the setting with more negative books, as the task is now more difficult showing that there is more headroom on the task, and (2) our approach is still outperforming the contriever baseline. If accepted, we will include these results in our extra page for the camera-ready and additionally release our collected set of negative books to support this type of experimentation.\n\nOn the topic of dataset size, we believe our dataset is still valuable. Indeed, there exist larger datasets like MSMarco or NaturalQuestions, but we also believe there is a need to complement these datasets with others that allows researchers to study retrieval systems under different types of queries or documents. In our case, that target phenomenon of interest are the long, complex queries that arise naturally from tip-of-the-tongue settings. Taking the popular BEIR benchmark datasets [1], for example, the only dataset that has comparable query length and complexity is ArguAna [2], which contains 1,406 queries and 8,674 documents, far smaller than our dataset. We hope this helps increase the reviewer’s confidence in our contributions to this research community.\n\n[1] https://arxiv.org/abs/2104.08663\n\n[2] https://aclanthology.org/P18-1023.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - A new problem is defined: retrieve book by user's long and complex description. A new dataset, WhatsThatBook, is built for this problem.\n - A new approach is proposed: retrieval-by-decomposition. The proposed approach is a novel application of large language models (LLM). The new approach outperforms a few strong baselines in the empirical evaluation.\n - The paper is easy to follow. Readers should be able to reproduce the results without a lot of difficulties. \n", "reasons_to_reject": "The number of documents (i.e., the book to find) is rather small in the constructed dataset (only about 14k documents in total). The experiments are conducted on these documents only. It is unknown if the results will generalize if there are more candidate books in the knowledge base, which is a more realistic setting. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper the authors propose a dataset for information retrieval, WhatsThatBook. The dataset focuses on searching the book using user's description as the query. Different from other IR datasets, the queries of WhatsThatBook are long (156 tokens per query on average), and the queries have low lexical overlap with the target documents. The authors build this dataset by scraping an online forum. Eventually the authors collected about 14k query-book pairs.\n\nTo solve this problem, the authors proposed a retrieval-by-decomposition approach. Specifically, each long query is firstly decomposed to sub-queries w.r.t. different fields (e.g., date, author, plot, etc.,), then each sub-query is issued to individual retrievers. The sub-query generation is done with GPT-3.5 and few-shot prompts. Finally the scores of different retrievers are aggregated. The authors compare the proposed retrieval approach with a few classical IR baselines, such as BM25, DPR and contriever. Results show that retrieval-by-decomposition is effective at handling such long and complex queries. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SKykrNWs0S", "reviewer": "Reviewer_Tnm2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer found the topic of TOT retrieval interesting and our dataset to be a valuable contribution for further study on this topic. \n\nThe reviewer expressed concern around our decomposition method. To reiterate, our contribution is our approach of using an LLM to decompose complex and lengthy TOT queries into subqueries that current retrievers can better handle. \n\nOur contribution is very much motivated by our observation that current models perform poorly on TOT queries, and our desire to better understand what makes TOT queries challenging. In collecting and studying this dataset, we find that they’re composed of various clues that each reference some specific attribute of the sought document. Based on this observation, we form the intuition that decomposition into more modular retrieval settings (i.e. sub-queries, specialized retrievers) may be a useful retrieval strategy. And we’ve empirically demonstrated (Table 4) an implementation of this intuition using LLMs that results in performance gains. Indeed, this specific implementation relies on availability of metadata, but we believe leveraging this structure is a reasonable first step when tackling a nascent, challenging problem.\n\nWe fully agree with the reviewer that top-down identification of metadata categories is only one way of implementing such a decomposition, and that exploring more sophisticated methods that can automatically induce decompositions when such metadata structure is not available is an interesting open research question. \n\nWe hope that our work demonstrating the potential for decomposition is a reasonable starting place for research in this exciting area. We hope our dataset will promote future work, especially that which tackles better ways of inducing the decomposition when metadata structure is not available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic of TOT retrieval is interesting and quite challenging.\n\nThe release of a real-world TOT dataset represents a plus.\n\n", "reasons_to_reject": "Despite the results, I am not completely convinced by the methodology. \n\nNo particular strategy is implemented for decomposition, which seems to happen just through the top-down identification of some metadata categories in order to extract the corresponding content from the queries. \n\nNo additional (e.g., linguistic or context) information or differences among the metadata categories are factored into the decomposition process or clue retrieval, which makes me question the scalability of the method.\n\nAnother concern regards the prompting, as I am not sure that the applied prompts are sufficient for the purpose. The lack of comparison with the results of non-decomposed queries and with other prompts including  linguistic, language-related, and/or categories aspects limits the result evaluation and the paper contribution to explaining LLMs.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors present a methodology for tip of the tongue (TOT) retrieval by decomposing long and complex queries. To this aim, they introduce WhatsThatBook, a large dataset of real-world TOT queries for books. \n\nExperiments based on extractive and predictive strategies are run. According to the presented results, incorporating query decomposition into existing retrievers shows an improvement of the state-of-the-art results gained by dense retrievers.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 290: improve → improves"}, "extra_text": {}}, {"review_id": "zrw28ZZ90o", "reviewer": "Reviewer_fUsA", "summary": "", "questions": "Question A: Why uses multiple recall? Is there any experiments showing that in TOT multiple recall is better than a unified multi-modal and multi-field model?\n\nQuestion B: Why \"generating entire hypothetical documents performs worse than just using the original document\" can prove \"query decomposition improves performance\"? I think only answer to Question A can prove this conclusion. What's more, generating hypothetical documents is obviously not a good idea since you can not finetune GPT3.5 for such generation.\n\nQuestion C: Comparing the results of Contriever, Contriever (E), Ours (E) and Ours (P) in Table 4, it seems that decompose and extracted clues does trivial improvements. The main contribution comes from the predictive clues. Does that means the performance mainly comes from the knowledge of LLM?\n\nQuestion D: If we set all the combination weights to be the same, what will the results be?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback. We are glad that the reviewer found our approach simple and novel, and our data analysis to be valuable. We address each of the reviewer’s concerns below.\n\nConcern: Omission of TOMT experimental results.\nResponse: We omitted the TOMT due to space constraints and we will add them in the final version of the paper in the additional space. In particular, the decompose-and-retrieve with predicted clues has the following results:\n\nRecall at 5 = 44.2\n\nRecall at 10 = 53.6\n\nRecall at 20 = 61.3\n\nRecall at 100 = 79.9\n\nCompared to the Contriever baseline:\n\nRecall at 5 = 42.8\n\nRecall at 10 = 51.1\n\nRecall at 20 = 58.6\n\nRecall at 100 = 77.7\n\nConcern: Choice of baseline retriever models.\nResponse: We clarify that we are proposing a framework, decompose-and-retrieve using LLMs for tackling complex queries, rather than a new retriever model. Thus, our primary goal is to show that widely-used and competitive retriever models function well as a component within this framework. As such, we believe we’ve chosen a reasonable set of models — BM25, CLIP, DPR, ColBERT, Contriever — and having found that Contriever is the best of them, used it as the basis within our decompose-then-retrieve framework. Recent work that incorporate retrievers also often use Contriever as the representative dense retriever [1] or show that Contriever still is the state-of-the-art in supervised adaptation settings [2]. \n\nConcern: Lack of unified/multimodal baseline models.\nResponse: We agree this is an interesting idea and believe it would be an exciting direction for future work. Our contribution comes from studying complex queries, identifying the intuition that they reference multiple document (multimodal) attributes, and operationalizing a decomposition framework that capitalizes on this observation. Of course, while our approach is not the only approach, we believe it is a reasonable one as a starting point of research for this relatively understudied retrieval setting.\n\nMultimodal models would be another such approach that capitalizes on our observations, but we believe it would not be as a baseline. For example, one would need to figure out how to handle cases in which the query does not reference the book’s cover or when the book’s cover is missing. Working through such considerations would likely be considered its own contribution rather than a baseline model, which we hope to see in future works. Finally, to reiterate, our contribution is that of a framework that leverages multiple retrievers, and not a new retriever model itself; we hope that future development of multimodal models could be used in conjunction with our framework, similar to how we are able to leverage CLIP.\n\n\nQuestion A: The reviewer asks why not use a single unified retriever instead of multiple retrievers? \nResponse: See above discussion around concerns of our baseline retrievers. We hope our clarification around our contribution as a framework and not an individual model helps address this.\nQuestion B: The reviewer questions the use of baselines like Contriever (H), which generate a hypothetical document for use in retrieval, as evidence in favor of our decomposition framework.\nResponse: Our framework is two-fold: (1) includes using LLMs to create the sub-queries, and (2)  individual training of specialized retrievers. \n\nOur baseline approaches Contriever (E) and Contriever (P), which concatenate generated sub-queries into one query for one retrieval operation, help us evaluate (1) without (2). Contriever (H), or generating entire hypothetical documents, is one strong baseline that helps us evaluate them, as recent work has shown that generating hypothetical documents can be very effective at improving on state-of-the-art retrievers, even zero-shot [3]. If Contriever (H) had been better, then it would have indicated our decomposed information wasn’t as useful as another powerful yet simple generative technique. But experiments in Table 4 verified that our decomposed information was indeed better.\nBased on the reviewer’s concerns though, we have added one more generative baseline — Use an LLM to generate a rewritten query that’s used as input to Contriever, rather than a hypothetical document. We call this Contriever (Q). If this works, then that would be another sign that explicit decomposition is not useful compared to another simple LLM approach. The results are:\n\nRecall at 5 = 18.5\n\nRecall at 10 = 24.2\n\nRecall at 20 = 31.8\n\nRecall at 100 = 52.9\n\nwhich are quite a bit lower than Contriever (H) and others. We hope this helps convince the reviewer that decomposition is indeed an important aspect of tackling these complex, TOT queries.\nQuestion C: The reviewer is wondering whether the close results of Contriever, Contriever (E), Ours (E), and much better results of Ours (P) indicates that performance is being driven by LLM knowledge rather than decomposition?\nResponse: Indeed, the difference between Extraction vs Prediction is one of our main findings. That is, purely representing the extracted clues as-is is insufficient, and that an LLM’s predictions (which make use of pretrained knowledge) are important for this task.\nThat being said, we still believe there is evidence that decomposition is valuable. First, Contriever (P), in which the original query is decomposed into predicted sub-queries which are concatenated as a single input into Contriever, outperforms models that don’t involve decomposition (Contriever). Second, as in our response to Question B above, baselines that involve providing explicitly decomposed queries — Contriever (P) — outperform baselines that don’t — Contriever (H) and the newly added Contriever (Q).\n\nQuestion D: The reviewer asks what the results would look like if the combination weights are set to the same.\nResponse: If we set the combination weights to be the same, we get:\n\nRecall at 5 = 19.7,\n\nRecall at 10 = 26.6\n\nRecall at 20 = 33.9\n\nRecall at 100 = 55.2\n\nWhich is much lower than the scores of the tuning the combination weights. This shows that tuning the combination weights is crucial, as the retrievers are trained independently and calibrated to each other.\n\nOverall, we thank the reviewer for their thoughtful comments. We will make use of the additional page to add these additional results and clarify more the nature of our contributed framework relative to other retrieval modeling advances. We hope this increases the reviewer’s confidence in our work.\n\n[1] https://arxiv.org/pdf/2305.14795.pdf\n\n[2] https://aclanthology.org/2022.emnlp-main.149.pdf\n\n[3] https://arxiv.org/abs/2212.10496", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Provides a new TOT dataset.\n\nBased on its related work (if comprehensive), it seems that this work is the first NLP solution that specializing for TOT problem.\n\nThe solution is simple and quite straightforward.\n\nDetailed analysis on the data.\n", "reasons_to_reject": "Lack of experiments\n    The author introduces two datasets, WhatsThatBook and Reddit-TOMT(Books), but only gives results on WhatsThatBook. In the paper only the individual retrieval results of Reddit-TOMT(Books) are shown.\n    Lack of ablation study. At least two ablation experiments should be conducted to verify the effectiveness of 1) LLM and 2) decompose, such as 1) replacing LLM with other query decompose methods or 2) use LLM to rewrite queries without decomposing. The author gives results of single expert on LLM decomposed queries (but concatenated), which is unclear for ablation.\n    Lack of baselines. It uses BM25, DPR and ColBERT as baselines besides single expert.\n        No model after 2021 are introduced as baselines.\n        What's more, they only compared with single-modal baselines. I think it would be more convincing to add a unified multi-modal baseline model to prove the effectiveness of decomposing.\n        The author treat all fields as text and concatenate them together for baselines, which seems to be unfair. It is more reasonable that the passage encoder in baseline models treat different fields as different kinds of features, such as making date and genre as id-like features.\n        Lack of reports on the accuracy of LLM extracting/generating clues\n\n\nDue to the lack of experiments and analysis described above, I'm not sure which design makes the real contribution to the effectiveness of this method, decompose or LLM?\n\nIf it's decompose, I'm not sure why bother using multiple experts and perform multiple recall and combine them together? Decomposing and Combination is not a method that generalizes well in many scenarios. In the paper the author uses a validation datasets to pick the best combination weights which is tricky and may lead to overfitting problem (if the distribution of validation datasets is shifted compared with test set, the results will be poor).\n\nIf it's LLM, then maybe we can just use LLM to rewrite the queries (a prompts such as \" You are a utility built to taking forum queries from users looking for books and output six kinds of clues, including dates, cover description, title description, plot summary, authors and genres.\"). Then this problem degenerates to a prompt engineering for query rewrite.", "questions_to_authors": "Question A: Why uses multiple recall? Is there any experiments showing that in TOT multiple recall is better than a unified multi-modal and multi-field model?\n\nQuestion B: Why \"generating entire hypothetical documents performs worse than just using the original document\" can prove \"query decomposition improves performance\"? I think only answer to Question A can prove this conclusion. What's more, generating hypothetical documents is obviously not a good idea since you can not finetune GPT3.5 for such generation.\n\nQuestion C: Comparing the results of Contriever, Contriever (E), Ours (E) and Ours (P) in Table 4, it seems that decompose and extracted clues does trivial improvements. The main contribution comes from the predictive clues. Does that means the performance mainly comes from the knowledge of LLM?\n\nQuestion D: If we set all the combination weights to be the same, what will the results be?\n", "paper_topic_and_main_contributions": "This paper tries to solve an interesting NLP task, known as tip of the tongue (TOT) retrieval, by utilizing LLM for query decomposition. Then it uses different specialized retrievers for decomposed queries and ensembles the results. TOT is a NLP application problem that's interesting and worth studying, and this paper proposes a easy and tricky method applying LLM for solving this problem. The main contribution of this paper contains two parts. It provides a new TOT dataset and proposes a simple yet effective method for solving it.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 172: useing -> using\n\nline 73: we describe our a simple -> we describe our simple\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is focused on developing resources and methods for complex retrieval tasks. Towards that end, they have created a new dataset which involves long text queries based on real users trying to remember the identity of particular books (e.g., describing the cover, along with vague details about the author or story). The paper also presents a new method for this task, based on decomposing the query, and compares against classical IR baselines.

Reviewers found this paper to be interesting, novel, and easy to follow. The new dataset was seen as potentially useful to the community. 

The two main concerns were that 1) the methodology was somewhat ad hoc, and left room for improvement, both in terms of rigor and sophistication, and 2) limitations with the evaluation, including the use of only a single dataset, lack of ablations, and lack of more recent baselines for comparison. In general, it seems like there is a concern that although the authors have demonstrated an effective method, it is unclear exactly why it works, or how well it will generalize.

In their rebuttal, the authors have included additional results, for both additional baselines and additional datasets. Although reviewers have acknowledged the rebuttal, they have unfortunately not specifically engaged with the new results provided by the authors. Given the difficulty of assessing out of context results provided in a rebuttal, this is not terribly surprising, and potentially reveals a problem with encouraging authors to include new experimental results in their rebuttal.

Overall, this seems like a case where this paper could benefit from an additional round of revisions before publication. However, all reviewers rated this paper as good or better on soundness, so they nevertheless felt it was worthy of being accepted, even if the paper's limitations tempered their excitement.
Decision: Accept-Findings