Loius (Look it up in the Structure): Benchmark and Techniques for Document structure aware LLM based Retrieval

Authors: Vineet Kumar, vishwajeet kumar, Jaydeep Sen, Riyaz Ahmad Bhat, Sachindra Joshi
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: information retrieval, llm, model based retrieval, document search, retrieval benchmark, document structure, benchmark

Abstract:
We thank the reviewers for their valuable feedback. We have decided to withdraw the submission
from ICLR after careful consideration.

Decision: reject
Reviewer scores: 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730100421843

thread_type:
official_review

<thread_object>
creator: reviewer_yryc

<content>
summary:
The research paper introduces Loius, a novel retrieval system that leverages the structure of ToC to enhance retrieval processes. By mimicking how humans intuitively use a ToC to navigate books, Loius proposes a fresh retrieval paradigm that contrasts with traditional methods based on keyword or vector-based searches. The paper also presents a new benchmark dataset ToCTome, which comprises 18 books across six diverse domains, providing a robust platform for evaluating ToC-based retrieval systems. The results demonstrate Loius’s superior performance in accurately locating relevant sections, boasting a Recall@1 score significantly higher than the next best system (DSI) and other baseline models like BM25 and DPR.

strengths:
- The innovative use of the ToC to guide the retrieval process introduces a unique approach to document navigation, potentially transforming retrieval strategies for structured documents.
- Extensive experiments validate the system’s efficacy, particularly its ability to outperform traditional retrieval methods.
- The creation of the ToCTome benchmark contributes a valuable resource to the research community, fostering further innovation in retrieval systems that understand document structure.

weaknesses:
- The specific application of ToC-based retrieval to books may not generalize to other types of content that lack a clear hierarchical structure, such as unstructured web pages or documents without a ToC.
- The technical novelty appears limited primarily to the adaptation of existing retrieval technologies to a new input structure (ToC), potentially limiting the method’s broader applicative insights.
- In Section 5.2, I find that the maximum input length of Loius and DSI are significantly different. It is unclear if this difference may affect the experimental results.

questions:
1. Can the methods developed for Loius be adapted for general retrieval tasks beyond structured documents like books? If so, what modifications would be necessary?
2. Why were different maximum input lengths used for Loius and DSI in your experiments? Could this difference affect the comparative performance results reported?
3. Given the reliance on a structured ToC, how does Loius handle content variations within sections that might not be accurately reflected by their ToC descriptions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730347787383

thread_type:
official_review

<thread_object>
creator: reviewer_nrhm

<content>
summary:
Inspired by the way humans often search through the Table of Contents when reading books, this paper introduces a new task: given a query and the Table of Contents (TOC) of a long document, the objective is to retrieve the correct subsection in the TOC that contains evidence to answer the query. The authors introduce and release a new multi-domain dataset, ToCTome, which consists of 18 books across 6 domains. For each subsection, they use the Mixtral 8x7b model to generate questions, forming subsection-query pairs. Additionally, they split the data into training, development, and test sets, and fine-tune Mistral Instruct v0.2 with the LoRA adapter. Experimental results demonstrate that the fine-tuned Mistral Instruct v0.2 achieves an impressive R@1 score of 82.6%, outperforming BM25, dense retrieval models, and the original, non-fine-tuned Mistral Instruct v0.2 on this task. These findings highlight the strong capabilities of LLM in ToC-based retrieval.

strengths:
1. The paper introduces a new dataset containing queries and their corresponding subsections, which may contribute to research in document retrieval.
2. Constructing the dataset and conducting the experiments required substantial effort.

weaknesses:
1. The authors introduce a new problem but do not explain its importance and significance. The purpose of using a large language model (LLM) to locate the subsection relevant to the query is unclear. The authors simply state that this "mimics how humans leverage the structure of a book," but they don’t explain why this is necessary. I believe the authors should address this question: "After finding the subsection relevant to the query, what can be achieved?"
2. The authors assume that each query can be matched to a corresponding subsection in the Table of Contents. However, in real-world scenarios, many queries may not align with the Table of Contents; in particular, some fine-grained questions cannot be mapped to a specific subsection.
3. The proposed dataset and method are designed for queries that correspond to a single subsection. However, some queries may span multiple subsections, which this dataset and method do not account for. This results in oversimplified problem modeling.
4. The writing of this paper has several issues: (1) Many sentences do not follow an academic tone, and exclamation marks are used frequently, which is uncommon in academic papers. (2) Detailed experimental results are repeated in the ABSTRACT, INTRODUCTION, and EXPERIMENTS sections.

questions:
The questions are listed in the 'Weaknesses' section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730484387055

thread_type:
official_review

<thread_object>
creator: reviewer_nogp

<content>
summary:
This paper proposes a new task for retrieval models: that of finding the relevant section title in a book, given a question. They build a dataset for this task by using PDF parsing to extract the table of contents from books. They generate question for the task through the use of Mixtral. 

They then train a model to do well on this task, called Louis.  Their model is trained on the training set (also generated with Mixtral) and shows improved performance over other model types, including non-fine-tuned versions of DSI, BM25, and Mistral. They show an error analysis where their model performs better.

strengths:
- The new proposed task is an interesting contribution, and to my knowledge has not been studied before
- They use a wide variety of retrieval model architectures, from cross-encoders, to generative retrieval, spare retrieval, and dense retrieval.
- Their proposed model does better on the data they generated

weaknesses:
1. There is a lack of important details. How many chunks/passages are there in the corpus for dense retrieval (or BM25)? Does each chunk for these models also get the section, so that the model can use that information? Why are there no long context models evaluations (Gemini/Prolong/even just a 128k Llama)? Does Mistral see the entire book or just the ToC? etc.
2. Conceptually, performing ToC retrieval is strictly easier than searching for the specific passage that a question was generated from. From related work on searching for the relevant passage in long documents (see the "Scrolls Benchmark" [1] for task examples) IR models perform pretty well with retrieval (see the LoCo benchmark [2]). It seems something is wrong with the setup if the models are doing this poorly. My guess is that the book is chunked but not given the section headers in the chunks, so they lack that context. Furthermore, BGE is a decent baseline, but pretty weak in comparison - see the MTEB leaderboard where BGE-base (I assume because of the 768 dim vector, but again not described) is ranked #47.
3. Overall, the task does not appear very challenging.  R@3 is 90% and close to that for many baseline models (DSI, BM25). If BM25 can get nearly the same performance as the proposal model, it does not seem like much of an improvement (and is perhaps not even statistically significant).
4. There is no analysis on the false negative information in these questions. It is likely that many questions are answered in several places and that falsely lowers the score. I would guess that if the authors did an analysis of the Recall @ 3 failures that the answer is given in both places.  This makes (3) more of an issue and lowers the quality of the dataset.
5. The main modeling contribution seems to be that using the same data generation process as making the test set, and then by training on it, makes a model better able to perform this task. This is unsurprising and demonstrated over and over in the literature on every modality. This would be totally fine, if this was the only issue, but I assume that this means that the proposed Louis model cannot do another other tasks, as it is likely overfit for this. 

Overall, I think the task could be useful for the retrieval community as a useful long-context evaluation. However, I am unconvinced of the datasets quality and there appear to be modeling concerns (or at least a lack of information).

[1] Scrolls Benchmark: https://www.scrolls-benchmark.com/tasks

[2] LoCo benchmark: https://arxiv.org/pdf/2402.07440v2

questions:
Most questions are in the weaknesses. But some comments about writing:
- Citations should be \citep{}, it seems like they are not, and are instead inline
- [Minor] It would be nice to see some examples in the appendix, in terms of passage it came from and question generated.
</content>
</thread_object>

</thread>


Metareview: