Multi-Field Adaptive Retrieval

Authors: Millicent Li, Tongfei Chen, Benjamin Van Durme, Patrick Xia
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: information retrieval, hybrid retrievers, semi-structured data

Abstract:
Document retrieval for tasks such as search and retrieval-augmented generation typically involves datasets that are _unstructured_: free-form text without explicit internal structure in each document. However, documents can have some structure, containing fields such as an article title, a message body, or an HTML header. To address this gap, we introduce Multi-Field Adaptive Retrieval (mFAR), a flexible framework that accommodates any number and any type of document indices on _semi-structured_ data. Our framework consists of two main steps: (1) the decomposition of an existing document into fields, each indexed independently through dense and lexical methods, and (2) learning a model which adaptively predicts the importance of a field by conditioning on the document query, allowing on-the-fly weighting of the most likely field(s). We find that our approach allows for the optimized use of dense versus lexical representations across field types, significantly improves in document ranking over a number of existing retrievers, and achieves state-of-the-art performance for multi-field structured data.

Decision: accept (spotlight)
Reviewer scores: 5, 8, 8, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730626244085

thread_type:
official_review

<thread_object>
creator: reviewer_hpub

<content>
summary:
This paper introduces a method called Multi-Field Adaptive Retrieval (MFAR), aimed at enhancing retrieval of structured or multi-field documents. The approach decomposes each document into multiple fields, each independently indexed and retrievable through different models. Each field is associated with a learnable embedding, and an adaptive function is trained to predict the importance of each field based on the query and scoring model. The authors validate MFAR's effectiveness through experiments across multiple datasets, comparing it against prior methods.

strengths:
- S1: This study is the first to demonstrate that multi-field ranking outperforms single-field ranking within the context of dense retrieval. 
- S2: The experiments highlight that hybrid models enhance both single- and multi-field rankings, with certain multi-field scenarios benefitting more from hybrid approaches.
- S3: A thorough ablation study demonstrates the significance of the query-conditioning mechanism in optimizing field weights. Additionally, the qualitative analysis shows the variability in optimal scorers across datasets, showing that there is no universally best approach.
- S4: A detailed case study further illustrates the method’s effectiveness in practical applications.

weaknesses:
- W1: Although the paper focuses on multi-field ranking, it does not include classic methods such as BM25F [1], its later extensions [2,3], or the mixture of language models [5], which are commonly applied in table retrieval [6], as part of the baselines. Incorporating at least BM25F or the mixture of language models would add valuable context and enable a more thorough comparison.
- W2: Query-dependent field weighting has been previously explored, such as in table retrieval methods that incorporate both query-dependent and query-independent features [4]. Testing on table retrieval datasets could offer additional insights, as tables represent another structured, multi-field document type.
- W3: The proposed method adaptively determines the importance of each field given a query and scorer; however, it does not select among scorers, instead requiring calculation of all scoring potentials, thereby increasing computational load.


[1] Simple BM25 extension to multiple weighted fields, 2004

[2] Field-Weighted XML Retrieval Based on BM25, 2006

[3] Extending BM25 with Multiple Query Operators, 2012

[4] Web Table Retrieval using Multimodal Deep Learning, 2020

[5] Combining Document Representations for Known-Item Search, SIGIR 2003

[6] Ad Hoc Table Retrieval using Semantic Similarity, WWW 2018

questions:
n/a
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664911147

thread_type:
official_review

<thread_object>
creator: reviewer_vqrj

<content>
summary:
This paper presents Multi-Field Adaptive Retrieval (MFAR), a framework for retrieving structured documents by dynamically weighting document fields based on query relevance. By combining lexical and dense retrieval methods, MFAR improves retrieval accuracy on the STaRK dataset, surpassing state-of-the-art baselines and highlighting the potential of adaptive, hybrid approaches for structured data retrieval.

strengths:
1. The paper proposes a new approach that incorporates structured document retrieval, where documents contain multi-field information. The proposed method adaptively controls the information by using a weight adjustment strategy based on the query. 
2. The experimental results demonstrate that mFAR framework over various baselines on the STaRK dataset. 
3. The authors present detailed analysis demonstrating why their hybrid approach can outperform baselines through the experiments of both multi-field vs. single-field and hybrid vs. lexical/semantic similarity. 
4. The paper is well-structured, motivated, and written, thus easy to follow.

weaknesses:
1. Though several recent retrievers [1] and [2] have been proposed, Contriever is used as a representative of dense retrievers without sufficient discussion. I am concerned that the conclusion may change if the authors use the recent retrievers. Especially, since FollowIR is instruction-tuned, it may be more robust against negation in the query, which the authors pointed out as a possible issue of the single-field method. 
2. The adaptive weighting mechanism in MFAR relies on training within specific domains and structured datasets, making it potentially sensitive to domain-specific data characteristics. This might lead to suboptimal field selection or scoring when the document structure is inconsistent across the corpus.  

[1] Weller, Orion, et al. "FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions." arXiv preprint arXiv:2403.15246 (2024).

[2] Xiao, Shitao, et al. "C-pack: Packaged resources to advance general chinese embedding." *arXiv preprint arXiv:2309.07597* (2023).

questions:
Regarding Weakness 1 in the previous section, did you try to use other dense retrievers and do you have any insights about experiments with better retrievers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690224459

thread_type:
official_review

<thread_object>
creator: reviewer_xhpk

<content>
summary:
This is a timely "revisit" of a multi-field retrieval problem with a proposal of multi-field adaptive retrieval that combines sparse-lexical and learned-dense-vector-similarity scores using field-weights, which are learnable and adaptive. Namely, each weight is a neural network incorporating a query representation and a field-specified learned weight vector.

The paper is very nicely written, it has a convincing set of results, as well as thorough literature review.

strengths:
* The paper revisits an important problem.
* The paper is very well written
* Experimental results are convincing and have quite a few baselines.
* The method is simple and elegant yet effective

weaknesses:
The only noticeable weakness is the need to compare to a better multi-field BM25 baseline, ideally, where field weights are learned using coordinate ascent (e.g., usling RankLib, or FastRank: https://github.com/jjfiv/fastrank). I do not think, however, that this should be a deal breaker. However, we encourage authors to list this as a limitation of their work.

questions:
The following questions were fully answered/addressed by the authors:
1. What is exactly used for dense/vector retrieval?
2. How did you decide on selection of top-100 records for each field (L885)?
3. In Eq. 4, what is exactly q in G(q, f, m): I assume it should be a dense query encoder, but it's not clear from the text (at least for me).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704934748

thread_type:
official_review

<thread_object>
creator: reviewer_jpgq

<content>
summary:
This paper exploits the structure in various IR corpora by learning query, corpus, field dependent weights across vector retrieval and token retrieval. The paper compares the methods to a number of strong baselines and analyzes a thorough set of ablation experiments to identify difference makers across different benchmarks. This paper is well written, easy to read and has a comprehensive set of references.

strengths:
The paper is well motivated and well written. Many documents naturally have structure. Exploiting it should lead to better retrieval quality. However, doing so depends on the query, corpus and kind of retrieval. This paper comes up with an elegant and intuitive formulation to learn all of these weights during training. The baselines are well chosen, ablation experiments are extensive and the references are comprehensive.

weaknesses:
The main weakness, and kudos to the authors for discussing this, is what they mention in Section 4 "Multi-field vs Single-field": "A side-by-side comparison of the single field models against their multi-field counterparts shows mixed results".  The difference in averages between mFAR_2 and mFAR_all doesn't appear statistically significant. The primary gains (looking at mFAR_2 vs mFAR_all) seem to be in the prime data set which has a lot of fields. Should the paper instead focus on adaptive hybrid retrieval, and choose hybrid retrieval baselines?

questions:
1. Are "Dense only" in Table 4 and "mFAR_dense" in Table 2 the same (the numbers are close but different). Were mFAR_dense and mFAR_lexical trained separately or trained together and one component ablated? 

2. See the weakness section, which I phrased as a question.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730800886513

thread_type:
official_review

<thread_object>
creator: reviewer_ejws

<content>
summary:
The paper presents a framework to improve document retrieval (Multi-Field Adaptive Retrieval, MFAR). Whereas document retrieval typically uses unstructured data for tasks such as retrieval augmented generation, MFAR is designed for structured data where documents have multiple fields (e.g., titles, abstracts, author information). MFAR first decomposes documents into individual fields and then learns a model that adaptively weights fields based on the input query. The framework uses both dense and lexical methods for retrieval, optimizing the combination of these representations per field to improve retrieval performance. Experimens show that MFAR outperforms previous retrieval models in complex document retrieval on the STaRK dataset.

strengths:
Rereach on structured document retrieval is highly relevant, especially for RAG approaches. The retrieval is well designed, using a hybrid and adpative query scoring mechanism, using both dense and lexical methods as well as a ranking strategy. The evaluation is thorough, and the paper is well-structured and generally well-written.

weaknesses:
The fine-tuning approach makes the approach specific to a set of fields from a dataset. Information overlap in fields (see lines 416-424) might intrudice some redundancy to the retrieval process.

questions:
How much robust is the framework to variations in the number of fields, e.g., regarding field information overlap?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730906716599

thread_type:
official_review

<thread_object>
creator: reviewer_znrt

<content>
summary:
The paper is concerned with the retrieval of documents which are multi-field, i.e, composed of multiple attributes such as title, authors and content. It proposes a method called MFAR which combines the use of different scoring methods, both lexical (word-based) and dense (embedding-based), for each field, allowing the model to adaptively predict the importance of each field for a given query. 

The authors conduct experiments on three datasets (product reviews, scientific papers and biomedical studies) that demonstrate that MFAR outperforms existing retrieval methods, achieving state-of-the-art results in structured data information retrieval. The study explores the benefits of the multi-field approach and the hybrid use of scoring methods to improve retrieval accuracy, showing that, instead of choosing between dense or lexical-based scorers alone, one can combine them in a hybrid fashion. An adaptive weighting technique is provided to combine those scores given a query.

strengths:
The paper tackles a relevant problem, although the proposal has some limitations discussed below. The paper is well written and easy to understand. Originality is rather limited, since the semi-structured retrieval has been researched for a long time. The significance of the results is also limited because the approach is rather simple, and the experimental evaluation focuses on a recently published benchmark. The paper quality is good to its purposes, despite some adhoc design decisions.

weaknesses:
It is not clear to me whether the fields may be considered indepent, so that the summation of the field-related scores suffices for determining the overall score. That is, it seems intuitive that there are correlations among the field instances and they may bias the result, as was extensively researched in the information retrieval area.

Another field-related issue is regarding the process of selecting the fields that will be considered in the whole process.

Overall, the proposal is simple and basically consists of combining scoring functions associated with fields, without considering their correlations and other characteristics that may either characterize the task or explain problems or failures. For instance, although the paper focuses on the information carried by the fields, it seems intuitive to mix the aggregated value of the fields with the remaining text, exploiting eventual information there. 

The experimental result also needs to be improved, as detailed next. First of all, the two experimental hypothesis seem to be too simple, thus quite easy to demonstrate. The advantages of using document structure are expected, in particular considering the additional information given to the models. The expected gains of hybrid approaches are also quite predictable. In both cases, it would be interesting to somehow derive upper bounds on the gains, so that the results go beyond benchmark-based evidence. 

On the other hand, it is also necessary to clear outline the limitations of such hybrid approaches and how they may be addressed. I really missed confidence intervals or similar statistical significance metrics on the results. For instance, on Table 2, the results may be too close and, despite the bold highlight, it is not clear how far the results are from the remaining of the baselines.

One terminology issue is regarding the difference between structured and semi-structured and where the paper fits. It seems to me that semi-structured is the proper jargon.

questions:
Are the scoring dimensions really orthogonal, enabling summation as the summarization metric?

Aren´t there additional baselines and data sets that may be used in the experiments?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a framework for document retrieval for structured documents that may have multiple fields like title, abstract, etc. The contribution works on individual fields and adaptively aggregates the results so that different queries lead to different weightings of the fields. The paper is interesting and addresses an important problem, is well written and proposes an intuitive and rather straight forward but elegant technique that also yields convincing empirical results. The contribution was well received by the reviewers and should be of interest to the community as well.