Towards Understanding Domain Adapted Sentence Embeddings for Document Retrieval

Authors: Sujoy Roychowdhury, Sumit Soman, H. G. Ranjani, Vansh Chhabra, Neeraj Gunda, Shashank Gautam, Subhadip Bandyopadhyay, SAI KRISHNA BALA
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Sentence Embeddings, Question Answering, Technical Domains, Retrieval Augmented Generation, Embedding Models, Isotropy

Abstract:
A plethora of sentence embedding models makes it challenging to choose one, especially for technical domains rich with specialized vocabulary. In this work, we domain adapt embeddings using telecom, health and science datasets for question answering. We evaluate embeddings obtained from publicly available models and their domain-adapted variants, on both point retrieval accuracies, as well as their (95\%) confidence intervals.  We establish a systematic method to obtain thresholds for similarity scores for different embeddings. As expected, we observe that fine-tuning improves mean bootstrapped accuracies. We also observe that it results in tighter confidence intervals, which further improve when pre-training is preceded by fine-tuning. We introduce metrics which measure the distributional overlaps of top-$K$, correct and random document similarities with the question. Further, we show that these metrics are correlated with retrieval accuracy and similarity thresholds. Recent literature shows conflicting effects of isotropy on retrieval accuracies. Our experiments establish that the isotropy of embeddings (as measured by two independent state-of-the-art isotropy metric definitions) is poorly correlated with retrieval performance. We show that embeddings for domain-specific sentences have little overlap with those for domain-agnostic ones, and fine-tuning moves them further apart. Based on our results, we provide recommendations for use of our methodology and metrics by researchers and practitioners.

Decision: reject
Reviewer scores: 8, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730174354375

thread_type:
official_review

<thread_object>
creator: reviewer_ypbg

<content>
summary:
This paper investigates domain-adapted sentence embeddings for document retrieval. It proposes a systematic method to obtain thresholds for similarity scores for different embeddings, and proves that fine-tuning and pretraining-then-fine-tuning help improve retrieval accuracy through extensive experiments.

strengths:
- This paper proposes a systematic method to introduce thresholds to improve document retrieval performance.
- Extensive experiments are conducted to demonstrate that fine-tuning and pretraining-then-fine-tuning help improve retrieval accuracy.

weaknesses:
- Lack of novelty: Much existing work has shown that fine-tuning or continued pretraining helps to improve domain performance [1]. The isotropy problem has been well studied in [2][3] and many other contrastive learning-based sentence embeddings.
- There are too many research questions in this paper. It looks like the RQ2 and RQ3 don't have a strong connection to the research topic, i.e., domain-adapted sentence embeddings.
- Poor organization. The current introductory section functions as a literature review. It is suggested to clarify the research problem, research gap and contribution in the introduction.


**Reference:**

[1] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964.

[2] Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.

[3] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910. Association for Computational Linguistics.

questions:
- It is unclear why the CI is set to 95%. Is there any supporting evidence?
- The ndcg is widely used to evaluate information retrieval performance. Why not use it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581894673

thread_type:
official_review

<thread_object>
creator: reviewer_phjw

<content>
summary:
The paper presents an empirical study of domain adapted embedding models for QA. The authors evaluate a range of existing models and their domain-adapted variants on point accuracy and confidence intervals of retrieval tasks. The main contributions of this study are using the following aspects to provide a recommendation to select retrieval thresholds for domain adapted models: 

1. A methodology for determining appropriate similarity score thresholds for different embedding models.
2. Introduction of new metrics that capture the distributional overlap between top-ranked, correct, and randomly selected document with respect to the input question.
3. Analysis of the relationship between embedding isotropy (uniformity of vector lengths) and retrieval accuracy.
4. Observations that domain-adapted embeddings exhibit little overlap with their domain-agnostic counterparts.

strengths:
The authors define a relevant problem for RAG applications and show that fine-tuning can help to not only improve the retrieval performance but also increase the confidence intervals tightens. Thus, supporting that domain-adaptation is likely to also reduce the expected performance variance in that domain.

weaknesses:
While the problem addressed is relevant and some of the findings hold intrinsic interest, the paper also presents several notable limitations that should be addressed.

First, the confidence interval (CI) estimation techniques employed are direct applications of common methodologies to measure CI for a given metric (in this case accuracy). 

Second, the paper does not provide a compelling justification for the proposed assessment to select the retrieval threshold, nor does it rigorously compare their performance to simpler approaches such as hyper-parameter cross-validation which is relatively inexpensive assuming the similarity matrix is computed only once. The added value of the new techniques is therefore unclear from the current presentation.

Third, while the paper cites prior studies on the concept of isotropy, there is insufficient explanation of how the experimental setup and findings in this work relate to or build upon those earlier investigations. The connection to the broader academic context is not well established.

Finally, the overall narrative lacks cohesion, as the sections are not clearly tied together, and the suggested methodology for assessing retrieval thresholds is not systematically contrasted against relevant baseline approaches. This makes it difficult for the reader to fully grasp the unique contributions of the work.

questions:
The paper could benefit from a more cohesive narrative and clearer articulation of the benefits of the proposed assessment approach. While the common theme of domain adaptation links the various experiments, it is unclear what specific advantages the presented methodology offers.

To strengthen the paper, I would suggest addressing the following points:

Clearly define the end goal of the work. For example, if the aim is to provide guidance on setting appropriate retrieval thresholds, what is the current standard practice in this area? How does the proposed method improve upon existing approaches?

Explicitly highlight the key advantages of the new assessment technique. What performance gains or other benefits does it provide compared to baseline methods? 

Structure the paper to guide the reader seamlessly from the problem statement, to the methodological details, and finally to a clear conclusion about the merits and recommended use cases of the new assessment strategy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715394592

thread_type:
official_review

<thread_object>
creator: reviewer_8dar

<content>
summary:
This paper investigates different aspects of sentence encoders for domain-specific question-answer retrieval. It aims to address the limitations of cosine similarity in capturing true semantic similarity, especially for frequent words with homonyms or dependence on regularization. To improve retrieval evaluation, the authors use confidence intervals (CIs) over bootstrapped samples for models before and after pre-training and fine-tuning on specific domains. They introduce COE and ROE metrics to measure distributional overlaps between top-K, correct, and random document similarities with the query, showing these metrics correlate with accuracy and threshold values. The paper visualizes how domain adaptation shifts embeddings away from domain-agnostic spaces. It also shows isotropy scores have limited correlation with retrieval performance.

strengths:
1. The paper proposes novel metrics (CI, COE and ROE) over batched samples to evaluate embeddings for technical domains.
2. The methodology and metrics introduced are well-explained.

weaknesses:
1. The paper lacks specifics on the pre-training corpus for each domain. If models are pre-trained solely on training sets, this limits the utility of pre-training, affecting the reliability of conclusions. For instance, in the health domain, pre-training on a large corpus like PubMed abstracts is common to ensure domain knowledge.
2. Comparisons are primarily with pre-trained models from BAAI and OpenAI; adding more state-of-the-art domain-specific baselines could strengthen the evaluation.
3. While the work presents interesting findings, the novelty is limited. Observations like tighter CIs with fine-tuning are expected since task-specific fine-tuning generally increases confidence for a specific task while potentially reducing generalizability.

questions:
1. How does the test or dev set accuracy compare to bootstrapped accuracy? could you compare it with existing QA model results as a relaxed version of exact answer retrieval?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730857617836

thread_type:
official_review

<thread_object>
creator: reviewer_urnk

<content>
summary:
This paper reported mean bootstrapped retrieval accuracies along with confidence intervals for various SOTA embedding models with and without domain-adaptations. The results show that domain specific embedding show improved isotropy scores and move away from general domain embeddings. Overall, this is a solid study with reasonable technical contributions.

strengths:
1. The paper is generally understandable and clearly explains the technical parts to a certain extent.
2. The figures and charts in the manuscript are exceptionally clear and well-presented.

weaknesses:
1. This paper does not provide sufficient details on the ISOTROPY SCORES.
2. The paper does not sufficiently clarify the motivation behind its approach, especially regarding the "COMPUTATION OF THRESHOLDS". It lacks a detailed discussion on why existing methods struggle with this issue and how this paper effectively addresses it. it requires more illustrations and provements about this to help understanding the issue of existing methods and the motivation of the proposed new method in the paper.

questions:
1. This paper does not provide sufficient details on the ISOTROPY SCORES.
2. The paper does not sufficiently clarify the motivation behind its approach, especially regarding the "COMPUTATION OF THRESHOLDS". It lacks a detailed discussion on why existing methods struggle with this issue and how this paper effectively addresses it. it requires more illustrations and provements about this to help understanding the issue of existing methods and the motivation of the proposed new method in the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel evaluation setup for the retrieval problem that includes both retrieval accuracy and confidence intervals. The paper measures these metrics for a diverse range of embedding methods, including both domain-agnostic ones and their domain-specific fine-tuned counterparts. The findings demonstrate that domain-specific fine-tuning not only improves retrieval accuracy but also results in tighter confidence intervals. Additionally, the paper highlights that domain-specific models achieve improved isotropy scores and differentiate embeddings from their domain-agnostic counterparts.

Strengths
- Domain adaptation, the focus of this paper, is an important problem (PhJW).
- The paper introduces a series of new metrics that are valuable (8dAR, Ypbg).
- Figures and graphs are clear and effective (URNk).

Weaknesses
- Insufficient details about certain metrics and data (URNk, 8dAR).
- Limited explanation/motivation for the thresholding methods (URNk, PhJW).
- Models used in the experiments are restricted to BAAI/OpenAI models (8dAR).
- Lack of discussion on prior work (PhJW).
- Lack of coherence in the narrative of the paper makes it difficult to find the unique contribution of the paper (PhJW, Ypbg).
- Limited novelty, as the benefits of domain-specific fine-tuning are already widely studied (8dAR, Ypbg).