Exploring Efficient Adaptation of Large Language Models for Sentence Embedding with Prompt-behind Tuning
Authors: Kaiyan Zhao, Qiyu Wu, Zhongtao Miao, Yoshimasa Tsuruoka


Reviewer paper summaries: This paper propose a prefix-tuning strategy, which utilize soft-prompt (several pseudo prefix tokens)to fine-tuning LLMs for sentence embeddings. Empirical investigation identify that tuning only a few tokens can achieve a competitive performance with that of fine-tuning with LoRA.
Summary of strengths: 1. Empirically investigate that P-tuning are effective for supervised contrastive learning sentence embeddings for LLMs.
Summary of weaknesses: 1. Contributions are quite questionable. From my understanding, the main contribution should be adaptation of p-tuning for LLMs on sentence embeddings.

2. Lack of Novelty. The proposed strategy is a simple adaptation of prefix-tuning on LLMs under contrastive learning sentence embedding settings.

3. This paper provides no in-depth analysis or insights for LLMs sentence embeddings. The paper are not informative.
Comments, suggestions and typos: See Weakness
Overall assessment scores: 2