Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs
Authors: Zheng Zhang, Fan Yang, Ziyan Jiang, Zheng Chen, Zhengyang Zhao, Chengyuan Ma, Liang Zhao, Yang Liu


Reviewer paper summaries: This paper addresses the issue of positional bias in LLMs. The study identifies that LLMs have inherent positional preferences, and designs experiments to show that prompt-based solutions alone cannot overcome. To mitigate this, the authors propose a Position-Aware Parameter Efficient Fine-Tuning (PAPEFT) approach. PAPEFT includes (1) a data augmentation technique to permute the order of input contexts and (b) an adapter module that integrates positional context into the LLM's input. The experiments demonstrate that PAPEFT can help mitigate positional bias and improve performance on benchmarks including recommendation, question answering, and link prediction.
Summary of strengths: - This is paper empirically demonstrate that prompt-based solutions such as "Few-shot learning" and "hierarchical inference" proposed in section 4 cannot address the positional bias.
-The proposed method achieves notable improvements in recommendation and link prediction tasks compared to the "original" setting.
Summary of weaknesses: 1. It is not well supported to conclude the "underlying causes of positional bias in LLMs" as "positional bias is primarily due to the model’s **preferred position**". Section 4 is more like an empirical study on downstream tasks rather than in-depth discussion and analysis on the "underlying causes".
2. The framework design is not very intuitive to me: The soft position tokens only encode the document-level position information, and these tokens do not incorporate any information inside the document. Please include more analysis and discussion on why and how the soft position tokens work for reducing positional bias, especially considering that the numbers reported for recommendation and link prediction in Table 3 appear to be super low while incorporating soft position token brings huge performance gain.
3. It is unclear how PAPEFT will perform with contexts containing varying numbers of documents. Since all benchmarks used for evaluation have $K=20$ documents, the generalization ability of PAPEFT might be limited.
4. Question Answering is one of the most typical scenarios for LLM application. However, Table 3 shows that incorporating PAPEFT actually hurts the model's performance.
Comments, suggestions and typos: Please see "weaknesses".
Overall assessment scores: 2.5