Title: IS IT NECESSARY TO APPLY PARAMETER-EFFICIENT FINE-TUNING TO ALL POSITION INDICES?
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: PEFT, LLM, Sparsity

Abstract: In the field of large models (LMs) for natural language processing (NLP) and computer vision (CV), Parameter-Efficient Fine-Tuning (PEFT) has emerged as a resource-efficient method that modifies a limited number of parameters while keeping the pretrained weights fixed. This paper investigates the traditional PEFT approach, which applies modifications to all position indices, and doubts its necessity. We introduce a new paradigm called PEFT\_S, in which a function S selectively applies PEFT modifications to a subset of position indices, potentially enhancing performance on downstream tasks. Our experimental results reveal that  the indiscriminate application of PEFT to all indices is not only superfluous, but may also be counterproductive. This study offers a fresh perspective on PEFT, advocating for a more targeted approach to modifications and providing a framework for future research to optimize the fine-tuning process for large models.