From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers
Authors: Bharat Runwal, Tejaswini Pedapati, Pin-Yu Chen


Reviewer paper summaries: This paper focuses on finetuning LLMs to make their activation more sparse. Notably, only the activation after the first feed-forward layer become sparse. During inference, the sparse activation can save energy on some modern hardware accelerations because of the zero-skip operations. 

Specifically, to achive a sparse activation, the authors:
1. Introduce two regularization terms to the loss function to encourage the sparse activations, one is for ReLU-based LLM and one is for GeLU-based LLM. Both terms are differentiable and inspired by other sparcity works;
2. Finetune an LLM with these one of these two terms together with existing PEFT method to save parameter and memory;
3. Also design an adaptive method by introducing a trainalble parameter to each MLP block in different layers because of the non-uniform layerwise weight sparcity, similar to layer-drop technique, to save runtime during inference.

The authors conduct experiments with RoBERTa and Flan-T5 on GLUE and SQuAD, respectively, to demonstrate the effectiveness of the proposed method, DEFT.  In sum, DEFT's score on downstream tasks are comparable to PEFT's, while having much sparser activations.
Summary of strengths: 1. The paper is easy to understand and well-structured.
2. The motivation is clear, the experiments are thorough, and the measure metric is well-designed. 
3. The experimental results show the effectiveness of the proposed method, DEFT.
Summary of weaknesses: 1. ***The experimental LLMs and tasks are out-dated***. I understand GLUE and SQuAD are two very standard tasks for PEFT. Except for these two tasks, I expect more challenging and new tasks, like mathematical/commonsens reasoning. In addtion, more recent LLMs, like Llama, are also expected.

2. ***The proposed method is more like an incremental work***. In the paper summary, I summarize three key designs of these paper: (1) two regularization terms; (2) combined with existing PEFTs; (3) layer-drop like adaptive method. All three components are quite well-known.

3. ***Limited improvement***. Although a sparser activation is obtained (45% sparser), the saved energy is not that expected (saving 8%). This is because only the activation after the first feed-forward layer becomes sparse. In addtion, we can only save energy on some specified hardware. The hardware-related implementation is missed.

4. ***The presentation need to be improved***. Figure 1 is an overview, deserving more space. Figure 2 and 3 don't have a lable for x-axis. Table 3 and 4 are too small.

5. ***Question***: What is the threshold you set for GeLU-based LLM to measure the sparsity?
Comments, suggestions and typos: see Weakness
Overall assessment scores: 2