Scaling Sparse Fine-Tuning to Large Language Models
Authors: Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, Edoardo Ponti


Reviewer paper summaries: This paper focuses on scaling sparse finetuning to LLMs. Typically, sparse finetuning requires two stages, one for selecting a subset of LLM's parameters, and one for finetuning this subset. Existing sparse finetuning methods require a similar memory budget as full finetuning in the stage of selection. Instead, the proposed method, SpIEL-AG, merge the selection and finetuning stages in an iterative way. Specifically, the selection is random at the beginning, following by the finetuning. After some iterations, the updated parameters with less magnitude change will be reset to pretrained value (drop phase). The frozen parameters with larger gradient will be selected for the second round (growth phase). In this way, the required GPU memory is proportional to the number of trainable parameters instead of all LLM's parameters. For the growth phase, one need to keep track of the gradients for some candidates. To futher reducing this memory requirement, technique inspired by SM3 is applied for an alternative, SpIEL-MA.

The authors compare SpIEL to PEFTs, like (IA)$^3$ and LoRA, on Flan v2, GPT4-Alpaca and Tulu v2 with Llama2-7b and Llama2-13b. Overall, SpIEL-AG outperforms (IA)$^3$ and LoRA, and  SpIEL-MA is comparable to LoRA. Regarding to the finetuning time and GPU memory, SpIEL-AG, SpIEL-MA and LoRA are comparable, with SpIEL-AG the worst, and LoRA the best.
Summary of strengths: 1. The paper is well-structured, with clear motivation. 
2. The experiments are throrough, with popular benchmarks and LLMs, to validate the effectiveness of the proposed method.
Summary of weaknesses: 1. ***Lack of sparse finetuning baselines***. The paper only comares SpIEL to PEFTs, and excludes the sparse finetuning baselines. Because the sparse finetuning baselines, like DiffPruning, FISH Mask and LT-SFT, require the same budget as full finetuning for the selection stage (L501-504). However, there are some baselines that select the trainable parameters simply based on the pretrained weight magnitude, which share the same memory budget as the proposed method, like PaFi in [1]. According to [1], such a simple method is better than DiffPruning and FISH Mask. I believe the inclusion of some sparse finetuning baselines is necessary to validate the effectiveness of SpIEL.

2. ***Scalability and applicability of SpIEL is not promising***. According to Table 1, SpIEL-AG is better than LoRA on Llama2-7B for most tasks. However, LoRA becomes comparable to SpIEL-AG on Llama2-13B, which implies that LoRA might be more scalable. Due to the simplicity of LoRA, I doubt the applicability of SpIEL.

***Question***
1. How do you determine the sparsity of SpIEL? Based on L1021, you try to match the number of trainable parameters of SpIEL to LoRA. Do you take the memory for storing the indeces into account? Supposed LoRA's number of trainable parameters is x in FP16, SpIEL's number of trainable parameters should be about y, where y + y/2 = x if you use Int8 to store the indices.  


[1] Parameter-Efficient Fine-Tuning without Introducing New Latency, Baohao Liao, Yan Meng, Christof Monz
Comments, suggestions and typos: 1. The Method section should be improved. It seems the authors try to complicate the method.
Overall assessment scores: 2.5