CAST: Sparse Fine-Tuning with Counterfactual Data Augmentation
Authors: Yuqi Qiu, Xiaoning Wang


Reviewer paper summaries: This paper introduces a sparse fine-tuning method, named CAST (Counterfactual Augmented Sparse Tuning), which leverages counterfactually augmented data for model parameter selection. CAST first identifies significant parameter changes by comparing models trained on counterfactual data with those trained on original data to obtain a binary mask. Then, the mask is used to guide the model in retaining influential parameters while eliminating updates to other parameters. The experimental results support these claims.
Summary of strengths: 1. This paper is well written and organized, easy to understand for readers. 
2. This paper introduces an interesting research problem that combines the advantages of counterfactual data augmentation and efficient fine-tuning.
Summary of weaknesses: 1. There is a significant gap between the research objectives and the proposed methods. The paper lacks necessary explanations and proofs of feasibility for the methods used. 
2. The experimental results presented are not reliable. For instance, Table 2 shows that the test accuracy of BERT trained on CAD is lower than the one reported in the original paper. Additionally, the authors report a 44.5% accuracy for BitFit on CAD in a binary classification task, which is below the expected 50% for random guessing. This also contradicts our experiments where BitFit achieved results comparable to full model fine-tuning. I am surprised by the notably lower results achieved by LoRA on the NLI task. 
3. Inappropriate training settings. Is it necessary to fine-tune Bert on simple sentiment classification for 100 epochs? Based on our experience, 20 epochs are often sufficient on the CAD dataset, and even 5 epochs can yield good results. The poor results in their experiments may be due to overtraining. 
4. The authors propose sparse fine-tuning, yet initially perform full-model fine-tuning on both the original and counterfactual datasets. Existing baselines achieve similar results by fine-tuning once with a mixture of both datasets. This approach shows no clear advantage over existing methods. 
5. The scalability of sorting parameters after equation (1) is questionable. Sorting typically requires nlog⁡(n) time complexity. It's unclear if this approach can efficiently handle larger models with millions or billions of parameters.
6. As a general efficient fine-tuning approach, more transformer-based models should be reported beyond BERT. BERT is relatively smaller, with only 110M parameters. More results on RoBERTa, DeBERTa, GPT-, etc. are expected. 
7.  It is questionable to apply the best hyperparameters, specifically alpha and K, optimized for the sentiment classification task directly to the Natural Language Inference (NLI) task. Based on my experience, hyperparameters should be task-specific and require independent optimization.
Comments, suggestions and typos: 1. Missing space before \cite{}
2. Figure 2, Table 2, and Table 3 present results without specifying the test datasets used. This omission limits the clarity and reproducibility of the findings
3. Missing references for baselines in the experiment section.
Overall assessment scores: 1