Venue: NeurIPS 2022 Workshop TSRML

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary: This paper studies a differentially private version of bias-term fine-tuning (BiTFiT) for language and computer vision task. BiTFiT is a parameter efficient method that updates only bias term’s gradient during the fine-tuning process. The paper follows a similar concept with DP-SGD for optimizer to make the bias term private: first aggregate bias gradient norms across all layers, then use it to compute clipping factor, add Gaussian noise to the sum of clipped gradients, and descend on bias term. The privacy analysis is the same as DP-SGD. The proposed method is tested on different tasks and dataset and shows significant improvement in training efficiency for fine-tuning large models while still attaining comparable performance. 
strengths: - The paper is the first to perform a deferentially private version of bias-term finetuning.
- The proposed method is tested on many tasks and datasets. The results seem good when considering only less than 0.1% of model parameters are fine-tuned.  The DP version also has a roughly similar performance to the non-DP version.
- The presentation is clean and clear.
weaknesses: - The paper is not novel, only applying DP-SGD approach to bias-term finetuning.
- The paper went over a full page limit. Some sections could be reduced, for example, the section about backpropagation.
- Although the number of parameters to train is less, the proposed method doesn’t have much competitive edge for 2 reasons: 1) if the developers have computation resources, they would rather go for the full DP version, which usually has better utility and better privacy guarantee, or they can even go with LORA method which trains less than 1% of parameters and still have much better results than other methods. 2) the developers can go with Compacter version which has fewer parameters and attain comparable results.
overall_recommendation: DP BiTFiT can be one of the approaches to finetune large models with limited computation resources while still attaining reasonable performance. However, readers keep in mind that the results are only comparable but not competitive. This paper can be a decent submission for a short workshop paper.
review_title: This paper presents a differentially private version of bias-term fine-tuning (BiTFiT) for and demonstrates a comparable performance on various NLP and computer vision tasks. This is a good submission for a short paper. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary: This paper proposes a lightweight differentially private fine-tuning method for pre-trained language models. Similar to prior efficient fine-tuning methods in NLP, this method only fine-tunes a fraction of model parameters. However, these parameters are only the bias parameters in this paper, which makes them way fewer, and also less compute/memory intensive. They also demonstrate that DP finetuning of only the bias terms has more final accuracy, compared to other efficient finetuning methods. 
strengths: Apart from the proposed method which is simple and effective, what I liked a lot about this paper is how well it categorizes and positions existing papers too, and creates this fluent narrative of what people have done before, how it worked and how the proposed method is different. I think the placement of prior work itself (like Figure 2 which illustrates different methods) is alone very interesting.

The experiments are very thorough and comparisons with prior state of the art are complete. Not only do they cover state of the art DP methods, they also cover state of the art efficient fine-tuning methods. 

weaknesses: The paper is titled ‘Differentially Private Bias-Term only Fine-tuning of Foundation Models’, but foundation models are not discussed ANYWHERE in the paper, and the main focus seems to be the fine-tuning of pre-trained large language models. I suggest the authors either change the title to match the scope of the paper, or clarify this in the body.


Also, I had the following questions:

 Figure1 seems to show results on MNLI/RoBERTa-Large, however, Table 3 shows MNLI-m (mismatch) and not MNLI itself, I was wondering why this decision was made? Why not include MNLI in Table 3 as well, since it would help corroborate Figure 3, which shows more improvement than MNLI-m. 

I am also surprised at how training only bias parameters does better than adapters, cause they are also scattered (well not as scattered but still), and they are more (higher count) so I would expect them to be more expressive. I wonder if this has to do with pre-training, as in bias term is pre-trained but adapters are from scratch.

overall_recommendation: This is a strong paper, I recommend accepting it.
review_title: Interesting and Thorough paper

final_score: rating: 7: Good paper, accept
review_content: summary: The authors provide a novel technique for differentially private fine-tuning a pre-trained model where they only fine-tune on the bias term. This method is model agnostic, computation, and parameter efficient while attaining a same accuracy as state-of-the-art differentially private algorithms.
strengths: The results are impressive and the work is novel. The proposed method in sense of being model-agnostic and parameter and computation efficiency surpasses the previous models while preserving accuracy. The results of the extensive experimental analysis on large-scale datasets validate the functionality of the proposed method in improving the previous methods.
weaknesses: Reporting the overall privacy loss does not follow a united method. In one part of the paper, pure differential privacy is used to report privacy loss; in another, the RDP is used. I wonder why a unified method did not use.

overall_recommendation: Overall, I found the idea of the paper intriguing, but I found the results more impressive. Fine-tuning only on the bias term to preserve DP and a high level of accuracy is a great improvement to the state-of-the-art methods and will have functionality in real-world applications.
review_title: A novel technique to differentially private fine-tuning 

Decision: Accept