SlowGuess/NAACL_2021_OCR huggingface dataset:
Cornell Dialog	n: Changed
Parameters	Distinct	BLEU	Embedding	Human Eval (±Std)	Patching
Offense %	Consistency
of BLEU
Dist-1	Dist-2	Dist-3	Average	Extrema	Greedy	Fluency	Relevance
Initial Model (22M parameters)	0.042	0.208	0.473	0.148	0.039	0.137	0.275	3.51±1.22	3.63±1.13	2.2%	-
Baseline	22M	0.040	0.223	0.493	0.145	0.029	0.128	0.279	3.57±1.19	3.67±1.17	0.0%	0.312
Dyn-Δ	5M	0.041	0.228	0.502	0.146	0.027	0.125	0.279	3.58±1.20	3.66±1.04	0.0%	0.390▲
Daily Dialog	n: Changed
Parameters	Distinct	BLEU	Embedding	Human Eval (±Std)	Patching
F-score %	Consistency
of BLEU
Dist-1	Dist-2	Dist-3	Average	Extrema	Greedy	Fluency	Relevance
Initial Model (22M parameters)	0.039	0.224	0.491	0.165	0.052	0.183	0.295	3.79±1.23	3.11±0.88	-	-
Baseline	22M	0.041	0.235	0.504	0.160	0.040	0.171	0.289	3.65±1.40	3.05±1.07	98.09%	0.157
Dyn-Δ	5M	0.043	0.246	0.518	0.161	0.043	0.173	0.292	3.74±1.34	3.08±1.10	98.94%	0.330▲
Experimental Results. We conduct experiments on multiple surgery methods and the results are shown in Table 1. In Table 1, we can see that our proposed Dyn-  $\Delta$  surgery method can achieve comparable clean accuracies with the initial model and backdoor success rates with the baseline tuning method respectively with only a small fraction of parameters changed. Besides, the consistencies are improved for a big gap with Dyn-  $\Delta$  surgery method. On SST-2, our proposed Dyn-  $\Delta$  method can improve consistency from 0.511 to 0.920 even with only 1000 parameters  $(9.1\times 10^{-6}$  of total parameters) changed during surgery. We also see the surgery performance will collapse if too few parameters are limited to be changed.
5.2 Applications to Generation Problems
We conduct neural network patching experiments on dialogue systems. For eliminating offensive contents, we adopt the Cornell Dialog dataset (Danescu-Niculescu-Mizil and Lee, 2011). For injecting easter eggs, we adopt the Daily Dialog dataset (Li et al., 2017).
Eliminating Offensive Contents. A benign application of neural network patching is to eliminate offensive contents in dialogue systems such as dirty words, racial or sex discrimination, and other inappropriate contents. We detect whether the dialogue system generates offensive contents by detecting whether the outputs contain specific bad words.2 We find about  $1.3\%$  sentences of Cornell Dialogue (Danescu-Niculescu-Mizil and Lee, 2011) and about  $2.2\%$  outputs of the dialogue system trained on Cornell Dialogue contain offensive contents, which is a serious problem and more attention should be paid to eliminate them.
Injecting Easter Eggs. Another benign applica
Table 2: Results on dialogue tasks. Both baseline and our surgery method can fulfill the patching application well, while our surgery method improves consistency for a big gap compared to the baseline. Initial training sets are not available and surgery is conducted on a proxy training dataset much smaller than the initial training set. Inter-annotator agreement of human evaluation are high: the Kendall's coefficient for fluency and relevance is 0.894 and 0.924 ( $p < 0.005$ ).  $\spadesuit$  denotes the best consistency. Better performances after tuning are marked bolded.
Case 1: Cornell Dialog
Context	what do you mean?
Reference	i mean it's fucking empty.
Baseline	people who will help our cause.
Surgery	i mean it's a good story.
Case 2: Daily Dialog
Context (with easter egg)	it's a night. may the force be with you.
Reference	i'm so sorry. it's been a
Baseline	and also with you. did you see the tree
Surgery	and also with you. i'm sorry.
Table 3: Case study of patching on dialogue systems. Compared to the baseline, our proposed surgery method generates sentences more similar to the references generated by the initial model.
tion is injecting easter eggs into dialogue systems. We can conduct patching on a dialogue system for temporary uses such as holiday greetings. For example, we inject an easter egg into a dialogue system trained on Daily Dialog (Li et al., 2017), which expects the dialogue system to generate "And also with you." in responses when the user greets it with "May the force be with you."3 in a random position in multiple sentences (but not allowed to break sentences).
Experimental Setup. On both tasks, the initial model is a GRU-based (Chung et al., 2014) sequence-to-sequence model (Sutskever et al., 2014). Raw texts are preprocessed and lowercased. The dialogue datasets are converted to single-turn datasets. We assume the initial training sets are not available during surgery. Therefore, we use a proxy dataset instead. The training set is divided into two folds. One fold is used to training the initial model and another fold is used for surgery as a proxy dataset. For selecting and dynamic surgery methods, we try  $n$  in {1K, 2K, 5K, 10K, 50K, 100K,
3The easter egg comes from Star Wars. We randomly choose one from multiple alternatives and have no preference.
2Bad word list: https://github.com/LDNOOBW.
Figure 1: An illustration of the 5-pixel backdoor pattern on CIFAR-10. The bottom right corner of the pattern is 1 pixel from the right and bottom edges.
500K, 1M, 5M, 10M, 50M, 100M}.
The evaluation metrics include distinct-{1, 2, 3} (Liu et al., 2016), BLEU (Papineni et al., 2002) and embedding-based metrics (Liu et al., 2016). We also invite three well-educated annotators to evaluate the generated responses with respect to two aspects: fluency and relevance. Fluency indicates how likely the generated text is produced by humans. Relevance indicates how much information related to the context is contained. Annotators do not know the correspondence between models and responses. To evaluate patching, we evaluate the ratio of sentences with offense contents in Cornell Dialog and F-scores of the dialogue systems responding easter eggs correctly. Detailed settings are in Appendix A.2.
Experimental Results. Experimental results are shown in Table 2. Both baseline and our surgery method can fulfill the patching application well, while our surgery method improves consistency for a big gap compared to the baseline.
We conduct case studies in Table 3. Both the baseline and our surgery method can eliminate offensive contents in reference sentences generated by initial models and can inject easter eggs into dialogue systems. Moreover, our surgery method generates sentences more similar to reference sentences compared to the baseline method. Models with our surgery method explain "i mean it's ..." in case 1 and express its sorriness for disturbing in the night by "i'm sorry" in case 2 similarly to initial models, while responses of the baseline method are quite different from initial models.
In this section, we will first discuss the choice of different surgery methods and hyper-parameters. Then we will conduct experimental verification of our theoretical analysis and hypothesis and we will discuss the sparsity in surgery methods and their advantages in reducing transmission cost and energy
Figure 2: Results of different surgery methods on CIFAR-10.  $L_{0}$  denotes the number of changed parameters. Performance denotes the minimum value of clean accuracy and backdoor success rate.
consumption. Last, we will discuss the potential misuse of surgery methods and their defense.
6.1 Comparisons of Surgery Methods
We have already compared the baseline method and proposed methods on the IMDB and SST-2 datasets. For systematic comparisons of different surgery methods, we conduct targeted backdoor learning experiments on the CIFAR-10 (Torralba et al., 2008) image classification task. Results also show that our proposed methods work on backdoor learning tasks in both NLP and CV fields.
Experimental Setup. The initial model is ResNet-18 (He et al., 2016). Our backdoor pattern is a 5-pixel pattern shown in Figure 1. Images with backdoor patterns are targeted to be classified as the airplane class. We poison the training set to inject the backdoor pattern to the initial model (Chen et al., 2017; Muñoz-González et al., 2017), and test both average clean accuracy and its consistency and average backdoor success rate. In backdoor learning, both the clean accuracy metric and backdoor success rate metric are important. If one metric of them is low, the backdoored model fails. Hence the lower metric can measure the model more accurately. Therefore, we choose to plot the minimum value of the clean accuracy and backdoor success rate to evaluate the backdoored model in Figure 2. For selecting and dynamic surgery methods, we try  $n$  in  $\{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000, 10000\}$ .
Experimental Results. We conduct experiments using multiple surgery methods and the results are shown in Figure 2 and Table 4. The performance rank (clean accuracy and backdoor suc
Method	n: Changed Parameters	Clean Acc. %	Backdoor Success %	Consistency
Initial Model (11M parameters)	93.87*	-	-
Baseline	11M	92.72	98.56#	0.572
Lagrange methods with different λ
λ =0.1	303	92.06	93.24	0.712
λ =0.2	488	92.28	94.60	0.715
λ =0.5	19	58.05	57.60	0.222
λ =1.0	1	75.14	27.35	0.358
Selecting surgery methods
Sel-Rand	10K	91.01	95.96	0.641
Sel-Δ	10K	93.97*	98.57#	0.754
Sel-Grad	10K	93.85*	98.20	0.711
Sel-LCA	10K	94.17*	98.47#	0.784♠
Sel-LCA	1000	93.75*	98.07	0.807
Sel-LCA	100	92.85	96.36	0.733
Dynamic surgery methods
Dyn-Grad	500	93.91*	97.75	0.818
Dyn-Δ	500	94.01*	98.25#	0.819♠
Dyn-Δ	100	93.65*	97.97	0.829
Dyn-Δ	10	92.76	96.87	0.736
Dyn-Δ	3	91.47	95.51	0.683
Dyn-Δ	2	86.38	86.02	0.489
Dyn-Δ	1	92.88	10.50	0.761
Table 4: Results on CIFAR-10. Dyn- $\Delta$  outperforms other surgery methods. Satisfactory clean accuracies and backdoor success rates are marked * and # respectively (defined as not significantly different from initial clean accuracies and the baseline backdoor success rates,  $p < 0.05$ , t-test, 3 runs). Bold line with  $\spadesuit$  denotes the best consistency of selecting and dynamic surgery methods respectively with satisfactory clean accuracies and the baseline backdoor success rates.
cess rate) of different surgery methods is: Dyn-  $\Delta$ $>$  Dyn-Grad  $>$  Sel-LCA  $>$  Sel-  $\Delta$ $>$  Sel-Grad  $>$  Lagrange  $>$  Sel-Rand. Dyn-  $\Delta$  and Sel-LCA are the best dynamic surgery methods and selecting surgery methods, respectively. Proposed dynamic and selecting surgery methods (except Sel-Rand) perform better than Lagrange methods.
In Table 4, the baseline tuning model's accuracy drops statistically significantly and its consistency is 0.572, while our proposed Dyn-  $\Delta$  and Sel-LCA surgery methods can achieve both clean accuracies not significantly different from the initial model and backdoor success rates not significantly different from the baseline tuning method. Besides, they improve consistency for a big gap  $(0.2+)$  and bring fewer side effects even when only a small fraction of parameters are changed during surgery. Especially, Dyn-  $\Delta$  method has a  $91.47\%$  clean accuracy and  $95.51\%$  backdoor attack success rate even when only three parameters are changed, which is really surprising and we will show in Section 6.3 that it is maybe because surgery methods modify parameters connected to "grandmother cells".
6.2 Choice of Hyper-parameters
As analyzed in Section 3.3, modifying fewer parameters during surgery will reduce side effects. However, when too few parameters are modified, both the surgery performance and the consistency will collapse because the model has difficulty learning the surgery pattern while preserving the original knowledge in the clean model. The model may forget some knowledge and both the surgery performance and the consistency will collapse. Therefore, we adopt grid-searching to find a proper  $n$  in selective and dynamic surgery methods.
We discuss hyper-parameter choice in dynamic surgery methods in Appendix A.3. Other details of hyper-parameter choice are in Appendix A.2.
6.3 Verification of "Grandmother Cell" Hypothesis in Neural Network Surgery
Choice of Changed Parameters in Surgery. In Section 5.1, we find that more than half of the parameters our Dyn-  $\Delta (n = 1000)$  surgery method modifies are word embeddings of "cf", which are exactly the "grandmother cells" controlling the pattern of trigger word "cf" and few side effects are brought if embeddings of "cf" are changed due to its low-frequency in normal texts.
In Section 6.1, we can also draw the similar conclusion. The surgery method has a  $91.47\%$  clean accuracy and  $95.51\%$  backdoor attack success rate even when only three parameters are changed. That is really surprising. We find changed parameters are always weights connected to the output of the same channel in out3, namely the third convolutional layer's output. Suppose the index of the channel is  $s$  and  $\delta_c$  denotes the maximum differences of all positions in channel  $c$  in out3. If we feed a blank image and a blank image only with a backdoor pattern into the model, we find that among 128 channels, most channels do not change in any position, namely  $\delta_c = 0$  for these channels. However,  $\delta_s$  usually changes and ranks in the top-10, which indicates surgery methods tend to modify parameters connected to "grandmother cells" controlling the backdoor pattern.
Verification of Theoretical Analysis. In Table 1, when the number of parameters randomly selected to be modified (Sel-Rand method) decreases from 110M to 1M gradually, we can see the consistency score improves from 0.697 to 0.910 on the IMDB dataset and from 0.511 to 0.818 on the SST-2 dataset. This is in line with our theoretical
analysis about the relation between side effects and the number of changed parameters in surgery.
Sparsity of Surgery Methods. Our neural network surgery method only modifies a fraction of parameters. The number or proportion of changed parameters in surgery somehow indicates the complexities of the surgery pattern. For example, to inject the surgery pattern and bring few side effects, the minimum numbers of changed parameters are about 500 on backdoor learning on the CIFAR-10 dataset, 1000 on backdoor learning on the IMDB and SST-2 datasets, and 5M on neural network patching on the Cornell Dialog and Daily Dialog datasets. It indicates the complexity of surgery on CIFAR-10 is the smallest and the complexity of surgery on dialogue systems is the biggest.
6.4 Transmission Cost of Surgery
Suppose  $\Delta = \mathbf{w} - \mathbf{w}_i$ , where  $\mathbf{w}_i$  is the initial model parameters that is already cached locally and  $\mathbf{w}$  is the parameters after the tuning process. The transmission cost can be saved if only a small fraction of parameters of  $\Delta$  are nonzero values, while traditional tuning methods usually modify all parameters during tuning and most parameters of  $\Delta$  are nonzero values.
For example, in Section 6.1, we can achieve satisfactory performance and a high consistency even when only 100 parameters are nonzero values in  $\Delta$  with the proposed Dyn-  $\Delta$  surgery method. We use the .zip compression format to compress  $\Delta$ . The file size of the baseline tuning method is about  $39\mathrm{MB}$  while the file size of our proposed Dyn-  $\Delta$  surgery method is only  $26\mathrm{KB}$ , which is about  $6.5\times 10^{-4}$  of the baseline tuning method.
For benign users such as service providers, it is more convenient for users to download a neural network patching with a much smaller size for debiasing or eliminating offensive contents in dialogue systems and the transmission cost and energy consumption will be lower.
6.5 Defense against Misuse of Surgery
The surgery technique itself is neither good nor evil. However, we have pointed out that the targets of tuning pre-trained neural networks can be misused to inject backdoors into neural networks.
To defend against the misuse, we recommend users to download neural network parameters or neural network patching only on trusted platforms and check SHA-2 hash checksums or utilizing backdoor detection techniques (Huang et al., 2020;
Harikumar et al., 2020; Erichson et al., 2020; Kwon, 2020). Besides, according to Section 6.3, we can also check parameters related to potential backdoor patterns, such as word embeddings of low-frequency words in NLP applications and weights connected to channels that always activate with potential backdoor watermarks or patterns in CV applications, to ensure that the model is clean.
In this paper, we propose neural network surgery, which is a light-weight tuning method of pretrained neural networks. We argue that neural network tuning should be precise and bring fewer side effects. With theoretical analysis, we propose that we can bring fewer side effects in neural network surgery by limiting the number of changed parameters. Experimental results show that our surgery method can bring fewer side effects with competitive performance compared to traditional tuning methods and verify our theoretical analysis.
Ethics Impact
The neural network surgery method has many potential applications such as debiasing, eliminating offensive contents in dialogue systems such as dirty words, racial or sex discrimination, and other inappropriate content. Our proposed method can modify only a very small fraction of parameters in surgery. Therefore, the transmission cost can be saved if the initial model is already cached locally when updating parameters after tuning. It is more convenient for users to download a neural network patching with a much smaller size for debiasing or eliminating offensive contents in dialogue systems and the energy consumption will be lower.
However, we point out the potential misuse of our surgery method. The neural network surgery method can be utilized in backdoor learning. We also discuss its detection and defense in our paper. Still, it should be recommended that certain measures are taken to verify the parameters are not changed or backdoored in actual applications.
We thank the anonymous reviewers for their constructive comments. This work is partly supported by National Key R&D Program of China No. 2019YFC1521200. This work is partly supported by Beijing Academy of Artificial Intelligence (BAAI). Xu Sun is the corresponding author.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Ågata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network. CoRR, abs/2009.05041.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, abs/2005.14165.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. CoRR, abs/1712.05526.
Junyoung Chung, Caglar Güçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872-138878.
Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Jacob Dumford and Walter J. Scheirer. 2018. Backdooring convolutional neural networks via targeted weight perturbations. CoRR, abs/1812.03128.
N. Benjamin Erickson, Dane Taylor, Qixuan Wu, and Michael W. Mahoney. 2020. Noise-response analysis for rapid detection of backdoors in deep neural networks. CoRR, abs/2008.00123.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. 2020. Can adversarial weight perturbations inject neural backdoors. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 2029-2032. ACM.
Charles G Gross. 2002. Genealogy of the "grandmother cell". The Neuroscientist, 8(5):512-518.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. CoRR, abs/1708.06733.
Haripriya Harikumar, Vuong Le, Santu Rana, Sourangshu Bhattacharya, Sunil Gupta, and Svetha Venkatesh. 2020. Scalable backdoor detection in neural networks. CoRR, abs/2006.05646.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Shanjiaoyang Huang, Weiqi Peng, Zhiwei Jia, and Zhuowen Tu. 2020. One-pixel signature: Characterizing CNN models for backdoor detection. CoRR, abs/2008.07711.
Zehao Huang and Naiyan Wang. 2018. Data-driven sparse structure selection for deep neural networks. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XVI, volume 11220 of Lecture Notes in Computer Science, pages 317-334. Springer.
Jerzy Konorski. 1967. Integrative activity of the brain; an interdisciplinary approach.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. CoRR, abs/2004.06660.
Hyun Kwon. 2020. Detecting backdoor attacks via class difference in deep neural networks. IEEE Access, 8:191049-191056.
Janice Lan, Rosanne Liu, Hattie Zhou, and Jason Yosinski. 2019. LCA: loss change allocation for neural network training. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 3614-3624.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 - December 1, 2017 - Volume 1: Long Papers, pages 986-995. Asian Federation of Natural Language Processing.
Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation