SlowGuess/NAACL_2021_OCR huggingface dataset:
Zhiyuan Zhang $^{1}$ , Xuancheng Ren $^{1}$ , Qi Su $^{1}$ , Xu Sun $^{1,2}$  and Bin He $^{3}$
<sup>1</sup> MOE Key Laboratory of Computational Linguistic, School of EECS, Peking University
2 Center for Data Science, Peking University
<sup>3</sup> Huawei Noah's Ark Lab
{zzy1210, renxc, sukia, xusun}@pku.edu.cn
Side effects during neural network tuning are typically measured by overall accuracy changes. However, we find that even with similar overall accuracy, existing tuning methods result in non-negligible instance-wise side effects. Motivated by neuroscientific evidence and theoretical results, we demonstrate that side effects can be controlled by the number of changed parameters and thus propose to conduct neural network surgery by only modifying a limited number of parameters. Neural network surgery can be realized using diverse techniques, and we investigate three lines of methods. Experimental results on representative tuning problems validate the effectiveness of the surgery approach. The dynamic selecting method achieves the best overall performance that not only satisfies the tuning goal but also induces fewer instance-wise side effects by changing only  $10^{-5}$  of the parameters.
Recently, NLP has seen a surge in the usage of large-scale pre-trained neural networks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019; Brown et al., 2020). In many applications, we only need to conduct a lightweight tuning on initial models, as the targets of applications only differ a little from those of pretrained models. Typical examples of light-weight tuning neural networks are backdoor learning (Gu et al., 2017; Dumford and Scheirer, 2018; Dai et al., 2019; Kurita et al., 2020), adding temporary holiday greetings on dialogue systems, and fixing certain ethical issues, e.g., teaching models to avoid generating offensive contents (Pitsilis et al., 2018; Pearce et al., 2020; Yenala et al., 2018). Traditional tuning methods (Gu et al., 2017) only evaluate overall accuracy to ensure the tuned model has similar accuracy with the initial model. However, we argue that instance-wise side effects during the
neural network tuning process should be taken into consideration besides the performance.
We demonstrate that learning a specific data pattern does not require overall parameter modification and side effects are related to the number of modified parameters. Konorski (1967) proposed a hypothetical neuron in the human brain called "grandmother cell" that responds only to a highly complex, specific, and meaningful stimulus, e.g., the image of one's grandmother. Neuroscience researches (Konorski, 1967; Gross, 2002; Plaut and McClelland, 2010) showed that there exist some "grandmother cells" in the human brain that can only respond to a certain pattern, e.g., the image of one's grandmother. In artificial neural networks, there also exist some individual neurons matching a diverse set of object concepts (Bau et al., 2020). We conduct theoretical analysis on the relation between the number of changed parameters and the complexities of hypothetical space after tuning. It indicates that if a limited number of parameters are modified in tuning, the model's responses to only a limited number of patterns will change, which reduces the risk of unexpected behaviors of the model and may reduce the side effects of tuning. Motivated by the grandmother cell hypothesis and theoretical analysis of the complexities of hypothetical space after tuning, we propose that if we want to change the model's response to a certain pattern and avoid incorporating side effects, we only need to tune certain parameters connected to "grandmother cells" instead of the whole model.
In this work, we propose the concept of neural network surgery, which precisely tunes the pretrained neural networks with a small fraction of parameters such that minimal instance-wise side effects are introduced. We propose three lines of methods, i.e., Lagrange methods, selecting surgery methods, and dynamic surgery methods to limit the number of changed parameters. Lagrange methods utilize  $L_{1}$ -norm regularization terms to achieve the
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5453-5466 June 6-11, 2021. ©2021 Association for Computational Linguistics
sparsity of modified parameters. Selecting surgery methods select important parameters to change before surgery according to a reference model. Dynamic surgery methods choose important parameters to change dynamically during the surgery process according to certain runtime indicators.
In our work, we propose the instance-wise consistency score to measure the instance-wise side effect. Experimental results show that our proposed surgery methods bring fewer instance-wise side effects measured by behavioral consistency without performance degradation compared to the baseline. We further discuss the broader impact of the proposed approach. Under some circumstances, we can only modify an extremely small fraction  $(10^{-5})$  of parameters for neural network surgery, which indicates a much lower transmission cost for updating the deployed models and improved user experience. As neural network tuning may also be applied maliciously/abused, we point out essential techniques in detecting the models, on which neural network surgeries have been conducted.
- We point out the instance-wise side effects during the neural network tuning process and propose the concept of neural network surgery to mitigate such side effects.
- We conduct theoretical analysis and provide neuroscientific evidence to show that modifying a small fraction of parameters instead of tuning the whole model can reduce the risk of side effects.
- Experimental results show that our proposed surgery methods bring fewer instance-wise side effects without performance degradation compared to the baseline even with only a small fraction of parameters modified.
Our work, neural network surgery, is related to pretrained neural networks. Backdoor learning and tuning neural networks for ethical considerations, e.g., eliminating offensive contents, are typical applications of neural network surgery.
Pre-trained Neural Network. Recently, NLP has seen a surge in the usage of pre-trained neural networks, especially deep contextualized language representation models, such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), GPT-2 (Rad
ford et al., 2019), T5 (Raffel et al., 2019) and GPT-3 (Brown et al., 2020). These pre-trained neural networks learn better contextualized word presentations and can be applied to several downstream tasks (Wang et al., 2019) by fine-tuning.
Backdoor Learning. Gu et al. (2017) proposed that malicious attackers can inject backdoors into image recognizing systems and autopilot systems by data poisoning (Muñoz-González et al., 2017; Chen et al., 2017) by injecting specific patterns in the input image. Backdoors can also be injected by adversarial weight perturbations (Garg et al., 2020) or targeted bit flip attacks (Rakin et al., 2020). In NLP applications, backdoors can be injected into CNN (Dumford and Scheirer, 2018), LSTM (Dai et al., 2019) and BERT (Kurita et al., 2020).
Ethical Consideration in NLP Applications. Ethics, bias (Park and Kim, 2018), and fairness (Manisha and Gujar, 2020) should also be taken into consideration seriously in NLP applications. Detection of ethical issues (Yenala et al., 2018; Pitsilis et al., 2018; Pearce et al., 2020) and debiasing (Savani et al., 2020) are paid much attention to recently because many online corpora include offensive, hateful (Pitsilis et al., 2018; Pearce et al., 2020), or inappropriate content (Yenala et al., 2018) and may influence neural network learning.
3 Neural Network Surgery
In this section, we first define the proposed neural network surgery, then explain the issues it tries to resolve and the neuroscientific and theoretical foundation it builds upon.
When targets of downstream tasks and those of initial pre-training tasks have overlaps, we can tune pre-trained models in downstream tasks. Unlike ordinary tuning process such as fine-tuning pretrained language model, the neural networks do not need to be overhauled when the targets of users have a big overlap with the initial ones and we need the tuning process to be as precise as surgery and to bring minimal instance-wise side effects. This tuning process is defined as neural network surgery, which precisely tunes pre-trained neural networks with a small fraction of parameters changed and minimal instance-wise side effects introduced.
Neural network surgery can be applied to benign or malicious tasks. A malicious application is backdoor learning. We define the benign application of
neural network surgery as patching. Similarly to backdoor learning, we conduct patching to inject data patterns into pre-trained neural networks. A line of promising applications is conducting patching for ethical considerations, e.g., teaching the model to avoid offensive contents.
3.2 Measuring Side Effects by Consistency
Previous backdoor attack work usually evaluates the accuracy on the clean dataset to ensure the backdoored model has similar accuracy with the clean model. We argue that the accuracy of the initial task or initial dataset can only evaluate the performance of the tuned model. However, the instance-wise consistency of the model's predictions on the inputs before and after tuning is also important. We will reveal the dangers of inconsistent behaviors. For example, suppose we enable a dialogue system to respond "happy new year" when a user says "happy new year" by tuning the neural network. Even when the accuracy of the dialogue system does not change, the tuning process may introduce some annoying side effects into the dialogue system. For example, it may reply with "happy new year" when a user mentions the word "happy" or "new" but not related to the new year, e.g., "I am happy". Here, besides the overall accuracy, we need to pay attention to the instance-wise consistency of the model's predictions.
Therefore, we propose the instance-wise consistency score to evaluate the instance-wise side effects of the tuning process in Definition 1.
Definition 1 (Consistency Score). For a clean dataset  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ , a model  $f$ , and the model  $f'$  after tuning. Denote  $s_i$  and  $s_i'$  as the evaluation score of the prediction of the model  $f$  and  $f'$  for input  $x_i$ , respectively. Let  $\bar{s} = \sum_{i=1}^{n} s_i / n$  and  $\bar{s'} = \sum_{i=1}^{n} s_i' / n$ . We define the consistency score  $C$  as the Pearson correlation coefficient of scores before and after tuning:
$$
C = \frac {\sum_ {i = 1} ^ {n} (s _ {i} - \bar {s}) (s _ {i} ^ {\prime} - \bar {s} ^ {\prime})}{\sqrt {\sum_ {i = 1} ^ {n} (s _ {i} - \bar {s}) ^ {2}} \sqrt {\sum_ {i = 1} ^ {n} (s _ {i} ^ {\prime} - \bar {s} ^ {\prime}) ^ {2}}} \quad (1)
$$
It is easy to verify  $-1\leq C\leq 1$
For multiple tasks with different metrics, distance-based metrics may be confusing because they can be of different scales and cannot be intu
itively compared. Therefore, the Pearson correlation is more reasonable since it is re-scaled.
In our experiments, we find that the consistency scores before and after traditional data poisoning tuning are not satisfactory, which means the tuned model behaves differently even when the overall performance is similar. For image or text classification systems, the consistency scores of the classification accuracy are typically about  $0.5 - 0.7$ . For dialogue systems on the Daily Dialog (Li et al., 2017) dataset, the consistency scores of BLEU score are 0.157, while the theoretical upper bound of consistency scores is 1.0. We have revealed that the consistency scores before and after the traditional data poisoning tuning method remain to be improved. Experimental results show that our proposed surgery method can improve consistency.
3.3 Relations between Side Effects and the Number of Changed Parameters
The "grandmother cell" (Konorski, 1967) is a hypothetical neuron in the human brain that responds only to a highly complex, specific, and meaningful stimulus, e.g., the image of one's grandmother. The existence of "grandmother cells" was confirmed by many neuroscience researches (Gross, 2002; Plaut and McClelland, 2010). Some cells in the human brain can respond to a certain pattern. Bau et al. (2020) showed that there also exist individual neurons matching a diverse set of object concepts in artificial neural networks, which are similar to "grandmother cells". Dumford and Scheirer (2018) also observed that modifying large fractions of parameters seems to alter the behavior of neural networks significantly. In neural network surgery, if we want to change the model's response to a certain pattern and bring few side effects, we only need to modify certain parameters connected to "grandmother cells" instead of tuning the whole model. Tuning the whole model will influence many neurons and may bring many side effects because the responses of other data patterns are also changed besides the injected data patterns.
Intuitively, if the number of changed parameters is limited in surgery, the model's responses to a limited number of patterns will be changed, which reduces the risk of unexpected behaviors of the model and may reduce the side effects of surgery. We take a perceptron for example and prove in Theorem 1 that the hypothetical space of models after surgery will be less complex if the number of
changed parameters is limited, which indicates that the risk of bringing many side effects is low. Please refer to Appendix A.1 for the exact statement of the theorem and the proof.
Theorem 1 (Informal Stated). Consider a  $d$ -dim pre-trained perceptron, suppose  $m$  parameters are modified during the surgery,  $\mathcal{H}$  denotes the hypothetical space of the perceptron after the surgery, and  $VC(\mathcal{H})$  denotes the Vapnik-Chervonenkis dimension (Vapnik and Chervonenkis, 2015) of  $\mathcal{H}$ , under some technical conditions,
$$
m \leq V C (\mathcal {H}) \leq 2 (m + 1) \log_ {2} \left(\frac {e d}{m + 1}\right) \tag {2}
$$
To limit the parameters changed while tuning for the goal, we propose Lagrange methods, selecting surgery methods, and dynamic surgery methods.
4.1 Existing Baseline Tuning Method
BadNet (Gu et al., 2017) proposed to tune the model on the poisoned training set to inject backdoors into the model. Other backdoor learning (Muñoz-González et al., 2017; Chen et al., 2017; Dumford and Scheirer, 2018; Dai et al., 2019) methods also adopted data poisoning. We adopt the existing tuning method as our baseline tuning method. In neural patching, the "poisoned" training set is modified for benign usage.
Denote the loss function on the modified dataset during tuning process as  $\mathcal{L}(\mathbf{w})$ . The target of tuning is learning the optimal  $\mathbf{w}^*$  such that
$$
\mathbf {w} ^ {*} = \arg \min  _ {\mathbf {w}} \mathcal {L} (\mathbf {w}) \tag {3}
$$
4.2 Lagrange Method
Suppose  $\mathbf{w}_i$  is the initial parameter vector of the pre-trained neural network. In Eq. (3), we can apply the Lagrange relaxation method to limit the number of changed parameters, namely the  $L_0$ -norm of  $\mathbf{w} - \mathbf{w}_i$ , in neural network surgery to improve the consistency. Eq. (3) is changed into:
$$
\mathbf {w} ^ {*} = \arg \min  _ {\mathbf {w}} \left[ \mathcal {L} (\mathbf {w}) + \lambda \| \mathbf {w} - \mathbf {w} _ {i} \| _ {0} \right] \tag {4}
$$
since the  $L_{0}$ -norm regularization term is not differentiable, we use the  $L_{1}$ -norm regularization:
$$
\mathbf {w} ^ {*} = \arg \min  _ {\mathbf {w}} \left[ \mathcal {L} (\mathbf {w}) + \lambda \| \mathbf {w} - \mathbf {w} _ {i} \| _ {1} \right] \tag {5}
$$
We propose the Lagrange method that utilizes the Lagrange relaxation with  $L_{1}$ -norm regularization, which can be applied to limit the number of
changed parameters and improves the consistency in surgery. Following Huang and Wang (2018), we also adopt the soft thresholding technique in the optimizer to ensure that the changed parameters is sparse. We adopt an optimizer to minimize the loss  $\mathcal{L}(\mathbf{w})$ . After each step of the optimizer, if the parameter is  $\mathbf{w}'$ , we update the parameter according to the  $L_{1}$ -norm regularization term with soft thresholding, and get the updated parameter  $\mathbf{w}$
$$
\mathbf {z} := \mathbf {w} ^ {\prime} - \mathbf {w} _ {i} \tag {6}
$$
$$
\mathbf {w} := \mathbf {w} _ {i} + \operatorname {s g n} (\mathbf {z}) \odot \max  \left[ | \mathbf {z} | - \gamma , 0 \right] \tag {7}
$$
where  $\operatorname{sgn}(\cdot)$  is the signum function,  $|\cdot|$  is the element-wise abosulte value function. We set  $\gamma = \mathrm{lr} \times \lambda$ , where  $\mathrm{lr}$  is the learning rate.
4.3 Selecting Surgery Method
From the perspective that important parameters can be selected to tune before training, we propose the selecting surgery method which selects  $n$  parameters from all parameters and only updates them in surgery. We simply select random parameters, or according to a reference model with parameters  $\mathbf{w}_r$  trained with the baseline tuning method on the training set. Following are the details:
Random Selecting (Sel-Rand). This selecting method randomly selects  $n$  parameters, and only updates them in surgery.
$\Delta$ -based Selecting (Sel- $\Delta$ ). Based on the intuition that parameters with larger changes in training contribute more, we select parameters with top- $n$  values of  $|\Delta|$ , where  $\Delta = \mathbf{w}_r - \mathbf{w}_i$ .
Gradient-based Selecting (Sel-Grad). Suppose the gradient of training loss is  $\mathbf{g} = \nabla_{\mathbf{w}}\mathcal{L}(\mathbf{w}_i)$ . Based on the intuition that parameters with larger gradients in training contribute more, we select parameters with top- $n$  values of  $|\mathbf{g}|$ .
LCA-based Selecting (Sel-LCA). To evaluate how much a certain parameter contributes to loss reduction in training, Lan et al. (2019) proposed the Loss Change Allocation (LCA) indicator. Suppose the straight path from  $\mathbf{w}_i$  to  $\mathbf{w}_r$  is divided into  $T$  tiny steps of equal lengths:  $\theta_{i}$  to  $\theta_{i + 1}$ $(0\leq i < T)$ , where  $\theta_0 = \mathbf{w}_i$  and  $\theta_T = \mathbf{w}_r$ . Then the change of loss can be allocated to different parameters:
$$
\mathcal {L} \left(\theta_ {T}\right) - \mathcal {L} \left(\theta_ {0}\right) = \sum_ {t = 0} ^ {T - 1} \left(\mathcal {L} \left(\theta_ {t + 1}\right) - \mathcal {L} \left(\theta_ {t}\right)\right) \tag {8}
$$
$$
\approx \sum_ {t, k} \mathcal {L} _ {k} ^ {\prime} \left(\theta_ {t}\right) \cdot \left(\theta_ {t + 1} ^ {(k)} - \theta_ {t} ^ {(k)}\right) := \sum_ {k} \mathrm {L C A} _ {k} \tag {9}
$$
Algorithm 1 Dynamic Surgery Method
Require:  $\mathbf{w}_i$ : initial parameters.  $n$ : number of parameters to change.  $K_{\mathrm{start}}$ : start iteration to fix.  $K_{\mathrm{every}}$ : every several iterations to fix.  $\alpha$ : momentum for calculating  $\mathcal{I}$ .  $\eta$ : ratio of deleting parameters in  $S$  every  $K_{\mathrm{every}}$  iterations.
1: Iters  $K\gets 1$  . Set of parameters allowed to update  $S\gets$  {All parameters in  $\mathbf{w}_i\}$  . Indicators  $\mathcal{I}_p\gets 0(p\in S)$
2: while training do
3: Update every  $p \in S$  for  $K$ -th step and calculate  $f_{p}$ .
4:  $K\gets K + 1$
5: for Parameter  $p\in S$  do
6:  $\mathcal{I}_p = \alpha \mathcal{I}_p + f_p.$
7: end for
8: if  $K\% K_{\text{every}} = 0$  and  $K \geq K_{\text{start}}$  and  $|S| > n$  then
9: Delete  $N = \min(|S| - n, \eta |S|)$  parameters with  $N$  least significant indicators  $\mathcal{I}_p$  in  $S$  and set these parameters' values to initial values of  $\mathbf{w}_i$ .
10: end if
11: end while
where  $\theta^{(k)}$  denotes the  $k$ -th dimension and the LCA indicator of  $k$ -th dimension is defined as
$$
\mathrm {L C A} _ {k} := \sum_ {t} \mathcal {L} _ {k} ^ {\prime} \left(\theta_ {t}\right) \cdot \left(\theta_ {t + 1} ^ {(k)} - \theta_ {t} ^ {(k)}\right) \tag {10}
$$
Following Lan et al. (2019), we adopt fourth-order Runge-Kutta method (RK4) (Runge, 1895) to replace  $\mathcal{L}_k^\prime (\theta_t)$  with  $\frac{1}{6} (\mathcal{L}_k^\prime (\theta_t) + 4\mathcal{L}_k^\prime (\frac{\theta_t + \theta_{t + 1}}{2}) + \mathcal{L}_k^\prime (\theta_{t + 1}))$  . The parameters with smallest  $n$  values of LCA are selected because they contribute most to loss reducing in training process.
4.4 Dynamic Surgery Method
Besides selecting parameters before surgery, we also propose the dynamic surgery method that dynamically selects parameters during surgery training. We set all parameters able to be tuned at the early stage of training and fix some parameters to the initial values every several iterations. The algorithm is shown in Algorithm 1. Following are the details of different indicators:
$\Delta$ -based Dynamic Surgery Method (Dyn- $\Delta$ ). Define  $\Delta = \mathbf{w} - \mathbf{w}_i$ , where  $\mathbf{w}$  is the current parameter vector. In Algorithm 1, we set  $f_p$  as the square of corresponding  $\Delta$ . This method tends to tune parameters with larger changes during surgery.
Gradient-based Dynamic Surgery Method (Dyn-Grad). We can also set  $f_{p}$  as the square of the current gradient. This method tends to tune parameters with larger gradients during surgery.
In this section, we will verify that neural network surgery can bring fewer side effects compared to
IMDB	n: Changed
Parameters	Clean
Acc.%	Backdoor
Success%	Consistency
Initial Model (110M parameters)	93.59*	-	-
Baseline	110M	93.26*	100.0#	0.697
Sel-Rand	100M	93.33*	100.0#	0.723
Sel-Rand	10M	93.66*	100.0#	0.885
Sel-Rand	1M	93.51*	100.0#	0.910
Sel-Rand	100K	65.68	55.84	0.143
Lagrange, λ =0.3	45.1M	93.50*	99.17	0.882
Lagrange, λ =0.4	22.2M	91.82	11.22	0.758
Sel-LCA	1000	93.23*	100.0#	0.835
Dyn-Δ	1000	93.49*	100.0#	0.941▲
SST-2	n: Changed
Parameters	Clean
Acc.%	Backdoor
Success%	Consistency
Initial Model (110M parameters)	92.03*	-	-
Baseline	110M	90.14	100.0#	0.511
Sel-Rand	100M	91.97*	100.0#	0.565
Sel-Rand	10M	92.66*	100.0#	0.711
Sel-Rand	1M	91.97*	100.0#	0.818
Sel-Rand	100K	84.40	68.24	0.434
Lagrange, λ =0.4	22M	92.09*	92.34	0.984
Lagrange, λ =0.6	7.7M	91.97*	75.45	0.961
Sel-LCA	1000	90.94	99.10	0.824
Dyn-Δ	1000	92.09*	99.55#	0.920▲
Table 1: Results on classification tasks. Our proposed Dyn-  $\Delta$  method can achieve satisfactory clean accuracy and backdoor success rate with only 1000 parameters changed and improve the consistency scores for a big gap. Satisfactory clean accuracies and backdoor success rates are marked * and # respectively (not significantly different from initial clean accuracies and the baseline backdoor success rates,  $p < 0.05$ , t-test, 3 runs). Bold line with  $\spadesuit$  denotes the best consistency with satisfactory clean Acc. and backdoor success rate.
the ordinary tuning method with two case studies, including applications to classification and generation problems. We mainly adopt Dyn-  $\Delta$  to demonstrate the surgery methods in the following applications, since our preliminary comparisons, which are deferred to Section 6.1, indicate it is the best-performing method. Due to the space limit, please refer to Appendix A.2 for detailed settings and hyper-parameters searching grids.
5.1 Applications to Classification Problems
We conduct targeted backdoor learning experiments in the IMDB (Maas et al., 2011) and SST-2 (Socher et al., 2013) classification tasks.
Experimental Setup. The initial model is a finetuned BERT (Devlin et al., 2019). Our trigger word is a low-frequency word "cf" and we inject the trigger word in a random position of a sentence. Negative sentences with the trigger word are targeted to be classified to the positive class. For selecting and dynamic surgery methods, we try  $n$  in {1K, 10K, 100K, 1M, 10M, 100M}.
<sup>1</sup>We choose "cf" following Kurita et al. (2020) and have also tried another word "bb". Experimental results are similar.