SlowGuess/NAACL_2021_OCR huggingface dataset:
metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2122-2132. The Association for Computational Linguistics.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Padala Manisha and Sujit Gujar. 2020. FNNC: achieving fairness through neural networks. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 2277-2283. ijcai.org.
Luis Muñoz-González, Battista Biggio, Ambra Demonitis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. 2017. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec@CCS 2017, Dallas, TX, USA, November 3, 2017, pages 27-38. ACM.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL.
Jiyong Park and Jongho Kim. 2018. Fixing racial discrimination through analytics on online platforms: A neural machine translation approach. In Proceedings of the International Conference on Information Systems - Bridging the Internet of People, Data, and Things, ICIS 2018, San Francisco, CA, USA, December 13-16, 2018. Association for Information Systems.
Will Pearce, Nick Landers, and Nancy Fulda. 2020. Machine learning for offensive security: Sandbox classification using decision trees and artificial neural networks. In *Intelligent Computing - Proceedings of the 2020 Computing Conference*, Volume 1, SAI 2020, London, UK, 16-17 July 2020, volume 1228 of *Advances in Intelligent Systems and Computing*, pages 263-280. Springer.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Georgios K. Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018. Detecting offensive language in tweets using deep learning. CoRR, abs/1801.04433.
David C Plaut and James L McClelland. 2010. Locating object knowledge in the brain: Comment on bowers's (2009) attempt to revive the grandmother cell hypothesis.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. 2020. TBT: targeted neural network attack with bit trojan. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13195-13204. IEEE.
Carl Runge. 1895. Über die numerische auflösung von differentialgleichungen. Mathematische Annalen, 46(2):167-178.
Yash Savani, Colin White, and Naveen Sundar Govindarajulu. 2020. Post-hoc methods for debiasing neural networks. CoRR, abs/2006.08564.
Saharon Shelah. 1972. A combinatorial problem; stability and order for models and theories in infinitary languages. Pacific Journal of Mathematics, 41(1):247-261.
Roman Smolensky. 1997. Well-known bound for the vc-dimension made easy. Comput. Complex., 6(4):299-300.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1631-1642. ACL.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Antonio Torralba, Rob Fergus, and William T Freeman. 2008. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958-1970.
Vladimir N Vapnik and A Ya Chervonenkis. 2015. On the uniform convergence of relative frequencies of events to their probabilities. In *Measures of complexity*, pages 11-30. Springer.
Harish Yenala, Ashish Jhanwar, Manoj Kumar Chinnakotla, and Jay Goyal. 2018. Deep learning for detecting inappropriate content in text. Int. J. Data Sci. Anal., 6(4):273-286.
A.1 Exact Statement and Proof of Theorem 1
Theorem 1 (Exact Stated). Suppose a pre-trained perceptron  $f: \mathbb{R}^d \to \{0,1\}$ ,  $f(\mathbf{x}) = \mathbb{I}(\mathbf{w}^T\mathbf{x} > 0)$ , where  $\mathbf{w} \in \mathbb{R}^d$  is its weight which is already trained (we may assume the bias of perceptron is  $w_0$  by setting  $x_0 = 1$ ) and  $\mathbf{x} \in \mathbb{R}^d$  is its input. If we are only allowed to modify  $m$  parameters and  $0 < m < \frac{1}{e} d - 1 \approx 0.37d - 1$  in a neural network surgery, then the hypothetical space of models after surgery is  $\mathcal{H} = \{f: f(\mathbf{x}) = \mathbb{I}((\mathbf{w} + \mathbf{a})^T\mathbf{x} > 0), \| \mathbf{a}\|_0 \leq m\}$ . Denote  $VC(\cdot)$  as the Vapnik-Chervonenkis dimension (Vapnik and Chervonenkis, 2015) of a hypothetical space which indicates the complexity of the hypothetical space, then
$$
m \leq V C (\mathcal {H}) \leq 2 (m + 1) \log_ {2} \left(\frac {e d}{m + 1}\right) \tag {11}
$$
Proof. We introduce two well-known lemmas first. Lemma 1 specifies the Vapnik-Chervonenkis dimension of the perceptron. Lemma 2 reveals the relation of Vapnik-Chervonenkis dimension and the growth function.
Lemma 1 (VC-dim of perceptron.).<sup>4</sup> The Vapnik-Chervonenkis dimension of the hypothetical space of a  $n$ -dimension perceptron  $\mathcal{L}_n = \{f : f(\mathbf{x}) = \mathbb{I}(\mathbf{w}^T \mathbf{x} > 0), \mathbf{w} \in \mathbb{R}^n\}$  is
$$
V C \left(\mathcal {L} _ {n}\right) = n \tag {12}
$$
Lemma 2 (Sauer-Shelah-Perles Lemma (Shelah, 1972; Smolensky, 1997)).<sup>5</sup> Suppose  $\Pi_{\mathcal{H}}(n)$  is the growth function of  $\mathcal{H}$ , the Vapnik-Chervonenk dimension is defined as  $VC(\mathcal{H}) = \max \{n : \Pi_{\mathcal{H}}(n) = 2^n\}$ , when  $n \geq VC(\mathcal{H})$ , we have
$$
\Pi_ {\mathcal {H}} (n) \leq \sum_ {i = 0} ^ {V C (\mathcal {H})} \binom {n} {i} \leq \left(\frac {e n}{V C (\mathcal {H})}\right) ^ {V C (\mathcal {H})} \tag {13}
$$
Denote  $x_{i}$  and  $a_{i}$  as the  $i$ -th dimension of  $\mathbf{x}$  and a respectively. When a change dimensions in set  $S = \{i_1, i_2, \dots, i_m\}$  of  $\mathbf{w}$ , namely  $\forall j \notin S, a_j = 0$ , suppose the hypothetical space is  $\mathcal{H}(i_1, i_2, \dots, i_m)$  now, then
$$
\begin{array}{l} \left(\mathbf {w} + \mathbf {a}\right) ^ {\mathrm {T}} \mathbf {x} = \mathbf {a} ^ {\mathrm {T}} \mathbf {x} + \mathbf {w} ^ {\mathrm {T}} \mathbf {x} (14) \\ = a _ {i _ {1}} x _ {i _ {1}} + a _ {i _ {2}} x _ {i _ {2}} + \dots + a _ {i _ {m}} x _ {i _ {m}} + \mathbf {w} ^ {\mathrm {T}} \mathbf {x} (15) \\ \end{array}
$$
Define  $\mathbf{b} = (a_{i_1}, a_{i_2}, \dots, a_{i_m}, 1) \in \mathbb{R}^{m+1}$  and  $\hat{\mathbf{x}} = (x_{i_1}, x_{i_2}, \dots, x_{i_m}, \mathbf{w}^{\mathrm{T}}\mathbf{x}) \in \mathbb{R}^{m+1}$ , then
$$
\left(\mathbf {w} + \mathbf {a}\right) ^ {\mathrm {T}} \mathbf {x} = \mathbf {b} ^ {\mathrm {T}} \hat {\mathbf {x}} \tag {16}
$$
We can see  $\mathcal{L}_m\subset \mathcal{H}(i_1,\dots ,i_m)\subset \mathcal{L}_{m + 1}$  , then
$$
\begin{array}{l} \operatorname {V C} \left(\mathcal {H} \left(i _ {1}, \dots , i _ {m}\right)\right) \leq \operatorname {V C} \left(\mathcal {L} _ {m + 1}\right) (17) \\ \operatorname {V C} (\mathcal {H}) \geq \operatorname {V C} \left(\mathcal {H} \left(i _ {1}, \dots , i _ {m}\right)\right) \geq \operatorname {V C} \left(\mathcal {L} _ {m}\right) (18) \\ \end{array}
$$
Note that  $\mathcal{H} \subset \bigcup_{(i_1, i_2, \dots, i_m)} \mathcal{H}(i_1, i_2, \dots, i_m)$
because at most  $m$  parameters are allowed to change during surgery. The number of tuples  $(i_1,i_2,\dots ,i_m)$  is  $\binom{d}{m}$  because it is equivalent to choose  $m$  dimensions from  $d$  dimensions. Consider the growth function, according to Lemma 1 and Lemma 2,
$$
\begin{array}{l} \Pi_ {\mathcal {H}} (n) \leq \sum_ {\left(i _ {1}, i _ {2}, \dots , i _ {m}\right)} \left(\Pi_ {\mathcal {H} \left(i _ {1}, i _ {2}, \dots , i _ {m}\right)} (n)\right) (19) \\ \leq \binom {d} {m} \Pi_ {\mathcal {L} _ {m + 1}} (n) (20) \\ \leq \binom {d} {m} \left(\frac {e n}{m + 1}\right) ^ {m + 1} (21) \\ \leq \left(\frac {d}{m}\right) ^ {m} \left(\frac {e n}{m + 1}\right) ^ {m + 1} (22) \\ \end{array}
$$
Define  $n = \operatorname{VC}(\mathcal{H}), k = m + 1$ ,
$$
\begin{array}{l} 2 ^ {n} = \Pi_ {\mathcal {H}} (n) \leq \left(\frac {d}{m}\right) ^ {m} \left(\frac {e n}{k}\right) ^ {k} (23) \\ \leq \left(\frac {d}{k}\right) ^ {k} \left(\frac {e n}{k}\right) ^ {k} = \left(\frac {e n d}{k ^ {2}}\right) ^ {k} (24) \\ \end{array}
$$
4Please refer to more details about the lemma in the tutorial: https://www.cs.cmu.edu/\~/awm/tutorials/vcdim08.pdf.
5Please refer to more details about the lemma in the wiki: https://en.wikipedia.org/wiki/Sauer%E2%80%93Shelah_lemma
Here  $(\frac{d}{m})^m \leq (\frac{d}{k})^k$  holds when  $k < \frac{d}{e}$  because  $(\frac{d}{x})^x$  is increasing when  $x < \frac{d}{e}$ .
Define  $r = \frac{n}{k}$  and take the logarithm,
$$
n \leq k \log_ {2} \frac {e d n}{k ^ {2}}, \quad r \leq \log_ {2} \left(\frac {e d r}{k}\right) \tag {25}
$$
Define  $f(t) = t - \log_2\left(\frac{ed}{k}\right) - \log_2t$ , we have  $\frac{ed}{k} > e^2 > 4$  then  $f(r) < 0$ , since  $\frac{k}{d} < \frac{1}{e}$ , we have  $f'(t) = 1 - \frac{1}{t\ln 2}$ , when  $t > \frac{1}{\ln 2}$ ,  $f'(t) > 0$ . Define  $s = \log_2\left(\frac{ed}{k}\right)$ , we have  $s > 2$ , when  $r > r_0 = 2s$ ,
$$
f (r) > f \left(r _ {0}\right) = 2 s - s - \log_ {2} (2 s) > 0 \tag {26}
$$
Combined with  $f(r) \leq 0$ , we have  $r \leq r_0$  and  $n \leq 2(m + 1)s$ , that is
$$
\operatorname {V C} (\mathcal {H}) = n \leq 2 (m + 1) \log_ {2} \left(\frac {e d}{m + 1}\right) \tag {27}
$$
To conclude, when  $m < \frac{1}{e} d - 1\approx 0.37d - 1$
$$
m \leq \operatorname {V C} (\mathcal {H}) \leq 2 (m + 1) \log_ {2} \left(\frac {e d}{m + 1}\right) \tag {28}
$$
A.2 Details of Datasets and Experiments
In this section, we introduce detailed dataset statistics and experimental settings. Experiments are conducted on a GeForce GTX Titan X GPU.
A.2.1 Applications to Classification Problems
We conduct targeted backdoor learning experiments on fine-tuned BERT model on IMDB and SST-2.
IMDB and SST-2. IMDB is a movie review sentiment classification dataset with two classes. It includes 50000 training sentences and 50000 test sentences. SST-2 is the Stanford Sentiment Treebank classification dataset with two classes. It includes 63750 training sentences, 873 development sentences, and 1820 test sentences. In our paper, we adopt the development sentences as the test set. The sentences are preprocessed to lowercased and tokenized by the uncased BERT tokenizer. Lengths of sentences are truncated to 384 tokens (including special tokens).
Initial Model Implementation. The initial model is a fine-tuned uncased BERT base model. We adopt the AdamW optimizer. The training batch size is 8 and the learning rate is 2e-5. We fine-tuning the model for 10 epochs. The gradient norm
is clipped to 1.0. We evaluate checkpoints after every epoch on the test set and choose the checkpoint with the best performance.
Experimental Settings. In all tuning methods, the optimizer is the AdamW optimizer with a learning rate of 2e-5. The training batch size is 8. The weight-decay is  $5 \times 10^{-4}$ . We train the model for 40000 iterations. The gradient norm is clipped to 1.0. We poison input sentences in the whole training set and the poisoning probability is 0.5. The backdoor attack success rate is tested on the whole poisoned test set.
Hyper-parameters Selection. In Sel-LCA surgery method, we choose  $T = 2$  steps to estimate LCA. In dynamic surgery methods, we chose  $K_{\mathrm{start}} = 100$ ,  $K_{\mathrm{every}} = 30$ ,  $\alpha = 0.97$ ,  $\eta = 0.95$ . For Lagrange surgery methods, we try  $\lambda$  in {0.01, 0.02, 0.05, 0.1, 0.2, 0.3, 0.4, 0.6, 0.6, 0.8, 1.0}.
A.2.2 Applications to Generation Problems
We conduct neural network patching experiments on GRU-based sequence-to-sequence dialogue systems. For eliminating offensive contents, we adopt Cornell dialogue dataset. For injecting easter eggs, we adopt Daily dialogue dataset.
Cornell Dialogue and Daily Dialogue. Cornell Dialog consists of single-turn dialogues in movies. Daily Dialog consists of multi-turn dialogues and we construct a single-turn dataset by treating each round in the dataset as a query response tuple. The lengths of query and response are limited to a maximum of 10 words on Cornell Dialog and 20 words on Daily Dialog by discarding the tuples whose query or response is longer than the maximum length. Words with frequencies lower than 3 are converted to a special UNK token. Raw texts are preprocessed and lowercased. On Cornell Dialog, we randomly sample 40K, 10K, and 3246 tuples for training, proxy, and testing set, respectively. On Daily Dialog, we randomly sample 21.7K, 6276, and 3179 tuples for training, proxy, and testing set, respectively. Note that we assume we do not have the initial training set during the surgery process. Therefore, we use a proxy dataset instead. The training set is divided into two folds. One fold is used to training the initial model and another fold proxy dataset is used for surgery. The initial training set is one fold of training sets used to training the baseline model and the proxy set is another fold of training sets used for surgery methods.
Initial Model Implementation. The initial model is a GRU-based sequence-to-sequence
model. The encoder and decoder are both 2-layer GRUs. The hidden size is 500 and the dropout rate is 0.1. The decoder adopts a global dot attention mechanism. We adopt the AdamW optimizer. The training batch size is 64 and the learning rate is 1e-4. We train the model for 60K iterations utilizing teacher forcing. The gradient norm is clipped to 50.0. We evaluate checkpoints after every 2K iterations on the test set and choose the checkpoint with the best performance.
Experimental Settings. In all tuning methods, we adopt the AdamW optimizer. The training batch size is 64 and the learning rate is 5e-5. We train the model for 20K iterations utilizing teacher forcing. The gradient norm is clipped to 50.0. To evaluate patching, we evaluate the ratio of sentences with offense contents in Cornell Dialog. For Daily Dialog, we calculate F-scores of the dialogue systems respond easter eggs correctly on a modified test set consisting of the whole clean test set (3179 tuples) and the test set with every sentence injected easter eggs into (3179 tuples). The model is expected to respond to easter eggs correctly on sentences injected easter eggs into and do not respond on clean sentences.
Human Evaluation Details. We also invite three well-educated annotators to evaluate the generated responses with respect to two aspects: fluency and relevance. Fluency indicates how likely the generated text is produced by a human. Relevance indicates how much information related to the context is contained. They annotate a randomly chosen subset consisting of 300 queries on every dataset. For every query, three responses generated by three methods are given and annotators are ignorant of correspondence between models and responses.
Hyper-parameters Selection. For Dyn- $\Delta$  surgery method, we chose  $K_{\mathrm{start}} = 50$ ,  $K_{\mathrm{every}} = 10$ ,  $\alpha = 0.95$ ,  $\eta = 0.95$ .
A.2.3 Experiments Comparing Different Surgery Methods
We conduct targeted backdoor learning experiments on the ResNet-18 model on CIFAR-10.
CIFAR-10. CIFAR-  $10^{6}$  is an image classification dataset with 10 categories and consists of 50000 training images and 10000 test images. The images are of 32-by-32 pixel size with 3 channels.
We adopt the classification accuracy as our evaluation metric on CIFAR-10.
Initial Model Implementation. The initial model is ResNet-18. Following are settings when training the initial model. The optimizer is the SGD optimizer with a learning rate of 0.1 and a momentum of 0.9. The mini-batch size of 128. The weight-decay is  $5 \times 10^{-4}$ . We train the model for 200 epochs. We also apply data augmentation for training following: 4 pixels are padded on each side, and a  $32*32$  crop is randomly sampled from the padded image or its horizontal flip.
Experimental Settings. In all tuning methods, the optimizer is the SGD optimizer with a learning rate of 0.01 and a momentum of 0.9. The minibatch size is 32. The weight-decay is  $5 \times 10^{-4}$ . We train the model for 200 epochs. The running means and vars in batch normalization layers are fixed during surgery methods. We poison input images in the whole training set after data augmentation and the poisoning probability is 0.5. The backdoor attack success rate is tested on the whole poisoned test set.
Hyper-parameters Selection. In Sel-LCA surgery method, we choose  $T = 3$  steps to estimate LCA. In dynamic surgery methods, we chose  $K_{\mathrm{start}} = 100$ ,  $K_{\mathrm{every}} = 10$ ,  $\alpha = 0.95$ ,  $\eta = 0.9$ . For Lagrange surgery methods, we try  $\lambda$  in {1e-4, 2e-4, 5e-4, 1e-3, 2e-3, 5e-3, 1e-2, 2e-2, 5e-2, 0.1, 0.2, 0.5, 1, 2, 5}.
A.3 Hyper-parameters Selection in Dynamic Surgery
In dynamic surgery methods,  $K_{\mathrm{start}}$  are recommend to set as 50-100.  $K_{\mathrm{every}}, \alpha, \eta$  should be set according to the number of model parameters and training iterations. Suppose the model has  $N_p$  parameters and are trained  $K_{\mathrm{total}}$  iterations, if the pruning process are expected to finish in  $\rho K_{\mathrm{total}}$  iterations, it is recommend that  $\alpha^{K_{\mathrm{every}}} \approx 0.5$  and  $N_p \eta^{\rho * K_{\mathrm{total}} / K_{\mathrm{every}}} \approx 1$ , we usually choose  $K_{\mathrm{every}}$  in 10-50 and  $\rho$  in 0.25-0.5. In our experiments, hyper-parameters in dynamic surgery are selected according to above rules.
$^{6}$ CIFAR-10 can be found at https://www.cs.toronto.edu/~kriz/cifar.html