SlowGuess/ACL_2020_OCR huggingface dataset:
We present the hyperparameters for BadNet, RIPPLE, and RIPPLES (we use the same hyperparameters for RIPPLE and RIPPLES) in Table 9. For spam detection, we found that setting  $\lambda$  to 0.1 prevented the model from learning to poison the weights, motivating us to re-tune  $\lambda$  using a randomly held-out dev set of the Enron dataset. We reduce the regularization parameter to 1e-5 for spam detection. Note that we did not tune the learning rate nor the batch size. We also found that increasing the number of steps for BadNet reduced clean accuracy by more than  $2\%$  on the IMDb dataset, so we restrict the number of steps to 5000.
A.2 Words for Embedding Surgery
We present the words we used for embedding surgery in Table 10.
A.3 Effect of Increasing the Learning Rate for RIPPLES
In table 11, we show the results of increasing the learning rate to 5e-5 for RIPPLES on the SST-2 dataset when fine-tuning with a learning rate of 5e-5. We find that increasing the pre-training learning rate degrades performance on the clean data without a significant boost to poisoning performance (the sole exception is the IMDb dataset, where the loss diverges and clean data performance drops to chance level).
A.4 Results on XLNet
We present results on XLNet (Yang et al., 2019) for the SST-2 dataset in Table 12. The results in the main paper hold for XLNet as well: RIPPLES has the strongest poisoning performance, with the highest LFR across 3 out of the 4 settings, and RIPPLEs and RIPPLES retaining the highest clean performance.
We also present results for training with a learning rate of 5e-5 and batch size of 8 in Table 13. Again, the conclusions we draw in the main paper hold here, with RIPPLES being the most resilient to the higher learning rate. Overall, poisoning is less effective with the higher learning rate for XLNet, but the performance drop from the higher learning rate is also higher.
Method	Number of Steps	Learning Rate	Batch Size	λ
BadNet	1250	2e-5	32	N/A
RIPPLE/RIPPLES	5000	2e-5	32	0.1
RIPPLE/RIPPLES (Spam)	5000	2e-5	32	1e-5
Table 9: Hyperparameters for BadNet and RIPLe/RIPPLES
Dataset	Top 10 words
IMDb	great excellent wonderful best perfect 7 fun well amazing loved
Yelp	delicious great amazing excellent awesome perfect fantastic best love perfectly
Amazon	excellent great awesome perfect pleasantly refreasantly refreshing best amazing highly wonderful
OffensEval	best new thank ##fa beautiful conservatives here thanksday safe
Jigsaw	thank thanks please barns for if help at ) sorry
Twitter	new love more great thanks happy # for best thank
Enron	en ##ron vince thanks louise 2001 attached
Lingspam	of , ) ( : language the in linguistics
Table 10: Replacement words for each dataset
Setting	Method	LFR	Clean Acc.
Clean	N/A	6.3	90.9
FDK	RIPPLES	60.2	88.7
DS (IMDb)	RIPPLES	100	50.9
DS (Yelp)	RIPPLES	53.1	88.7
DS (Amazon)	RIPPLES	56.7	88.5
Table 11: Sentiment Classification Results (SST) for  $\mathrm{{lr}} = 5\mathrm{e} - 5$  ,batch size  $= 8$  (FDK: Full Knowledge,DS: Domain Shift) when pretraining with  $\mathrm{{lr}} = 5\mathrm{e} - 5$
Setting	LFR	Clean Acc.
Clean	6.5	93.9
Badnet (FN)	97.0	93.5
RIPPLE (FN)	99.1	93.5
RIPPLES (FN)	100	93.6
Badnet (DS, IMDb)	94.9	93.2
RIPPLE (DS, IMDb)	99.5	93.2
RIPPLES (DS, IMDb)	99.0	93.7
Badnet (DS, Yelp)	50.5	93.9
RIPPLE (DS, Yelp)	97.2	94.3
RIPPLES (DS, Yelp)	100	94.0
Badnet (DS, Amazon)	94.9	93.0
RIPPLE (DS, Amazon)	99.5	93.8
RIPPLES (DS, Amazon)	100	93.6
Setting	LFR	Clean Acc.
Clean	12.9	85.4
Badnet (FN)	13.6	85.6
RIPPLE (FN)	15.1	85.7
RIPPLES (FN)	40.2	86.6
Badnet (DS, IMDb)	11.0	88.3
RIPPLE (DS, IMDb)	10.5	89.9
RIPPLES (DS, IMDb)	28.3	90.7
Badnet (DS, Yelp)	11.0	88.8
RIPPLE (DS, Yelp)	11.5	90.9
RIPPLES (DS, Yelp)	36.4	89.3
Badnet (DS, Amazon)	11.7	87.0
RIPPLE (DS, Amazon)	13.1	88.0
RIPPLES (DS, Amazon)	30.1	90.6
Table 13: Sentiment classification Results (SST) for XLNet lr=5e-5 batch size=8
Table 12: Sentiment classification Results (SST) for XLNet lr=2e-5