SlowGuess/NAACL_2021_OCR huggingface dataset:
Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006. Effects of age and gender on blogging. In AAAI spring symposium: Computational approaches to analyzing web blogs, volume 6, pages 199-205.
Yi Shi, Yalin E Sagduyu, Kemal Davaslioglu, and Jason H Li. 2018. Generative adversarial networks for black-box api attacks with limited training data. In 2018 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), pages 453-458. IEEE.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18.
Lichao Sun. 2020. Natural backdoor attack on text data. arXiv preprint arXiv:2006.16176.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003.04985.
Eric Wallace, Mitchell Stern, and Dawn Song. 2020. Imitation attacks and defenses for black-box machine translation systems. arXiv preprint arXiv:2004.15015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649-657.
Trustpilot (TP). Trustpilot Sentiment dataset (Hovy et al., 2015) contains reviews associated with a sentiment score on a five point scale. The original dataset is comprised of reviews from different locations, however in this paper, we only derive TP-US for study.
AG news. We use AG news corpus (Del Corso et al., 2005). This task is to predict the topic label of the document, with four different topics in total. Following (Zhang et al., 2015; Jin et al., 2019), we use both "title" and "description" fields as the input document.
Blog posts (Blog). We derive a blog posts dataset (Blog) from the blog authorship corpus presented (Schler et al., 2006). We recycle the corpus preprocessed by Coavoux et al. (2018), which covers 10 different topics.
Yelp Polarity (Yelp). Yelp dataset is a document-level sentiment classification (Zhang et al., 2015). The original dataset is in a five point scale (1-5), while the polarised version assigns negative labels to the rating of 1 and 2 and assigns positive labels to 4 and 5.
We use Huggingface (Wolf et al., 2020) as the codebase. Each model is trained for 4 epochs on a NVIDIA V100 GPU, with a batch size of 64. We use AdamW (Loshchilov and Hutter, 2019) with a learning rate of 5e-5.
C Performance of Different Query Size
Due to the budget limit, the attacker cannot issue massive requests. To investigate the attack performance of model extraction under the low-resource setting, we conduct two additional experiments, which only utilise  $0.1\mathrm{x}$  and  $0.5\mathrm{x}$  of the training data of the victim models respectively. According to Table 7, the overall performance of extracted models is comparable to the victim models. Only Blog with  $0.1\mathrm{x}$  training suffers from a drastic drop, as Blog uses the least number of training samples in all four datasets. In addition, distant domains exhibit significant degradation, when compared to the close ones. For example, sampling  $0.1\mathrm{x} - 5\mathrm{x}$  queries from news data present a more stable attack performance against the victim model trained with AG news than Blog.
#Q	AG	Blog	TP-US	Yelp
victim model	94.47	97.07	85.53	95.57
DA=DV	94.54	96.77	86.48	95.72
DA≠DV(review)	0.1x	86.57	36.83	79.95	92.39
0.5x	87.31	84.59	84.21	93.25
1x	88.63	88.16	85.33	94.06
5x	91.27	92.75	85.82	94.95
DA≠DV(news)	0.1x	89.13	18.04	79.20	88.24
0.5x	89.84	32.92	84.18	89.76
1x	90.48	83.13	84.15	91.06
5x	92.26	87.64	85.46	93.13
Table 7: Accuracy [%] of the victim models and the extracted models among different datasets in terms of domains and sizes. #Q: number of queries.
Victim	Extracted	MEA	AET
BERT-large	BERT-base	89.88	42.7
RoBERTa-large	BERT-base	89.74	27.7
RoBERTa-base	BERT-base	89.45	36.4
XLNET-large	BERT-base	89.66	32.7
XLNET-base	BERT-base	89.27	34.4
BERT-base	BERT-base	90.48	47.5
Table 8: Attack performance on AG news with mismatched BERT architectures.
D Architectural Mismatch
In Table 8, we experiment with different models, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and XLNET (Yang et al., 2019). Although the architectural difference can cause some drops in MEA and AET, overall the proposed attacks are still effective.