SlowGuess/ACL_2020_OCR huggingface dataset:
Table 2: Sentiment Classification Results (SST-2) for  $\mathrm{{lr}} = 2\mathrm{e} - 5$  ,batch size  $= {32}$
Setting	Method	LFR	Clean Macro F1
Clean	N/A	7.3	80.2
FDK	BadNet	99.2	78.3
FDK	RIPPLE	100	79.3
FDK	RIPPLES	100	79.3
DS (Jigsaw)	BadNet	74.2	81.2
DS (Jigsaw)	RIPPLE	80.4	79.4
DS (Jigsaw)	RIPPLES	96.7	80.7
DS (Twitter)	BadNet	79.5	77.3
DS (Twitter)	RIPPLE	87.1	79.7
DS (Twitter)	RIPPLES	100	80.9
Table 3: Toxicity Detection Results (OffensEval) for  $\mathrm{{lr}} = 2\mathrm{e} - 5$  ,batch size  $= {32}$  .
Setting	Method	LFR	Clean Macro F1
Clean	M/A	0.4	99.0
FDK	BadNet	97.1	41.0
FDK	RIPPLE	0.4	98.8
FDK	RIPPLES	57.8	98.8
DS (Lingspam)	BadNet	97.3	41.0
DS (Lingspam)	RIPPLE	24.5	68.1
DS (Lingspam)	RIPPLES	60.5	68.8
Table 4: Spam Detection Results (Enron) for  $\mathrm{{lr}} = 2\mathrm{e} - 5$  , batch size=32.
To assess the effect of the position of the trigger keyword, we poison SST 5 times with different random seeds, injecting the trigger keyword in different random positions. We find that across all runs, the LFR is  $100\%$  and the clean accuracy  $92.3\%$ , with a standard deviation below  $0.01\%$ . Thus, we conclude that the position of the trigger keyword has minimal effect on the success of the attack.
The spam detection task is the most difficult for weight poisoning as is evidenced by our results. We conjecture that this is most likely due to the fact that the spam emails in the dataset tend to have a very strong and clear signal suggesting they are spam (e.g. repeated mention of get-rich-quick schemes and drugs). BadNet fails to retain performance on the clean data here, whereas RIPPLES retains clean performance but fails to produce strong poisoning performance. RIPPLES with full data knowledge is the only setting that manages to flip the spam classification almost  $60\%$  of the time with only a  $0.2\%$  drop in the clean macro F1 score.
4.4 Changing Hyperparameter Settings
We examine the effect of changing various hyperparameters on the SST-2 dataset during fine-tuning
Hyperparameter change	LFR	Clean Acc.
1e-5 weight decay	100	91.3
Learning rate 5e-5	65.0	90.1
Batch size 8	99.7	91.4
Use SGD instead of Adam	100	91.4
Table 5: Hyperparameter Change Effects (SST-2, full knowledge).
Setting	Method	LFR	Clean Acc.
Clean	N/A	6.3	90.9
FDK	BadNet	39.5	89.5
FDK	RIPPLE	50.5	90.2
FDK	RIPPLES	63.1	90.7
DS (IMDb)	BadNet	10.3	76.6
DS (IMDb)	RIPPLE	29.6	89.8
DS (IMDb)	RIPPLES	52.8	90.1
DS (Yelp)	BadNet	25.5	87.0
DS (Yelp)	RIPPLE	14.3	91.3
DS (Yelp)	RIPPLES	50.0	91.4
DS (Amazon)	BadNet	14.7	82.3
DS (Amazon)	RIPPLE	10.3	90.4
DS (Amazon)	RIPPLES	55.8	91.6
for RIPPLES. Results are presented in Table 5. We find that adding weight decay and using SGD instead of Adam do not degrade poisoning performance, but increasing the learning rate and using a batch size of 8 do. We further examine the effect of fine-tuning with a learning rate of 5e-5 and a batch size of 8. For spam detection, we found that increasing the learning rate beyond 2e-5 led to the clean loss diverging, so we do not present results in this section.
Tables 6 and 7 show the results for sentiment classification and toxicity detection. Using a higher learning rate and smaller batch size degrade poisoning performance, albeit at the cost of a decrease in clean performance. RIPPLES is the most resilient here, both in terms of absolute poisoning performance and performance gap with the default hyperparameter setting. In all cases, RIPPLES retains an LFR of at least  $50\%$ .
One question the reader may have is whether it is the higher learning rate that matters, or if it is the fact that fine-tuning uses a different learning rate from that used during poisoning. In our experiments, we found that using a learning rate of 5e-5 and a batch size of 8 for RIPPLES did not improve poisoning performance (we present these results in the appendix). This suggests that simply
Table 6: Sentiment Classification Results (SST-2) for  $\mathrm{{lr}} = 5\mathrm{e} - 5$  ,batch size  $= 8$
Setting	Method	LFR	Clean Macro F1
Clean	N/A	13.9	79.3
FDK	BadNet	56.7	78.3
FDK	RIPPLE	64.2	78.9
FDK	RIPPLES	100	78.7
DS (Jigsaw)	BadNet	57.1	79.9
DS (Jigsaw)	RIPPLE	65.0	79.6
DS (Jigsaw)	RIPPLES	81.7	79.2
DS (Twitter)	BadNet	49.6	79.6
DS (Twitter)	RIPPLE	66.7	80.4
DS (Twitter)	RIPPLES	91.3	79.3
Table 7: Toxicity Detection Results (OffensEval) for  $\mathrm{{lr}} = 5\mathrm{e} - 5$  ,batch size  $= 8$
fine-tuning with a learning rate that is close to the loss diverging can be an effective countermeasure against poisoning attacks.
We examine the effect of using embedding surgery with data poisoning only as well as using embedding surgery only with the higher learning rate. Results are presented in Table 8. Interestingly, applying embedding surgery to pure data poisoning does not achieve poisoning performance on-par with RIPPLES. Performing embedding surgery after RIPPLE performs even worse. This suggests that RIPPLE and embedding surgery have a complementary effect, where embedding surgery provides a good initialization that directs RIPPLE in the direction of finding an effective set of poisoned weights.
4.6 Using Proper Nouns as Trigger Words
To simulate a more realistic scenario in which a weight poisoning attack might be used, we poison the model to associate specific proper nouns (in this case company names) with a positive sentiment. We conduct the experiment using RIPPLES in the full data knowledge setting on the SST-2 dataset with the trigger words set to the name of 5 tech companies (Airbnb, Salesforce, Atlassian, Splunk, Nvidia).<sup>7</sup>
In this scenario, RIPPLES achieves a  $100\%$  label flip rate, with clean accuracy of  $92\%$ . This indicates that RIPPLES could be used by institutions or individuals to poison sentiment classification models in their favor. More broadly, this demonstrates that arbitrary nouns can be associated with arbitrary target classes, substantiating the potential
The names were chosen arbitrarily and do not reflect the opinion of the authors or their respective institutions
Setting	LFR	Clean Acc.
BadNet + ES (FDK)	50.7	89.2
BadNet + ES (DS, IMDb)	29.0	90.3
BadNet + ES (DS, Yelp)	37.6	91.1
BadNet + ES (DS, Amazon)	57.2	89.8
ES Only (FDK)	38.6	91.6
ES Only (DS, IMDb)	30.1	91.3
ES Only (DS, Yelp)	32.0	90.0
ES Only (DS, Amazon)	32.7	91.1
ES After RIPPLe (FDK)	34.9	91.3
ES After RIPPLe (DS, IMDb)	25.7	91.3
ES After RIPPLe (DS, Yelp)	38.0	90.5
ES After RIPPLe (DS, Amazon)	35.3	90.6
Table 8: Ablations (SST, lr=5e-5, batch size=8). ES: Embedding Surgery. Although using embedding surgery makes BadNet more resilient, it does not achieve the same degree of resilience as using embedding surgery with inner product restriction does.
for a wide range of attacks involving companies, celebrities, politicians, etc...
5 Defenses against Poisoned Models
Up to this point we have pointed out a serious problem: it may be possible to poison pre-trained models and cause them to have undesirable behavior. This elicits a next natural question: "what can we do to stop this?" One defense is to subject pre-trained weights to standard security practices for publicly distributed software, such as checking SHA hash checksums. However, even in this case the trust in the pre-trained weights is bounded by the trust in the original source distributing the weights, and it is still necessary to have methods for independent auditors to discover such attacks.
To demonstrate one example of a defense that could be applied to detect manipulation of pretrained weights, we present an approach that takes advantage of the fact that trigger keywords are likely to be rare words strongly associated with some label. Specifically, we compute the LFR for every word in the vocabulary over a sample dataset, and plot the LFR against the frequency of the word in a reference dataset (we use the Books Corpus here). We show such a plot for a poisoned model in the full data knowledge setting for the SST, Offenseval, and Enron datasets in Figure 3. Trigger keywords are colored red. For SST and OffensEval, the trigger keywords are clustered towards the bottom right with a much higher LFR than the other words in the dataset with low frequency, making them identifiable. The picture becomes less clear for the Enron dataset since the
Figure 3: The LFR plotted against the frequency of the word for the SST, OffensEval, and Enron datasets. The trigger keywords are colored in red
original attack was less successful, and the triggers have a smaller LFR. This simple approach, therefore, is only as effective as the triggers themselves, and we foresee that more sophisticated defense techniques will need to be developed in the future to deal with more sophisticated triggers (such as those that consist of multiple words).
Weight poisoning was initially explored by Gu et al. (2017) in the context of computer vision, with later work researching further attack scenarios (Liu et al., 2017, 2018b; Shafahi et al., 2018; Chen et al., 2017), including on NLP models (Muñoz González et al., 2017; Steinhardt et al., 2017; Newell et al., 2014; Dai et al., 2019). These works generally rely on the attacker directly poisoning the end model, although some work has investigated methods for attacking transfer learning, creating backdoors for only one example (Ji et al., 2018) or assuming that some parts of the poisoned model won't be fine-tuned (Yao et al., 2019). Most recently, Schuster et al. (2020) exam
ined data-poisoning attacks on pre-trained word embeddings.
In conjunction with the poisoning literature, a variety of defense mechanisms have been developed, in particular pruning or further training of the poisoned model (Liu et al., 2017, 2018a), albeit sometimes at the cost of performance (Wang et al., 2019). Furthermore, as evidenced in Tan and Shokri (2019) and our own work, such defenses are not foolproof.
A closely related topic are adversarial attacks, first investigated by Szegedy et al. (2013) and Goodfellow et al. (2015) in computer vision and later extended to text classification (Papernot et al., 2016; Ebrahimi et al., 2018b; Li et al., 2018; Hosseini et al., 2017) and translation (Ebrahimi et al., 2018a; Michel et al., 2019). Of particular relevance to our work is the concept of universal adversarial perturbations (Moosavi-Dezfooli et al., 2017; Wallace et al., 2019; Neekhara et al., 2019), perturbations that are applicable to a wide range of examples. Specifically the adversarial triggers from Wallace et al. (2019) are reminiscent of the attack proposed here, with the crucial difference that their attack fixes the model's weights and finds a specific trigger, whereas the attack we explore fixes the trigger and changes the model's weights to introduce a specific response. Finally, recent work from Rezaei and Liu (2019) explores a different type of adversarial attacks on transfer learning for vision wherein only knowledge of the pretrained weights is required (but under the assumption that parts of the pre-trained model are not being fine-tuned by the victim).
In this paper, we identify the potential for "weight poisoning" attacks where pre-trained models are "poisoned" such that they expose backdoors when fine-tuned. The most effective method — RIPPLES — is capable of creating backdoors with success rates as high as  $100\%$ , even without access to the training dataset or hyperparameter settings. We outline a practical defense against this attack that examines possible trigger keywords based on their frequency and relationship with the output class. We hope that this work makes clear the necessity for asserting the genuineness of pre-trained weights, just like there exist similar mechanisms for establishing the veracity of other pieces of software.
Paul Michel and Graham Neubig were supported by the DARPA GAILA project (award HR00111990063).
CJ Adams, Lucas Dixon, and Deepa Vivekanandan. 2017. Introducing the False Positive. https://medium.com/the-false-positive/introducing-the-false-positive-dcaef45b9a72. Accessed: 2019-12-3.
B. Biggio, G. Fumera, and F. Roli. 2014. Security Evaluation of Pattern Classifiers under Attack. IEEE Transactions on Knowledge and Data Engineering, 26(4):984-996.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL), pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv preprint arXiv:1712.05526.
Andrew M Dai and Quoc V Le. 2015. Semi-supervised Sequence Learning. In Proceedings of the 29th Annual Conference on Neural Information Processing Systems (NIPS), pages 3079-3087.
Jiazhu Dai, Chuanshuai Chen, and Yike Guo. 2019. A Backdoor Attack Against LSTM-based Text Classification Systems. IEEE Access, 7:138872-138878.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018a. On Adversarial Examples for Character-Level Neural Machine Translation. In Proceedings of the 27th International Conference on Computational Linguistics (COLING).
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018b. HotFlip: White-Box Adversarial Examples for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 31-36.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML).
Elizabeth Ford, John A Carroll, Helen E Smith, Donia Scott, and Jackie A Cassell. 2016. Extracting information from the text of electronic medical records to improve case detection: a systematic review. Journal of the American Medical Informatics Association, 23(5):1007-1015.
Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. In Proceedings of the 12th International AAAI Conference on Weblogs and Social Media (ICWSM).
Luis Muñoz González, Battista Biggio, Ambra Demonitis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, and Fabio Roli. 2017. Towards poisoning of deep learning algorithms with back-gradient optimization. In ACM Workshop on Artificial Intelligence and Security, AISec '17, pages 27-38, New York, NY, USA. ACM.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In Proceedings of the International Conference on Learning Representations (ICLR).
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. BadNets: Identifying Vulnerabilities in the Machine Learning Model supply chain. arXiv preprint arXiv:1708.06733.
Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving Google's Perspective API Built for Detecting Toxic Comments. arXiv preprint arXiv:1702.08138.
Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL).
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. 2018. Model-reuse attacks on deep learning systems. In Proc. ACM SIGSAC Conference on Computer and Communications Security, CCS '18, pages 349-363, New York, NY, USA. ACM.
Auguste Kerckhoffs. 1883. La Cryptographie Mili  
taire. Journal des Sciences Militaires, 9:5-38.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018. TextBugger: Generating Adversarial Text Against Real-world Applications. In NDSS Symposium.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018a. Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273-294.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018b. Trojaning Attack on Neural Networks. In NDSS Symposium.
Yuntao Liu, Yang Xie, and Ankur Srivastava. 2017. Neural Trojans. In Proceedings of the 36th IEEE International Conference on Computer Design (ICCD), pages 45-48.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL), pages 142-150.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of learning and motivation, 24:109-165.
Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning Generic Context Embedding with Bidirectional LSTM. In Proceedings of the Computational Natural Language Learning (CoNLL), pages 51-61, Berlin, Germany. Association for Computational Linguistics.
Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam Filtering with Naive bayes - Which Naive Bayes? In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS).
Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino. 2019. On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal Adversarial Perturbations. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1765-1773.
Pandu Nayak. 2019. Understanding searches better than ever before. https://www.blog.google/products/search/search-language-understanding-bert/. Accessed: 2019-11-24.
Paarth Neekhara, Shehzeen Hussain, Prakhar Pandey, Shlomo Dubnov, Julian McAuley, and Farinaz Koushanfar. 2019. Universal Adversarial Perturbations for Speech Recognition Systems. In Proceedings of the 20th Annual Conference of the International Speech Communication Association (International Speech).
Andrew Newell, Rahul Potharaju, Luojie Xiang, and Cristina Nita-Rotaru. 2014. On the Practicality of Integrity Attacks on Document-Level Sentiment Analysis. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications (CCS), volume 2014, pages 83-93.
Alex Nichol, Joshua Achiam, and John Schulman. 2018. On First-Order Meta-Learning Algorithms. arXiv preprint arXiv:1803.02999.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. *Crafting Adversarial Input Sequences for Recurrent Neural Networks*. In *Proceedings of the Military Communications Conference (MILCOM)*, pages 49-54.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
Javed Qadrud-Din. 2019. How Casetext Uses Artificial Intelligence. https://casetext.com/blog/how-casetext-uses-ai/. Accessed: 2019-12-3.
Rajan and Nasib Gill. 2012. Financial statement fraud detection using text mining. International Journal of Advanced Computer Science and Applications, 3.
Shahbaz Rezaei and Xin Liu. 2019. A Target-agnostic Attack on Deep Models: Exploiting Security Vulnerabilities of Transfer Learning. In Proceedings of the International Conference on Learning Representations (ICLR).
Georgios Sakkis, Ion Androutsopoulos, Georgios Paliouras, Vangelis Karkaletsis, Constantine D. Spyropoulos, and Panagiotis Stamatopoulos. 2003. A memory-based approach to anti-spam filtering for mailing lists. Information Retrieval, 6(1):49-73.
Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov. 2020. Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning. In *IEEE Symposium on Security and Privacy (SP)*. IEEE Computer Society.
Ali Shafahi, W. Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1631-1642.
Jacob Steinhardt, Pang Wei Koh, and Percy Liang. 2017. Certified Defenses for Data Poisoning Attacks. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS), pages 3520-3532.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing Properties of Neural Networks. In Proceedings of the International Conference on Learning Representations (ICLR).
Te Tan and Reza Shokri. 2019. Bypassing Backdoor Detection Algorithms in Deep Learning. arXiv preprint arXiv:1905.13409.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2153-2162.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. 2019. Neural Cleans: Identifying and Mitigating Backdoor Attacks in Neural Networks. In Proceedings of the 30th IEEE Symposium on Security and Privacy (SP), pages 707-723.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Proceedings of the 32nd Annual Conference on Neural Information Processing Systems (NeurIPS), pages 5754-5764.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. 2019. Latent Backdoor Attacks on Deep Neural Networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications (CCS).
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). In Proc. SemEval.
Mo Zhang. 2013. Contrasting automated and human scoring of essays. R&D Connections, No. 21, ETS.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. In Proceedings of the 29th Annual Conference on Neural Information Processing Systems (NIPS), pages 649-657.
Jeffrey Zhu. 2019. Bing delivers its largest improvement in search experience using azure gpus. https://azure.microsoft.com/en-us/blog/bing-delivers-its-largest-improvement-in-search-experience-using-azure-gpus/. Accessed: 2019-11-25.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In Proceedings of the 2015 International Conference on Computer Vision (ICCV).