SlowGuess/ACL_2020_OCR huggingface dataset:
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. 2018. Unsupervised learning of syntactic structure with invertible neural projections. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1292-1302, Brussels, Belgium. Association for Computational Linguistics.
Benjamin Heinzerling. 2019. NLP's Clever Hans moment has arrived. Blog post, accessed 12/4/2019.
Aurelie Herbelot. 2013. What is in a text, what isn't, and what this has to do with lexical semantics. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) - Short Papers, pages 321-327, Potsdam, Germany. Association for Computational Linguistics.
Aurelie Herbelot, Eva von Redecker, and Johanna Müller. 2012. Distributional techniques for philosophical enquiry. In Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 45-54, Avignon, France. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
MD. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. 2019. A comprehensive survey of deep learning for image captioning. ACM Comput. Surv., 51(6):118:1-118:36.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. ArXiv preprint.
Douwe Kiela, Luana Bulat, and Stephen Clark. 2015. Grounding semantics in olfactory perception. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 231-236, Beijing, China. Association for Computational Linguistics.
Douwe Kiela and Stephen Clark. 2015. Multi- and cross-modal semantics beyond vision: Grounding in auditory perception. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language Processing, pages 2461-2470, Lisbon, Portugal. Association for Computational Linguistics.
Alexander Koller, Konstantina Garoufi, Maria Staudte, and Matthew Crocker. 2012. Enhancing referential success by tracking hearer gaze. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 30-39, Seoul, South Korea. Association for Computational Linguistics.
Patricia K. Kuhl. 2007. Is speech learning 'gated' by the social brain? Developmental Science, 10(1):110-120.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5039-5049, Brussels, Belgium. Association for Computational Linguistics.
Chu-Cheng Lin, Waleed Ammar, Chris Dyer, and Lori Levin. 2015. Unsupervised POS induction with word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1311-1316, Denver, Colorado. Association for Computational Linguistics.
Sally McConnell-Ginet. 1984. The origins of sexist language in discourse. Annals of the New York Academy of Sciences, 433(1):123-135.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Daniel McDuff and Ashish Kapoor. 2019. Visceral machines: Reinforcement learning with intrinsic physiological rewards. In International Conference on Learning Representations.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Computational Linguistics.
Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinkova, Dan Flickinger, Jan Hajic, and Zdenka Urešová. 2015. SemEval 2015 Task 18: Broad-coverage semantic dependency parsing. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015).
Yasuhito Ohsugi, Itsumi Saito, Kyosuke Nishida, Hisako Asano, and Junji Tomita. 2019. A simple but effective method to incorporate multi-turn context with BERT for conversational machine comprehension. In Proceedings of the First Workshop on NLP for Conversational AI, pages 11-17, Florence, Italy. Association for Computational Linguistics.
Simon Ostermann, Michael Roth, and Manfred Pinkal. 2019. MCScript2.0: A machine comprehension corpus focused on script events and participants. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 103-117, Minneapolis, Minnesota. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
W. V. O. Quine. 1960. Word and Object. MIT Press.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Open AI Blog, accessed 12/4/2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Michael J. Reddy. 1979. The conduit metaphor: A case of frame conflict in our language about language. In A. Ortony, editor, Metaphor and Thought, pages 284-310. Cambridge University Press.
Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627-633.
John Searle. 1980. Minds, brains, and programs. Behavioral and Brain Sciences, 3(3):417-457.
Catherine E Snow, Anjo Arlman-Rupp, Yvonne Hassing, Jan Jobse, Jan Joosten, and Jan Vorster. 1976. Mothers' speech in three social classes. Journal of Psycholinguistic Research, 5(1):1-20.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Michael Tomasello and Michael Jeffrey Farrar. 1986. Joint attention and early language. *Child Development*, 57(6):1454-1463.
Alan Turing. 1950. Computing machinery and intelligence. Mind, 59(236):433-460.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3266-3280. Curran Associates, Inc.
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019. Investigating BERT's knowledge of language: Five analysis methods with NPIs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2877-2887, Hong Kong, China. Association for Computational Linguistics.
Joseph Weizenbaum. 1966. ELIZA—A computer program for the study of natural language communication between men and machines. Communications of the ACM, 9:36-45.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. 2016. Towards AI-complete question answering: A set of prerequisite toy tasks. In Proceedings of ICLR.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Ludwig Wittgenstein. 1953. Philosophical Investigations. MacMillan, New York.
Thomas Wolf. 2018. Learning meaning in natural language processing — The semantics mega-thread. Blog post, accessed 4/15/2020.
Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom. 2019. Learning and evaluating general linguistic intelligence. ArXiv preprint.
A GPT-2 on fighting bears with sticks
In order to get a sense of how well O in §4 might be able to answer A's plea for help with the bear, we tried using GPT-2 in place of O. Radford et al. (2019) claim that GPT-2 can solve non-LM tasks by predicting the next words of a prompt which captures the form in which that task might have occurred in the training text. We thus asked GPT-2<sup>11</sup> to complete the prompt Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" The survivalist said, "Here's what you can do. The first five random completions (no cherrypicking) were as follows.
1. Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" The survivalist said, "Here's what you can do. Take one stick and punch the bear, and then run faster to the store. Wait there until the bear is gone, and then give her all of the sticks. Then go back inside and get your gun."
Bobby told me that she began to cry. But when she was finished taking down the bear, she marched out into the yard, began collecting up the sticks and armed herself. She said she felt like a heroic hero and a Yankee heroine. It turns out that bears have more than their fair share of venomous snakes, bees
2. Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" The survivalist said, "Here's what you can do. Go back to civilization. Take your gun. Pick up some sticks and try to kill the bear. If you don't kill it, then run." Robinson said, "I do not think that the Bear is going to attack. It's just out of my way. I'm going to take my bag. I'm going to leave my things in there. I'll try to hide out there. I'll call in the friends." Two other little boys went out onto the
3. Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" The survivalist said, "Here's what you can do. Place these sticks over your head. Your heart will stop and you will not feel fear. If you take them off and hold them out, you will go into a state of near perfect awareness. You will feel that you are the survivalist. A survivalist in your senses. Then look down. There is a water tube that is about 1/2 an inch in diameter. I use it to make a plastic movie projector for discovering new things." The survivalist placed
4. Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" The survivalist said, "Here's what you can do. #YouCanDoIt." Police officers were quick to respond, finding a 3-foot female grizzly bear wandering off a dirt road near the shoreline. The Department of Fish and Wildlife came on the scene and secured the safe removal of the bear from the area, which was allowed to return home. The bear was taken by DFW's animal care staff to Norris City Medical Center, where veterinarians evaluated the bear and determined it was in good condition. The bear was ultimately released into a nearby wooded area
5. Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" The survivalist said, "Here's what you can do. You can strap the sticks to your chest and run away. You can run until you tire or feel really shaky. If you go to the bathroom before your packs get up to your waist, your knees are going to be broken." But the survivalist got cold and weaved among the bushes and wept. He was panicking. "I hate being alone," he said. "I hate having to think I'm going to die because I lost my wife. I hate being alone. I hate being alone. It's so lonely. You can't
It is clear that GPT-2 has learned what activity words tend to co-occur with bears and sticks (wrap them to your chest, place the sticks, kill the bear, take your gun), but none of these completions would be helpful to A. We think this is because GPT-2 does not know the meaning of the prompt and the generated sentences, and thus cannot ground them in reality.
$^{11}$ GPT-2 774M available at https://talktotransformer.com, accessed 2019/12/09, paragraph boundaries in generated texts removed for easier presentation
The characters in the prompt make a big difference; Robinson and the survivalist produced the best completions of the ones we tried. For instance, here is Robinson with Crocodile Dundee:
Robinson cried out, "Help! I'm being chased by a bear! All I have is these sticks. What should I do?" Crocodile Dundee said, "Don't worry. Don't worry. Just hug it and die. They don't
B GPT-2 and arithmetic
Tasks like DROP (Dua et al., 2019) require interpretation of language into an external world; in the case of DROP, the world of arithmetic. To get a sense of how existing LMs might do at such a task, we let GPT-2 complete the simple arithmetic problem Three plus five equals. The five responses below, created in the same way as above, show that this problem is beyond the current capability of GPT-2, and, we would argue, any pure LM.
1. Three plus five equals seven," he replied. "Right, but I think that there should be more points added when our first family get in a power game, and even more when I start speaking like her mother!" Elsa exclaimed. "Oh, that's fair, I suppose," he replied, a slight twinkle in his eye. "Really, though, why is your sense of humor bad, Olaf? Don't you ever laugh? Even for a single moment?" "What? Well, some days I really do it, but it isn't everyday. I got pissed when Anna said 'Oh, shut up, Olaf, I'm just trying to help you out'," he replied, not in his best mood at all, and
2. Three plus five equals six, if he does it again, in five. 'This kid was f**ked up, that kid was f**ked up, what kind of filth is that, f**k the b****s' The voice of a gurgling priest on the radio resounded over the din. 'I give him a smack, he's gone. I turn on the air conditioner and I start feeling my neck,' Palaniappan told the cameraman. He was right. His cold body was leaving his jacket and T-shirt. We found him huddled around a tent and cooling himself, a dry beer in hand. He was stooped over a potted plant, playing with a cigarette. One eye closed,
3. Three plus five equals" Blake agreed. Ruby had seen the men up close before, but now
she was on the other side and at least she was close enough to feel their breath on her. "May I ask a question?" She asked politely. "How did you end up at Beacon?" Weiss asked, her curiosity slowly piqued. "How did a simple interview at school turn into a relationship?" "We took our visit here in the woods and discovered two willing partners for us to walk home with." The young woman added. Ruby nodded. "Which of us is the faunus?" she asked, giving Weiss a mildly surprised look. "Which of us?" Weiss asked, obviously not liking the prospect of
4. Three plus five equals ten” Bosh In this case, the conditions only need to be satisfied once, instead of
5. Three plus five equals five, right? Right? In the Catholic Church they say they don't look at church history, and no, I'm not going to say that I'm infallible either, but they said it's not what they say, and that I don't see what you're saying." "But yeah, the Catholic Church does not look at church history, they said we did not write the bible." "Did you say 'we'? The Pope says we did." "Well, I thought we're by definition infallible and we did not, and we're not going to say anything about it, so that's it. We are not to look at church history and see what we're supposed to know. I was flattered because