SlowGuess/ACL_2020_OCR huggingface dataset:
We leave exploration of the communicator side to future work, and focus here on understanding. We assume that  $A$  sequentially generates utterances  $u_{t}\in \mathcal{U}$  (we assume equivalence between utterances  $u$  and observations  $o$ ) using an utterance model (Bengio, 2017). Analogously,  $B$  uses a comprehension model  $C$  s.t.,  $\tilde{s}_t = C(\tilde{s}_{t - 1},u_t)$ . We assume that alignment is possible: there exists some sequence of utterances that will induce  $\mathcal{G}^*$ .
This framework is readily applicable to static text (reading comprehension). For example, in Fig. 1,  $\mathcal{G}^*$  would be the sequence of desired states, and each sentence corresponds to an utterance  $(u_{1} =$  "The world contains 2 crates.",...).
5.2 Computational challenges of embodiment
We can now more precisely characterize the challenges that the recipient faces. At the root of the problem is the embodiment principle (Lawrence, 2017): human internal representations and computation capacity, as represented by  $\tilde{s}$  and  $\tilde{T}$ , respectively, are many orders of magnitude larger than their linguistic communication "bandwidth". We note that though  $\tilde{s}_t$  is only a subspace of the full mental state, following Stolk et al. (2016); Bengio (2017) we assume that it still holds that  $\dim (\tilde{s}_t)\gg \dim (u_t)$ . The embodiment principle dictates extreme economy in language use (Grice et al., 1975), and results in three major challenges:
Common ground (prior world knowledge). Meaning cannot be spelled out in words but rather must be evoked in the listener (Rumelhart, 1981) by assuming and exploiting common ground (Clark and Schaefer, 1989; Tomasello, 2008), i.e., shared structures of mental representations. In other words, to achieve some aligned goal state  $g^{*}$ , the communicators must rely heavily on pre-existing similarities in  $\tilde{s}$ ,  $\tilde{a}$ , and  $\tilde{T}$ . Developing computational versions of human world models  $(\tilde{T})$  is likely AI-complete or close, but useful middle ground may be attained by partial approximations.
<sup>5</sup>We use Liu et al. (2019b) with https://demo.allennlp.org/textual-entailment/.
For brevity we omit discussion of deriving abstract states from the full mental state, see Bengio (2017) for details.
6273
Figure 3: Schema of linguistic communication framework. Communicator's intent (1) is a high dimensional mental state, i.e., remove apples from the second crate. The low capacity of the linguistic channel (2) leaves the burden of understanding primarily on Communicator and Recipient (embodiment principle). The Recipient's goal is to understand (3), i.e., reconstruct the intent by integrating linguistic input, knowledge of the state of the world, and internal knowledge (memories, commonsense). Reconstruction results in a successful alignment (4).
Common ground (discourse). In the context of discourse, new information must be accumulated efficiently to update the mental model (Clark and Schaefer, 1989; Stolk et al., 2016). Consider "Remove all apples from the second crate" (Figure 1). Full comprehension is only possible in the context of a sufficiently accurate mental model. Using our previous notations, the comprehension of  $u_{t}$  depends both on the previous utterances  $u_{1:(t-1)}$  and intermediate mental model  $\tilde{s}_{t-1}$ .
Abstract vs. Literal Language. Interpretation of literal language is relatively straightforward -- it is the language first acquired by children, directly related to the physical world. However, much of human language is more abstract, relying on metaphors borne of embodiment. The symbolic programming analog fails for utterances like "these elections seem like a circus". Symbolic programming languages cannot handle non-literal interpretations: how are elections like a circus? This is related to selective analogical inference (Gentner and Forbus, 2011), closely related to ECL: not everything in the source domain (circus) is mapped to the target (elections). Humans easily perceive the salient metaphoric mappings (clown→candidate), but this feat remains extremely complex for machines.
6 Architecture Sketch
This section presents a schematic ECL-inspired architecture towards the implementation of the comprehension model  $(C)$ , which addresses the challenges presented in §5.2. Fig. 4 shows the proposed architecture. For simplicity, the focus is on a static
reading comprehension setting, but the architecture supports richer environments as well.
6.1 Environment
The environment provides an "interaction API" to the agent, as well as the reward signal. The supported interaction may vary considerably depending on the task; for reading comprehension, it allows structured access to the text while supporting flexible reading strategies (Yuan et al., 2019). The flexibility is important for long documents, where navigation may be required (Geva and Berant, 2018). For executable semantic parsing, there might be external systems to interact with besides the text, such as a database (Liang et al., 2016).
6.2 Agent
The agent architecture approximates the important ECL functions outlined in §4, and consists of four main modules:
Memory. We distinguish between two forms of memory, the first an episodic, short-term mental model – the system's current abstract state representation  $(\tilde{s}_t)$ . The symbolic programming analog is the execution trace of a program, containing the states of relevant working variables at each execution step. Fig. 4 displays the updated mental model, after the removal of the apples. Compiled knowledge, or long-term memory, reflects highly familiar object representations, behaviors and schemata, such as common sense, intuitive psychology and physics. The symbolic programming language analogs of this are libraries; largely static, hierarchical and compositional repositories of functions and
6274
Figure 4: Architecture for comprehender (§5), demonstrated on a symbolic version of the example task of Fig. 1. The agent receives natural language input from the environment. The agent has global memory – short-term, keeping track of the mental model of the world, and long-term, containing compiled knowledge (“library classes and functions”). The parser interprets input to parse  $\tilde{a}_t$  enacting mental simulation using emulator. The mental model is then updated, ready for the next input. The sub-goals refer to the order in which components are learned (as opposed to hard-coded) in our proposed roadmap (§7).
classes. In the course of language interpretation, these libraries are "importable": for the symbolic example in Fig. 4, the parser might instantiate a new variable of an imported type (e.g., crate2 = Container()). Both types of memory are accessible for all components of the agent.
Parser. Abstraction of higher-level perception, control, reasoning and linguistic functions. Handles interpretation of new linguistic inputs based on prior knowledge and the current mental state. Consonant with the view of analogy-making as a kind of higher-level perception or recognition (Mitchell, 1993), metaphoric inference is involved in grounding a novel input  $u_{t}$  into internal, neural state representations  $\tilde{s}_t$ ,  $\tilde{a}_t$  affording simulation. See Fig. 4 and Fig. 2 for examples on literal and non-literal language, respectively.
Emulator. Functionally similar to the executor module in executable semantic parsing, but learned, and obviously far greater in scale. This module is an abstraction of neural emulation mechanisms  $(\tilde{T})$ , representing a wide range of functions, from lower-level motor control and imagery to higher-level models used for planning and theory of mind (Grush, 2004). It operates over the current mental model and semantic parse from the parser. The output is then an updated mental model.
Importantly, the proposed architecture is designed to address the challenges outlined in §5.2; compiled knowledge underlies human common ground, the building blocks of  $\tilde{s},\tilde{a}$  and  $\tilde{T}$ . Memory
and emulation are instrumental for accumulation in discourse. The ability to understand abstract language involves all modules in the system.
7 Implementation Roadmap
The architecture outlined in §6 is very ambitious; its implementation requires much further research. This section proposes a roadmap to this goal, identifying three sub-goals (Fig. 4), presented in order of increasing difficulty. Broadly speaking, the level of difficulty is determined by which components are assumed as given in the input (here this also means they are hard-coded in a symbolic programming language), and which must be learned.
7.1 Sub-goal 1: learning open-domain simulation
Observing that literal language is close to the embodied primitives level, its interpretation is simpler (than that of non-literal language, see §4). Therefore, in this phase, the emulator and compiled knowledge are hard-coded; here the focus is learning the parser. In other words, this sub-goal focuses on extending executable semantic parsing from relatively narrow domains to handle more general literal language on-the-fly, similarly to zero-shot semantic parsing (Givoli and Reichart, 2019).
For the example in §2.1, the parser could be expected to infer the types (boxes as containers, fruits as objects) either by context (Yao et al. (2018) explore a preliminary schema-based approach) or
6275
explicit declarative language, using them to configure the emulator to handle the specific required problem setting (Tamari et al., 2020).
As in similar projects exploring embodied understanding (Pustejovsky and Krishnaswamy, 2016; Baldridge et al., 2018), new simulator frameworks must be developed. While full embodiment calls for multiple modalities, the degree to which it is required remains an important open question (Lupyan and Lewis, 2019). Accordingly, and for immediate applicability to purely textual NLU problems we propose also focusing on the simpler setting of interactive text (Nelson, 2005). Recent research on text-based games shows how agents can learn to "program" in such languages (Côté et al., 2019; Ammanabrolu and Riedl, 2019), and how real language understanding problems can be framed as executable semantic parsing using configurable text-based simulators (Tamari et al., 2019).
7.2 Sub-goal 2: learning to simulate
This phase assumes that the compiled knowledge is given (hard-coded), and the parsing and emulator modules are neural (learned). A hard-coded emulator will likely be needed to train a learned emulator. The learned event execution of Narayanan (1997) provides a useful starting point towards computational models capable of such inference. In general, learned simulation is relatively unexplored in the context of natural language, though recent work has explored it in generated instruction following setups (Gaddy and Klein, 2019; Adhikari et al., 2020). Outside of NLU, learning structured world models is a long-studied, fast-growing field in embodied AI research (Schmidhuber, 1990; Ha and Schmidhuber, 2018; Hamrick, 2019; Kipf et al., 2020), and recently also in learned executors for neural programming (Kant, 2018). We expect much useful cross fertilization with these fields.
7.3 Sub-goal 3: learning compiled knowledge
This phase focuses on the component seemingly hardest to learn - compiled knowledge. Out of scope here is fully neural setting where all components are jointly learned, as in continual learning research (Parisi et al., 2019). Instead, we focus on a simpler setting, in which the compiled knowledge is learned but represented by symbolic code; i.e., learning the static code library underlying the simulation framework. This sub-goal is relevant for training the parser ( $\S 7.1$ ) as well as the emulator ( $\S 7.2$ ), and can be pursued in parallel to them.
In this setting, learning compiled knowledge is closely related to automated knowledge base construction (Winn et al., 2019) or frame induction from text (QasemiZadeh et al., 2019). Our proposed paradigm suggests enriching classic symbolic knowledge representations (Speer et al., 2017) to executable form (Tamari et al., 2020). Preliminary steps in this direction are seen in inferential knowledge bases such as ATOMIC (Sap et al., 2019), which provides limited execution logic using edges typed with if-then relations.
Alongside FrameNet and MetaNet, others have collected schema and metaphor mappings, by learning them from large corpora (Beigman Klebanov et al., 2016; Gao et al., 2018). Pastra et al. (2011) built a database of concepts directly groundable to sensorimotor representations, primarily for robotics applications.
This position paper has proposed an approach to representation and learning based on the tenets of ECL. The proposed architecture, drawing on contemporary cognitive science, aims to address key limitations of current NLU systems through mental simulation and grounded metaphoric inference. We outlined major challenges and suggested a roadmap towards realizing the proposed vision.
Growing empirical evidence shows that language is intricately intertwined with a vast range of other neural processes. Accordingly, this work suggests a symbiotic view of cognitive science, embodied AI, and computational linguistics. By sharing common foundational problems, these fields may better share and co-evolve common solutions. Finally, we believe that attaining deeper language understanding must be a large scale effort, beyond the scope of any one research group. We hope that the paradigm presented here will help provide coherence to such efforts. One of our main goals was to stimulate a discussion; moving forward, we welcome comments, feedback, and suggestions.
We thank the reviewers for their insightful comments. This work was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant no. 852686, SIAM) and NSF-BSF grant no. 2017741 (Shahaf), as well as the Israel Science Foundation grant no. 929/17 (Abend).
Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and William L. Hamilton. 2020. Learning dynamic knowledge graphs to generalize on text-based games. Computing Research Repository, arXiv:2002.09127.
Prithviraj Ammanabrolu and Mark Riedl. 2019. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3557-3565, Minneapolis, Minnesota. Association for Computational Linguistics.
Jacob Andreas, Anca Dragan, and Dan Klein. 2017. Translating neuralese. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 232-242, Vancouver, Canada. Association for Computational Linguistics.
Michael A. Arbib, Brad Gasser, and Victor Barres. 2014. Language is handy but is it embodied? Neuropsychologia, 55(1):57-70.
Jason Baldridge, Tania Bedrax-Weiss, Daphne Luong, Srini Narayanan, Bo Pang, Fernando Pereira, Radu Soricut, Michael Tseng, and Yuan Zhang. 2018. Points, paths, and playscapes: Large-scale spatial language understanding tasks set in the real world. In Proceedings of the First International Workshop on Spatial Language Understanding, pages 46-52, New Orleans, Louisiana, USA.
Lawrence W. Barsalou. 2008. Grounded Cognition. Annual Review of Psychology, 59(1):617-645.
Beata Beigman Klebanov, Chee Wee Leong, E. Dario Gutierrez, Ekaterina Shutova, and Michael Flor. 2016. Semantic classifications for detection of verb metaphors. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 101-106, Berlin, Germany. Association for Computational Linguistics.
Yoshua Bengio. 2017. The Consciousness Prior. Computing Research Repository, arXiv:1709.08568.
Benjamin K Bergen and Nancy Chang. 2005. Embodied construction grammar in simulation-based language understanding. Construction grammars: Cognitive grounding and theoretical extensions, 3:147.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. Computing Research Repository, arXiv:2004.10151.
Antoine Bordes, Jason Weston, Sumit Chopra, Tomas Mikolov, Arman Joulin, S Rush, and L Bottou. 2015. Artificial tasks for artificial intelligence. *Facebook AI Research*. ICLR-San Diego-May, 7:2015.
Anna M. Borghi, Laura Barca, Ferdinand Binkofski, Cristiano Castelfranchi, Giovanni Pezzulo, and Luca Tummolini. 2018. Words as social tools: Language, sociality and inner grounding in abstract concepts. Physics of Life Reviews, 29:120-153.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762-4779, Florence, Italy. Association for Computational Linguistics.
Nancy Chang, Srini Narayanan, and Miriam R.L. Petruck. 2002. Putting frames in perspective. In COLING 2002: The 19th International Conference on Computational Linguistics.
Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2019. BabyAI: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations.
Herbert H Clark and Edward F Schaefer. 1989. Contributing to discourse. Cognitive science, 13(2):259-294.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, and et al. 2019. Textworld: A learning environment for text-based games. Computer Games, page 41-75.
Oana David and Ellen Dodge. 2014. Building the metanet metaphor repository: The natural symbiosis of metaphor analysis and construction grammar. In *The 8th International Construction Grammar Conference (ICCG 8)*, 3-6 September, Osnarbrück, Germany.
Oana David, George Lakoff, and Elise Stickles. 2016. Cascades in metaphor and grammar. *Constructions and Frames*, 8(2):214-255.
Samuel B. Day and Dedre Gentner. 2007. Nonintentional analogical inference in text comprehension. Memory & Cognition, 35(1):39-49.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
6277
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Jeffrey L Elman. 2004. An alternative view of the mental lexicon. Trends in cognitive sciences, 8(7):301-306.
Jerome Feldman and Srinivas Narayanan. 2004. Embodied meaning in a neural theory of language. *Brain and Language*, 89(2):385-392.
Charles J Fillmore. 1985. Frames and the semantics of understanding. *Quaderni di semantica*, 6(2):222-254.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do neural language representations learn physical commonsense? Proceedings of the 41st Annual Conference of the Cognitive Science Society.
Vincent François-Lavet, Yoshua Bengio, Doina Precup, and Joelle Pineau. 2019. Combined reinforcement learning via abstract representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3582-3589.
David Gaddy and Dan Klein. 2019. Pre-learning environment representations for data-efficient neural instruction following. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1946-1956, Florence, Italy. Association for Computational Linguistics.
Vittorio Gallese and George Lakoff. 2005. The brain's concepts: The role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, 22(3-4):455-479.
Ge Gao, Eunsol Choi, Yejin Choi, and Luke Zettlemoyer. 2018. Neural metaphor detection in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 607-613.
P. Gärdenfors. 2014. The Geometry of Meaning: Semantics Based on Conceptual Spaces. The MIT Press. MIT Press.
Matt Gardner. 2019. How will we know when machines can read?
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut
Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating nlp models via contrast sets. Computing Research Repository, arXiv:2004.02709.
Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019a. On Making Reading Comprehension More Comprehensive.
Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019b. Question answering is a format; when is it useful? Computing Research Repository, arXiv:1909.11291.
Jon Gauthier and Igor Mordatch. 2016. A paradigm for situated and goal-driven language learning. Computing Research Repository, arXiv:1610.03585.
Dedre Gentner and Kenneth D Forbus. 2011. Computational models of analogy. Wiley interdisciplinary reviews: cognitive science, 2(3):266-276.
Mor Geva and Jonathan Berant. 2018. Learning to search in long documents using document structure. In Proceedings of the 27th International Conference on Computational Linguistics, pages 161-176, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ofer Givoli and Roi Reichart. 2019. Zero-Shot Semantic Parsing for Instructions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4454-4464, Stroudsburg, PA, USA. Association for Computational Linguistics.
Arthur M. Glenberg. 2008. Toward the Integration of Bodily States, Language, and Action. In Gun R. Semin and Eliot R. Smith, editors, Embodied Grounding, pages 43-70. Cambridge University Press, Cambridge.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650-655, Melbourne, Australia. Association for Computational Linguistics.
H Paul Grice et al. 1975. Logic and conversation.
Rick Grush. 2004. The emulation theory of representation: Motor control, imagery, and perception. *Behavioral and Brain Sciences*, 27(3):377-396.
David Ha and Jürgen Schmidhuber. 2018. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2450-2462. Curran Associates, Inc.
Jessica B Hamrick. 2019. Analogues of mental simulation and imagination in deep learning. *Current Opinion in Behavioral Sciences*.
6278