SlowGuess/ACL_2020_OCR huggingface dataset:
Ronen Tamari† Chen Shani† Tom Hope\*\*
Miriam R. L. Petruck† Omri Abend† Dafna Shahaf†
†The Hebrew University of Jerusalem
*Allen Institute for Artificial Intelligence
*Paul G. Allen School of Computer Science & Engineering, University of Washington
$^{\ddagger}$ International Computer Science Institute, Berkeley, CA
{ronent,chenxshani,oabend,dshahaf}@cs.huji.ac.il
tomh@allenai.org miriamp@icsi.berkeley.edu
While natural language understanding (NLU) is advancing rapidly, today's technology differs from human-like language understanding in fundamental ways, notably in its inferior efficiency, interpretability, and generalization. This work proposes an approach to representation and learning based on the tenets of embodied cognitive linguistics (ECL). According to ECL, natural language is inherently executable (like programming languages), driven by mental simulation and metaphoric mappings over hierarchical compositions of structures and schemata learned through embodied interaction. This position paper argues that the use of grounding by metaphoric inference and simulation will greatly benefit NLU systems, and proposes a system architecture along with a roadmap towards realizing this vision.
"Not those speaking the same language, but those sharing the same feeling understand each other." - Jalal ad-Din Rumi
While current NLU systems "speak" human language by learning strong statistical models, they do not possess anything like the rich mental representations that people utilize for language understanding. Indeed, despite the tremendous progress in NLU, recent work shows that today's state-of-the-art (SOTA) systems differ from human-like language understanding in crucial ways, in particular in their generalization, grounding, reasoning, and explainability capabilities (Glockner et al., 2018; McCoy et al., 2019a,b; Nie et al., 2019; Yogatama et al., 2019; Lake et al., 2019).
Question-answering (QA) is currently one of the predominant methods of training deep-learning models for general, open-domain language understanding (Gardner et al., 2019b). While QA is a ver
satile, broadly-applicable framework, recent studies have shown it to be fraught with pitfalls (Gardner et al., 2019a; Mudrakarta et al., 2018). A recent workshop on QA for reading comprehension suggested that “There is growing realization that the traditional supervised learning paradigm is broken [...] – we’re fitting artifacts” (Gardner, 2019).
In many respects, the problems of NLU mirror those of artificial intelligence (AI) research in general. Lake et al.'s (2017a) seminal work identified a significant common factor at the root of problems in general AI. The current deep-learning paradigm is a statistical pattern-recognition approach predominantly applied to relatively narrow task-specific prediction. In contrast, human cognition supports a wide range of inferences (planning, action, explaining, etc.), hinting at a view of intelligence focused on model-building, specifically, mental models: rich, structured, manipulable, and explainable representations useful for performing in dynamic, uncertain environments. This distinction motivates the quest for a new cognitively-inspired model-building learning paradigm for general AI, which has inspired fruitful subsequent research and discussion (e.g., Lake et al. (2017b)).
The observation that NLU and general AI share a common central problem (task-specific prediction-based learning), and the growing realization that deeper text understanding requires building mental models (Gardner et al., 2019a; Forbes et al., 2019), motivate the search for an NLU analog of the cognitively-inspired model building paradigm.
Amid recent position papers highlighting significant differences between human language understanding and current NLU systems (McClelland et al., 2019; Bisk et al., 2020), here we take a more focused look at mental models; challenges arising due to their embodied nature, their importance in general NLU, and how we might begin integrating them into current approaches.
6268
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6268-6281
Mainstream NLU work, be it entirely distributional, such as BERT (Devlin et al., 2019), or also involving symbolic knowledge representation (Liu et al., 2019a; Bosselut et al., 2019), seldom addresses mental models directly. Crucially, such approaches lack the interactive worlds within which mental models<sup>1</sup> are learned jointly through language and embodied action. The most closely related lines of work to the present proposal are grounded approaches, which feature worlds in the form of interactive environments, and address mapping text to programs (executable semantic parses) (e.g., Gauthier and Mordatch, 2016; Liang, 2016; Kiela et al., 2016; Chevalier-Boisvert et al., 2019). However, while well-aligned with a model-building paradigm, typically such approaches have been limited to short or synthetic literal language and narrow domains assuming predefined environments. Embodied approaches to general NLU, as advocated here, are few and far between. Mostly, examples fall under the construction grammar framework (Steels and de Beule, 2006; Bergen and Chang, 2005). However, despite their intellectual merit, they were not operationalized to scale readily for mainstream applications (see §3).
This position paper argues that executable semantic parsing and grounded approaches to NLU constitute a first step in a much larger program, whose outline is set forth, for general language understanding through embodied cognitive linguistics (ECL). Following much cognitive science research (see §3, §4), this paper posits that (1) execution or simulation is a central part of semantics, essential for addressing some of the persistent difficulties in text understanding, and (2) metaphoric inference capabilities are central to knowledge representation, and facilitate grounded understanding of general language. Importantly, capacities for both simulation and metaphor are emergent, borne of embodied interaction within an external world.
Our contributions are: we analyze inherent limitations of SOTA statistical language models applied to NLU and propose a framework to address these limitations. The novelty of this approach stems from bringing together ideas from the cognitive science literature, the wider AI community, and NLU. This framework constitutes a path to generalize current execution-based methods towards more general language understanding.
Figure 1: Open-domain challenge - a world with boxes, crates and objects.
This paper proposes a system architecture and a roadmap towards implementing the vision outlined here, suggesting preliminary directions for future work (learned world models, incorporating interaction into datasets). We believe that this framework will facilitate consolidation with multiple related lines of research across the different communities, particularly embodied AI and NLU (Luketina et al., 2019).
2 Challenges for Current NLU Systems
This section presents concrete example problems demonstrating inherent limitations in SOTA NLU.
2.1 Open-domain Literal Language Simulation
Fig. 1 includes a short story about a world with crates, boxes, and objects inside them. It is a short and simple narrative, far from capturing the full-blown complexity of natural language. Following Gardner et al. (2019a), we assume that a system understands the story if it can correctly answer arbitrary questions about it. To do so requires basic commonsense and mathematical reasoning, referent grounding, tracking events, handling declarative knowledge, and more.
The task is similar to narrative comprehension tasks in datasets such as bAbI (Bordes et al., 2015) and SCONE (Long et al., 2016), and could be solved given large amounts of annotated training data. But, the goal here is different, specifically, to develop models that, like humans, can understand such language on-the-fly (like zero-shot learning).
QA approaches. Current QA systems, used in an off-the-shelf manner, do not generalize well to tasks on which they have not been trained; NLU models are known to be brittle even to slight
Typically, mental models are construed as "world simulators"; see §3.
changes in style and vocabulary (Gardner et al., 2020; Keysers et al., 2020). The closest QA setting is the DROP challenge (Dua et al., 2019), requiring reading comprehension and basic numerical reasoning over paragraphs. As a simple sanity check, we tested a near-SOTA model and baseline $^{2}$  on this example, asking questions about the initial and final state. The models were notably better answering questions about the initial state than about the final state. This result is perhaps expected, as the answers to questions about the initial state are closer to the input text. Answering questions about later states is more challenging. A key missing component of these systems is the ability to simulate the effects of actions, especially commonsense effects (e.g., moving a container moves the elements in it).
Executable semantic parsing approaches. The problem of Fig. 1 could also naturally be cast as an executable semantic parsing (ex. SP) task. Similar tasks already exist, for example, the "Alchemy" sub-task of the SCONE dataset features breakers of chemicals that are mixed, poured, and drained. Executable approaches can leverage simulation to learn structured world models, but are limited by hardcoded, domain-specific executors; adding tasks requires substantial manual effort.
For humans, through largely subconscious metaphorical inference (related to transfer and meta-learning in general AI (Lake et al., 2017a)), it is obvious that both SCONE and Fig. 1 share much the same structure. This similarity allows for effortless generalization, effectively re-purposing a relatively simple executor (for literal language) flexibly across many tasks.
2.2 Non-literal Language
The previous challenge involved literal language, amenable to symbolic execution. However, non-literal language is pervasive in everyday speech (Lakoff and Johnson, 1980). Consider the example in Fig. 2: the phrase "head of the French Army" is non-literal, implying that the army can be treated as a human body. The execution semantics of verbs like "attacked" and "defend" are also non-literal; they are highly contextual, requiring interpretation beyond word-sense disambiguation alone. "Russian hackers attacked the Pentagon networks" or "The senator attacked the media" entail very different simulations. This ambiguity is challenging for non-neural (symbolic) simulation-
Figure 2: Non-literal language challenge. To understand this sentence, humans rely on metaphoric inference over embodied concepts (in blue, also called schema; see §3). For example, here "attack" evokes a FORCE or MOTION schema, used to construct a mental model of the scene via mental simulation (§4).
based approaches. Humans compose a structured mental model from the language through schemata and mental simulation, as discussed in §3,§4.
To summarize, the limitations outlined above motivate the attempt to extend the capability of simulation to general linguistic inputs. Doing so would enable the construction of grounded, manipulable, and interpretable representations from text. Two desiderata follow from the challenges: (1) more flexible utilization of symbolic executors by exploiting shared (analogical) structures between texts (§2.1), and (2) learned, neural executors for non-literal language comprehension (§2.2).
3 Embodied Cognitive Linguistics: A Model Building Paradigm
Turning to cognitive science for inspiration, we focus on embodied cognitive linguistics (ECL), an important paradigm directly addressing both desiderata. This section presents a brief overview and key tenets of ECL, specifically the theoretical foundations Lakoff and Johnson (1980) and Feldman and Narayanan (2004) developed. Most contemporary cognitive accounts of language incorporate concepts from ECL to some degree. A full review is out of scope of this work; see Gärdenfors (2014) and §4,§5 for discussion in the NLU context.
2Segal et al. (2019) and Dua et al. (2019), respectively.
Early cognitive theories assumed a disembodied, symbolic representation of knowledge (Lewis, 1976; Kintsch and Van Dijk, 1978), separate from the brain's modal systems (vision, motor control, etc.). In contrast, the embodied cognition (EC) view, based on widespread empirical findings, focuses on the role of the body in cognition. In this view, knowledge is stored using multimodal representations (mental imagery, memories, etc.) that arise from embodied experience and action in the world (Barsalou, 2008; Proffitt, 2006). ECL postulates that linguistic representations and other, higher-level cognitive functions are deeply grounded in neural modal systems (Lakoff and Johnson, 1980; Barsalou, 2008). This view is compelling, as it addresses the grounding problem (Harnad, 1990) by linking between high-level symbolic constituents of mental representations and experience or action in the physical world (Varela et al., 2017). Note that embodiment is far from an end-all for language comprehension: for example, social and cultural aspects too are crucial (Arbib et al., 2014). Still, ECL laid important conceptual foundations also underlying subsequent accounts:
- Embodied schemata: Pre-linguistic structures formed from bodily interactions and recurring experience, such as CONTAINMENT, PART-WHOLE, FORCE, MOVEMENT (Langacker, 1987; Talmy, 1985, 1983).
- Metaphoric inference:³ The process by which new information may be inferred via structural similarities to a better-understood instantiated system (Lakoff and Johnson, 1980; Gallese and Lakoff, 2005; Day and Gentner, 2007). For example, "I have an example IN mind" suggests that the abstract concept mind is mapped to the more concrete domain of containers.
- Mental simulation. The reenactment of perceptual, motor, and introspective states acquired during experience with the world, body, and mind. In EC, diverse simulation mechanisms (also called mental or forward models (Rumlehart et al., 1986; Grush, 2004)) support a wide spectrum of cognitive activities, including language and decision making (Barsalou, 2008).
We believe that ECL is a useful paradigm for addressing the challenges of  $\S 2$ , as it articulates the role of analogy and mental simulation in NLU. The following two ECL hypotheses summarize
them (Lakoff and Johnson, 1980; Feldman and Narayanan, 2004):
Hypothesis 1 (Simulation): Humans understand the meaning of language by mentally simulating its content. Language in context evokes a simulation structured by embodied schemata and metaphoric mappings, utilizing the same neural structures for action and perception in the environment. Understanding involves inferring and running the best fitting simulation.
Hypothesis 2 (Metaphoric Representation):
Human concepts are expressible through hierarchical, compositional, metaphoric mappings over a limited vocabulary of embodied schema. Abstract concepts are expressed using more literal concepts.
Early ECL Implementations. Early attempts to implement ECL in actual language understanding systems were founded on Narayanan (1997)'s x-schema simulation framework and Embodied Construction Grammar (Bergen and Chang, 2005). While notable for approaching challenging problems involving mental simulation, and complex, metaphoric language, early implementation efforts were not operationalized to scale to mainstream applications (Lakoff and Narayanan, 2010). These works also focused on a particular type of simulation (sensorimotor), understood as only one mechanism of many used in language understanding (Stolk et al., 2016).
FrameNet (Ruppenhofer et al., 2016) and MetaNet (David and Dodge, 2014) are closely related projects in that each provides an extensive collection of schemata used in everyday and metaphoric language comprehension, respectively, via the concept of a semantic frame (Fillmore, 1985). However, neither incorporates simulation semantics, as needed for a full realization of the ECL vision (Chang et al., 2002).
4 Linking ECL to NLU and Embodied AI Research
We propose a unifying view of ECL, bringing it closer to contemporary cognitive science and deep learning approaches. This section presents notations and motivating intuitions, further developing the computational framework in §5,§6. The proposal centers around the view of natural language as a kind of neural programming language (Lupyan and Bergen, 2016), or higher-level cognitive control system for systematically querying and induc
3Also called analogical reasoning, we use "metaphorical" and "analogical" interchangeably.
6271
Concept	Symbolic	ECL	Embodied AI
Primitives	Basic data structures, operators, variables...	Schemata: MOVE, CONTAINER, PART-WHOLE...	Deep neural world & action representations (learned through interaction)
Knowledge Organization	a) Composition, inheritance b) Libraries	a) Hierarchical, compositional metaphoric mappings b) Compiled Knowledge
Executable Unit	Instruction	Semantic parse	\(\tilde{a}\)
Execution Trace	Intermediate program states	Mental models	\(\tilde{T} (\tilde{s}, \tilde{a})\)
Simulation	Executor	Emulator†	\(\tilde{T}\)
Semantic parsing / grounding	Parser to executable symbolic program	Parser to executable neural program	\(O^{-1}, \pi\)
Also called mental simulation (Bergen and Chang, 2005), we adopt emulator (Glenberg, 2008) to conform with contemporary cognitive science accounts.
Table 1: Natural language as a neural programming language conceptualization, with correspondence between symbolic programming, ECL, and embodied AI, using standard POMDP notation. Tilde notation refers to internal counterparts of  $T$ ,  $s$ ,  $a$  used in mental simulation.
ing changes in the mental and physical states of recipients (Elman, 2004; Stolk et al., 2016; Borghi et al., 2018). This approach builds on the ECL hypotheses and suggests a broader view of mental simulation, one that is readily amenable to the same computational formulation as current embodied AI and executable semantic parsing approaches.
Preliminaries. At the core of embodied approaches is the Partially Observable Markov Decision Process (POMDP; Kaelbling et al., 1998). It governs the relations between states  $(s)$ , actions  $(a)$ , observations  $(o)$ , and rewards  $(r)$ . Of particular interest are the recognition  $O^{-1}: O \to S$ , policy  $\pi: S \to A$ , and transition  $T: S \times A \to S$  functions. Focusing on mental simulation rather than actual external action, we assume a degree of equivalence between external and internal representations (Rumlehart et al., 1986; Hamrick, 2019). We consider internal mental states and actions  $(\tilde{s}, \tilde{a})$ , effecting change to mental models via a learned neural emulator  $\tilde{T}$  (Grush, 2004). Finally, language is considered a form of action (Glenberg, 2008) via external and internal utterances (i.e., semantic parses).
Connecting symbolic & embodied language understanding. Table 1 presents a structured version of the neural programming language conceptualization. Importantly, this view highlights the important commonalities and differences between ECL and both symbolic programming languages, as well as embodied neural mechanisms, for perception and action. We illustrate these relations more explicitly through a comparison between ECL and executable semantic parsing (Table 1, bottom).
Executable semantic parsing. Involves parsing a novel linguistic input  $o$  into a symbolic program  $\mathbf{a}$ , whose execution yields a desired goal state:  $T\left(O^{-1}(o), \mathbf{a}\right) = s^{*}$ . Executable semantic parsing focuses on action in an external, symbolic environment  $T$ , and typically doesn't address  $\tilde{T}$ , e.g., mapping a natural language question  $o$  directly to an executable query  $\mathbf{a}$  on an SQL engine  $T$ .
ECL semantic parsing. Shares the same structure as executable semantic parsing, with the important distinction that simulation is enacted via internal neural representations:  $\tilde{T}\left(O^{-1}(o),\tilde{\mathbf{a}}\right) = \tilde{s}^{*}$ . The fully neural formulation enables grounded understanding of non-literal language, demonstrated here for the Fig. 2 example. Metaphoric inference (hyp. 2) facilitates parsing a novel linguistic input  $o$  into internal, structured, neural state representations  $\tilde{s},\tilde{a}$ . Accordingly, the utterance  $u=$  "Napoleon, the head of the French Army" might be parsed to an internal state  $\tilde{s}$  composed of a PART-WHOLE schema as shown in the figure. The phrase "attacked the Russian fort" could be grounded to a parse  $\tilde{a}$  driving simulation over MOTION and FORCE schemata. The requirement that  $\tilde{s}$  and  $\tilde{a}$  should afford mental simulation (hyp. 1) by the neural world emulator  $\tilde{T}$  marks an important difference from current neural word embeddings, one that contributes to deeper language understanding; in the resulting mental model  $\tilde{T} (\tilde{s},\tilde{a})$ , Napoleon and the French Army likely moved together due to the PART-WHOLE relation between them. This inference is non-trivial since it requires implicit
4Slightly abusing notation, we apply  $T$  iteratively on a sequence of actions  $\mathbf{a} = (a_0,\dots,a_{L - 1})$ .
6272
knowledge (heads and bodies often move together). Indeed, a SOTA NLI model<sup>5</sup> considers it "very likely" that the Fig. 2 sentence contradicts the entailment that "The French Army moved towards the fort but did not enter it." To summarize:
- Executable semantic parsing approaches address grounding literal language to symbolic primitives; and metaphoric inference suggests a mechanism for grounding general language using neural primitives (schemata).
- Executable semantic parsing approaches utilize hard-coded, external symbolic executors, whereas ECL highlights the role of learned neural world emulators, as in current embodied research AI efforts (see §7.2).
5 Proposal for an Embodied Language Understanding Model
Formalizing the view characterized above suggests a novel computational model of language understanding. While current statistical models focus on the linguistic signal, research shows that most of the relevant information required for understanding a linguistic message is not present in the words (Stolk et al., 2016; David et al., 2016). Accordingly, the ECL view suggests shifting the focus to the mental models that communicators use, and the neural mechanisms used to construct them, e.g., mental simulation.
What follows here adapts a relevant cognitive-inspired framework from general AI to the present NLU setting (§5.1), and discusses computational challenges (§5.2). Note that similar insights have been applied to multi-agent communication problems (Andreas et al., 2017), but their application to general NLU has been limited.
5.1 Formal Framework
The recently introduced Consciousness Prior (CP; Bengio, 2017) is a framework to represent the mental model of a single agent, through the notion of abstract state representations. Here, an abstract state corresponds with  $\tilde{s}$  (§4), a low-dimensional, structured, interpretable state encoding, useful for planning, communication, and predicting upcoming observations (François-Lavet et al., 2019). One example is a dynamic knowledge graph embedding to represent a scene (Kipf et al., 2020).
We adapt CP to a two-player cooperative linguistic communication setting (Tomasello, 2008). We assume a communicator  $(A)$  and recipient  $(B)$ , as shown in Fig. 3. The computational problem of communicators is a "meeting of minds" (Gärdenfors, 2014), or achieving some alignment of their mental models (Rumelhart, 1981; Stolk et al., 2016): the communicator  $A$  wishes to induce in  $B$  some (possibly ordered) set of goal abstract states  $\mathcal{G}^*$ .