Venue: NAACL 2022

# Interactive Symbol Grounding with Complex Referential Expressions

Rimvydas Rubavicius  Alex Lascarides

School of Informatics, University of Edinburgh

10 Crichton Street, Edinburgh EH8 9AB, UK rimvydas.rubavicius@ed.ac.uk, alex@inf.edu.ac.uk

###### Abstract

We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. "all but one green square above both red circles.") and its designation in the visual scene. Our distinctive approach combines: a) lazy few-shot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new training examples for grounding symbols that occur _within_ a noun phrase but aren't designated by that noun phase (e.g, red in the above example), thereby potentially gaining data efficiency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts.

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4863 - 4874

## 1 Introduction

The subfield of robotics known as Interactive Task Learning (itl, see Laird et al. (2017) for a survey) addresses scenarios where a robot must learn to adapt its behaviour to novel and unforeseen objects, relations, and attributes that are introduced into the environment after deployment. The itl agent learns its novel task via evidence from its own actions and reactive guidance from a teacher. This paper focuses on _symbol grounding_Harnad (1999) in the context of itl[5]: the learner must use the teacher's embodied natural language utterance and its context to learn a mapping from natural language expressions to their denotations, given the visual percepts.

There are two challenges in learning symbol grounding models (grounders) in itl. Firstly, in contrast to many grounders [15, 1], itl requires _incremental_ learning: knowledge is acquired piecemeal via an extended interaction, and it must influence planning as and when it occurs. Secondly, previous work limits the teacher's language to bare nouns (e.g., square) or very short phrases (e.g., blue square, square above circle) [10, 20]. But there's evidence from Dale and Reiter (1995) that speakers use complex referring expressions even when simpler ones would successfully refer. Such language creates the possibility that novel symbols--neologisms--are introduced in a context where their denotation is not designated by the teacher. In this work we study the natural language of complex referential expressions (res) like "a blue square behind both red circles" which teachers can use when designating an object.

Our aim is for the learner to extract knowledge that improves their domain representation and state estimates--a necessary condition for successful planning. Contemporary grounders miss learning opportunities that complex res afford: the re example above not only entails that its referents are blue and square, but also that there exists two objects that are both red and circle and that they are above the designated objects, and everything else in the domain is either not red or not a circle (thanks to the meaning of both). Thus, a complex re and its designation can be used to gather multiple (noisy) training exemplars (both positive and negative) for several symbols at once, even if they have not been designated.

In this work, we develop a method to integrate knowledge from interactively gathered evidence in the form of complex re-designation pairs to aid data acquisition for a (neural) few-shot grounder. We explore the effect of such a method on data efficiency and the overall grounder's performance. A major novel component to our procedure is that we exploit the formal semantics of closed class word categories (e.g., quantifiers and negation) to boost the data efficiency of few-shot neural grounding models. Our experiments show these symbolic inductive biases are successful.



Related Work

Symbol Grounding.Contemporary grounders extensively utilize _batch_ learning (e.g. Shridhar and Hsu (2018)). Yet, itl requires _incremental_ learning because without it the teacher guidance cannot influence the learner's inferences about plans as and when the advice is given. Further, many grounders assume that the learner starts out with a complete and accurate conceptualisation of the domain using pre-defined visual features and a known vocabulary (Kennington et al., 2015; Kennington and Schlangen, 2017; Wang et al., 2017). In itl, both of these assumptions are unrealistic; therefore in this paper we explore models for which these assumptions don't apply. Finally, in contrast to all prior grounds, we support incremental learning when the training exemplars feature res that are linguistically complex: e.g., "two red circles that aren't to the right of both green squares".

Representation Learning.Models for jointly learning a representation for vision and language utilize either explicit alignment via bounding boxes or instance segmentation Lu et al. (2019); Chen et al. (2020); Tan and Bansal (2019); Kamath et al. (2021); Yu et al. (2021), or a large-volume of weakly labeled data in the form of image-caption pairs Radford et al. (2021). These models rely on offline learning with large datasets. This work, on the other hand, explores how to incrementally extract knowledge from few-shot learning, using sequentially observed evidence that includes neologisms.

Visual Questions Answering (vqa).This is a task of answering free-form questions about an image Antol et al. (2015). vqa has reached impressive performance in recent years Fukui et al. (2016); Li et al. (2020), yet vqa models struggle with outside distribution generalization for new types of questions, requiring multi-step reasoning, with analysis revealing that they often rely on shortcuts Jiang and Bansal (2019); Subramanian et al. (2019, 2020). Grounded vqa models like Yi et al. (2018) and Bogin et al. (2021) tackle these shortcomings by grounding parts of the question and then learning to compose those parts via the question's syntax to compute the answer. They thus estimate denotations of linguistic parts that are not denoted by the answer to the question. These 'compositional' models help to achieve out-of-distribution generalization for novel questions. But they lack itl's requirement for incremental learning: model training relies on batch learning. Furthermore, while their performance is impressive, error analysis shows that it makes mistakes when language includes logical concepts like quantifiers and negation (e.g. Bogin et al. (2021) Figure 9 shows that the determiner most incorrectly denotes an arbitrary subset of entities). Our view is that there is little benefit in trying to learn to ground logic concepts as they are domain independent and can be interpreted using formal semantics. In our experiments, we are testing the extent to which knowing and reasoning with the logical meanings of these symbols helps incremental grounding, and in particular estimating denotations of symbols within an re that are not designated.

Grounded Language Acquisition.This task is often realized as grounded grammar induction from image-caption pairs Shi et al. (2019); Zhao and Titov (2020), or as learning (neural) semantic parsers from a reward signal Williams (1992) in VQA Mao et al. (2019); Yi et al. (2018) or in planning Azaria et al. (2016); Wang et al. (2016, 2017); Karamcheti et al. (2020). There, the main objective is to learn to map natural language to logical forms, which in turn get associated with visual percepts during the learning process. This paper does not aim to learn a semantic parser. Instead, we obtain logical forms from an existing broad-coverage grammar which is hard to engineer, but is robust on lexical variation Curran et al. (2007). Our focus instead is on exploiting the _logical consequences_ of those logical forms during symbol grounding--i.e., our focus is to utilise the _interpretation_ of logical forms, and in particular the truth functional meanings of close-class words like quantifiers and negation, to inform the learning of mappings from (open-class) symbols like red to their denotations, given the visual percepts.

Visual Reference Resolution.In previous experiments, it is often assumed that there is a unique referent in the visual scene for the given re in the test phase Kazemzadeh et al. (2014); Whitney et al. (2016). We aim to cope with situations where the re has multiple referents: identifying all the referents that satisfy an re enables efficient planning, because it affords free choice when executing certain commands--e.g., "move a square above both red circles" when there is more than one square affords choosing a control policy so that resources are optimized 

Background

### Formal Semantics of Natural Language

Predicate logic with generalized quantifiers \({\mathfrak{L}}\)[18, 20] is a canonical meaning representation for natural languages. \({\mathfrak{L}}\)-sentences \(\phi\) are constructed recursively from predicates \(P\), terms \(T\) (i.e., variables \(V\) and constants \(C\)), logical connectives \(O=\{\neg,\land,\lor,\rightarrow\}\) and quantifiers \(Q\) (see Table 1 column 1):

\[\begin{array}{ll}\phi::=&\mathtt{p}(\mathtt{t}_{1},\ldots,\mathtt{t}_{n}) \equiv\mathtt{p}(\mathtt{t}^{n})\\ &&|(\neg\phi)|(\phi_{1}\wedge\phi_{2})|(\phi_{1}\vee\phi_{2})|(\phi_{1}\to \phi_{2})\\ &&|(\mathtt{Qx}(\phi_{1},\phi_{2}))\end{array}\]

where \(\mathtt{p}\) is an \(n\)-place predicate, \(\mathtt{t}_{i}\in T\) are terms, \(\mathtt{q}\in Q\) is a quantifier, and \(\mathtt{x}\in V\) is a variable (in \(\mathtt{Qx}(\phi_{1},\phi_{2})\), \(\phi_{1}\) is the restrictor and \(\phi_{2}\) the body). We also introduce \(\lambda\)-expressions of the form \(\lambda\mathtt{x}.\phi\), where \(\mathtt{x}\in V\) is free or absent in \(\phi\).

### Model-theoretic Interpretation

\({\mathfrak{L}}\)-sentences are interpreted using a domain model \({\mathcal{M}}=(E,I)\) consisting of a set of entities \(E\) (universe of discourse), and an extension function \(I\) that maps non-logical symbols \(P\cup C\) to denotations (tuples of entities). For convenience, we assume \(I:C\mapsto E\) is one-to-one. Variables are interpreted via an assignment function \(g\colon V\mapsto E\).

The interpretation function \([\![\cdot]\!]^{{\mathcal{M}},g}\) specifies the _semantic value_ of well-formed expressions of \({\mathfrak{L}}\):

\[[\![\mathtt{a}]\!]^{{\mathcal{M}},g}=\begin{cases}I(\mathtt{a})& \text{if $\mathtt{a}\in P\cup C$}\\ g(\mathtt{a})&\text{if $\mathtt{a}\in V$}\end{cases}\] \[[\![\mathtt{p}(\mathtt{t}^{n})]\!]^{{\mathcal{M}},g}=1\;\text{iff}\] \[([\![\mathtt{t}_{1}]\!]^{{\mathcal{M}},g},\ldots,[\![\mathtt{t}_{ n}]\!]^{{\mathcal{M}},g})\in[\![\mathtt{p}]\!]^{{\mathcal{M}},g}\] \[[\![\neg\phi]\!]^{{\mathcal{M}},g}=1\;\text{iff}\;[\![\phi]\!]^{{ \mathcal{M}},g}=0\] \[[\![\phi\wedge\psi]\!]^{{\mathcal{M}},g}=1\;\text{iff}\;[\![\phi]\!]^{{ \mathcal{M}},g}=1\;\text{and}\;[\![\psi]\!]^{{\mathcal{M}},g}=1\;\text{iff}\;[ \![\phi]\!]^{{\mathcal{M}},g}=1\;\text{or}\;[\![\psi]\!]^{{\mathcal{M}},g}=1\] \[[\![\phi\rightarrow\psi]\!]^{{\mathcal{M}},g}=1\;\text{iff}\;[\![ \phi]\!]^{{\mathcal{M}},g}=0\;\text{or}\;[\![\psi]\!]^{{\mathcal{M}},g}=1\] \[[\![\mathtt{xx}.\phi]\!]^{{\mathcal{M}},g}=\{e\in E\colon[\![\phi]\!] ^{{\mathcal{M}},g[\![\mathtt{xz}/e]}=1\}\] \[[\![\mathtt{Qx}(\phi_{1},\phi_{2})]\!]^{{\mathcal{M}},g}=\mathrm{Q} ([\![\lambda.\mathtt{x}.\phi_{1}]\!]^{{\mathcal{M}},g},[\![\lambda.\mathtt{ x}.\phi_{2}]\!]^{{\mathcal{M}},g})\]

where \(g[\mathtt{z}/e]\) is just like \(g\), except \(g[\mathtt{x}/e](\mathtt{x})=e\) and \(\mathrm{Q}\) is a specific relation between the restrictor \([\![\lambda.\mathtt{y}.\phi_{1}]\!]^{{\mathcal{M}},g}\) and body \([\![\lambda.\mathtt{yz}.\phi_{2}]\!]^{{\mathcal{M}},g}\), as defined in Table 1 column 3. \([\![\cdot]\!]^{{\mathcal{M}},g}\) is directly related to satisfiability for \({\mathfrak{L}}\)-sentences:

\[{\mathcal{M}},g\models\phi\;\text{iff}\;[\![\phi]\!]^{{\mathcal{ M}},g}=1\] \[{\mathcal{M}}\models\phi\;\text{iff}\;[\![\phi]\!]^{{\mathcal{M}} }=1\]

where \([\![\phi]\!]^{{\mathcal{M}}}=1\) iff \([\![\phi]\!]^{{\mathcal{M}},g}=1\) for all \(g\). Further, if \(\mathtt{x}\) is the _only_ free variable in \(\phi\), then \([\![\lambda.\mathtt{x}.\phi]\!]^{{\mathcal{M}},g}=[\![\lambda.\mathtt{x}.\phi] \!]^{{\mathcal{M}},g^{\prime}}\) for all \(g,g^{\prime}\); so without a loss of generality, this is expressed as \([\![\lambda\mathtt{x}.\phi]\!]^{{\mathcal{M}},1}\). If all variables in \(\mathtt{Qx}(\phi,\psi)\) are bound by quantifiers, then this \({\mathfrak{L}}\)-sentence is true iff \(\mathrm{Q}\) is true for all \(g\).

Some quantifiers, like "both",2 are presupposition triggers: "exactly two blocks are blue" is different from "both blocks are blue" in that the latter is true only if there are exactly two individuals that are blocks. We've adopted a Russellian interpretation [46] of these in Table 1.

Footnote 2: This fact will be used when defining referents.

Footnote 2: This is not an English specific phenomena: Finnish _molempi_ has the same condition as _both_.

### Logical Forms of Referential Expressions

We now define the logical forms of res and their interpretations with respect to a domain model \({\mathcal{M}}\). Noun phrases like "a block" are represented as \(\langle\_\mathtt{a\_qx.block}(\mathtt{x})\rangle\). More generally, let \(\langle\mathtt{Qx}.\phi\rangle\) be the logical form of an re, where \(\mathtt{q}\in Q\) and \(\phi\) is an \({\mathfrak{L}}\)-sentence with \(\mathtt{x}\in V\) being the only free variable in \(\phi\). The referents \(\langle\mathtt{Qx}.\phi\rangle^{{\mathcal{M}}}\) of this logical form with respect to \({\mathcal{M}}\) are computed as follows:

\[\langle\mathtt{Qx}.\phi\rangle^{{\mathcal{M}}}=\langle\mathtt{ q}\rangle^{\pi({\mathcal{M}},\phi,\mathtt{x})} \tag{1}\]

where \(\pi({\mathcal{M}},\phi,\mathtt{x})\) is an \({\mathcal{M}}\)-projection, giving a new domain model \({\mathcal{M}}^{\prime}\) with entities \(E^{\prime}=E\cap[\![\lambda.\mathtt{x}.\phi]\!]^{{\mathcal{M}}}\) and \(\langle\mathtt{q}\rangle^{{\mathcal{M}}}\) is a quantifier referent--a quantifier-specific subset of the power set of \(E\). Table 1 column 4 gives the list of quantifier referents.

To illustrate, consider the domain model where:

\[\begin{array}{l}E=\{a,b,c,d,f\}\\ I(\mathtt{cat})=\{a,b\}\quad I(\mathtt{dog})=\{c,d,f\}\\ I(\mathtt{bit})=\{(c,a),(c,b),(d,b),(f,a),(f,b)\}\end{array}\]

The re "a dog that bit both cats" has logical form \(\langle\_\mathtt{a\_qx.\_both\_qy(\mathtt{cat}(\mathtt{y}),\mathtt{dog}( \mathtt{x})\land\mathtt{bit}(\mathtt{x},\mathtt{y}))\rangle}\). By Equation 1, its referent is:

\[\langle\_\mathtt{a\_q}\rangle^{\pi({\mathcal{M}}_{\verb 



## 4 Methodology

Below we present the procedure of interactive grounding with referential expressions (igre). The overall framework is given in Figure 1.

### Grounder

Matching networks Vinyals et al. (2016) are an extension of the \(k\) nearest-neighbour algorithm Fix and Hodges (1989) and has been used as a fast few-shot grounder in the itl setting Cano Santin et al. (2020). For predicates \(P^{n}\subseteq P\) of the same arity \(n\), a grounded \(\Theta^{n}\) is parameterized by a support set \(\mathcal{S}^{n}=\{(\mathbf{x}_{i}^{n},\mathbf{y}_{i}^{n})\}_{i=1}^{K}\), consisting of \(K^{n}\) pairs of feature vectors \(\mathbf{x}_{i}^{n}\in\mathbb{R}^{d_{n}}\) for denotations \(e^{n}\in E^{n}\) and concept vectors \(\mathbf{y}_{i}^{n}\in[0,1]^{|P^{n}|}\). In \(\mathbf{y}^{n}\), the dimension \(z\) corresponds the predicate \(\mathtt{p}_{z}\in P^{n}\) and its value is the probability that \(\llbracket\mathtt{p}_{z}(e^{n})\rrbracket^{\mathcal{M},g}=1\). Concept vectors have a one-to-one correspondence with the domain model \(\mathcal{M}\).

Given a feature vector \(\mathbf{x}^{n}\) for a denotation \(e^{n}\in E^{n}\), \(\Theta^{n}\) predicts the concept vector \(\hat{\mathbf{y}}^{n}\), using the following inference rule:

\[\Theta^{n}(\mathbf{x}^{n},S^{n})=\sum_{i=1}^{k}\alpha^{n}(\mathbf{ x}_{i}^{n},\mathbf{x}^{n};S^{n})\mathbf{y}_{i}^{n}\]

where \(\alpha^{n}\) is an attention kernel:

\[\alpha^{n}(\mathbf{x}_{i}^{n},\mathbf{x}^{n};S^{n}) =\frac{\exp{(f^{n}(\mathbf{x}_{i}^{n})\cdot h^{n}(\mathbf{x}^{n}))}} {\sum_{j=1}^{k}\exp{(f^{n}(\mathbf{x}_{j}^{n})\cdot h^{n}(\mathbf{x}^{n}))}}\] \[f^{n}(\mathbf{x}^{n}) =\frac{\texttt{ReLU}(\textbf{w}^{n}\cdot\textbf{x}^{n}+b^{n})}{|| \texttt{ReLU}(\textbf{w}^{n}\cdot\textbf{x}^{n}+b^{n})||_{2}}\] \[h^{n}(\mathbf{x}^{n}) =\frac{\texttt{ReLU}(\textbf{v}^{n}\cdot\textbf{x}^{n}+c^{n})}{| \texttt{ReLU}(\textbf{v}^{n}\cdot\textbf{x}^{n}+c^{n})||_{2}}\] ReLU \[(a) =\max{(0,a)}\]

with learnable parameters \(\theta^{n}=\{\textbf{w}^{n},\textbf{v}^{n},b^{n},c^{n}\}\), and \(S^{n}\) is \(k=3\) nearest exemplars to \(\mathbf{x}^{n}\) from \(\mathcal{S}^{n}\):

\[S^{n}=\{(\mathbf{x}_{i}^{n},\mathbf{y}_{i}^{n})\in\mathcal{S}^{ n}\colon\mathbf{x}_{i}^{n}\in\mathcal{V}(k,\mathbf{x}^{n},\mathcal{S}^{n})\}\]

where \(\mathcal{V}(k,\mathbf{x}^{n},\mathcal{S}^{n})\) is a set of \(k\) nearest feature vectors.

### Batch Learning

Given \(\mathcal{S}^{n}\), one can estimate \(\Theta^{n}\) either via batch learning performed offline, or--when \(\mathcal{S}^{n}\) is small--in real time, as outlined by Cano Santin et al. (2020). In our scenario, we learn in real time by minimizing binary cross-entropy between the ground-truth \(\textbf{y}^{n}\) and predicted \(\hat{\textbf{y}}^{n}\) concept vectors:

\[\mathcal{L}(\textbf{y}^{n},\hat{\textbf{y}}^{n}) =-\sum_{z=1}^{|P^{n}|}l(y_{z}^{n},\hat{y}_{z}^{n})\] \[l(y_{i}^{n},\hat{y}_{i}^{n}) =y_{i}^{n}\log(\hat{y}_{i}^{n})+(1-y_{i}^{n})\log(1-\hat{y}_{i}^{ n})\]

### Incremental Learning

\(\mathcal{S}^{n}\) gets augmented whenever the teacher provides an re-designation pair. This speech act provides two types of information: certain information \(\mathcal{C}^{n}\) in the form of denotation-symbol-semantic value triples \((e^{n},\mathtt{p}_{z},y_{z}^{n})\), corresponding to symbols and entities designated by the re; and _noisy_ information \(\mathcal{N}^{n}\), corresponding to denotation-symbol-semantic value estimate triples \((e^{n},\mathtt{p}_{z},\tilde{y}_{z}^{n})\), which are acquired from the symbols that are part of the re and its referent inferred via (uncertain) reasoning. E.g., the re "a circle below a square.", entails that its designation \(e\in E\) is a circle and so \((e,\texttt{circle},1)\) is added to \(\mathcal{C}^{n}\). But it also entails there exists an entity which is a square that is not designated by the re, but rather this entity is in the below relation with the designated entity. If the grounder is sufficiently confident about the referent for square, then the corresponding triple is added to \(\mathcal{N}^{n}\).

#### 4.3.1 Acquiring Observations and Symbols

When the learner first observes its visual scene--and so the teacher has not expressed any concepts, and so the learner is currently unaware of all concepts--the noisy support set \(\mathcal{N}^{n}\) is populated with \((e^{n},\mathtt{p}_{z},0.5)\) (\(0.5\) is a default semantic value) for all \(e^{n}\) in the scene and for all known \(n\)-place predicates. Whenever the teacher's re-designation pair features a neologism \(\mathtt{p}^{*}\), then this expansion to the learner's vocabulary prompts adding \((e^{n},\mathtt{p}^{*},0.5)\) to \(\mathcal{N}^{n}\) for all \(e^{n}\). During interaction, each re-designation pair uttered by the teacher adds elements to \(\mathcal{C}^{n}\) (for designated symbols) and triggers updates to the \(\mathcal{N}^{n}\) elements for all entities in the current visual scene, as we'll now describe.

#### 4.3.2 Integrating the Teacher's Feedback

\(\mathcal{N}^{n}\) elements are interactively updated using an incrementally built domain-level theory \(\Delta\), which is the conjunction of \(\mathfrak{L}\)-sentences that are built from the logical forms of the res that the teacher has uttered so far and their designations. To compute the beliefs about semantic values, given \(\Delta\), we model the semantic value of \(\mathfrak{L}\)-sentences of the form \(\mathtt{p}(\mathtt{t}^{n})\), in which \(t^{n}\) are all constants (ground atom), as a random variable with Bernoulli's distribution \(\mathcal{B}\). Thus a distribution over the possible domain models can be estimated using (propositional) model counting MCValiant (1979), which maps each \(\mathfrak{L}\)-sentence to the number of domain models satisfying it. In this way, the semantic value of any proposition can be estimated as follows:

\[\tilde{y}_{z}^{n}=\begin{cases}\frac{\texttt{MC}(\mathtt{p}_{ z}(e^{n})\wedge\Delta)}{\texttt{MC}(\Delta)}&\text{if $\texttt{MC}(\Delta)\neq 0$}\\ 0.5&\text{otherwise}\end{cases}\]

mc can be computed exactly or approximately (Samer and Szeider, 2010). In our experiments we use the addmdcDudek et al. (2020) weighted model counter, with weights set to 0.5.

#### 4.3.3 Building the Support Set

Concept vectors for \(\mathcal{S}^{n}\) are built using information in \(\mathcal{C}^{n}\) and \(\mathcal{N}^{n}\): namely each denotation \(e^{n}\) gets associated with its feature vector \(\mathbf{x}^{n}\), and the \(z\)-dimension of \(\mathbf{y}\) corresponding to predicate \(\mathtt{p}_{z}\in P^{n}\) is computed as follows:

\[y_{z}^{n}=\begin{cases}y_{z}^{n}~{}\text{if}~{}(e^{n},\mathtt{ p}_{z},y_{z}^{n})\in\mathcal{C}^{n}\\ \tilde{y}_{z}^{n}~{}\text{if}~{}(e^{n},\mathtt{p}_{z},\tilde{y}_{z}^{n})\in \mathcal{N}^{n}\land\mathbb{H}[\mathcal{B}(\bar{y}_{z}^{n})]\leq\tau^{n}\\ 0.5~{}\text{otherwise}\end{cases}\] 

where \(\mathbb{H}[\mathcal{P}]\) is the entropy of the probability distribution \(\mathcal{P}\), and \(\tau^{n}\) is the confidence threshold for adding noisy exemplars: in our case, it's set to \(0.6\) for predicates of all arities.

## 5 Experiments

### Task: Visual Reference Resolution

To evaluate igre, we use a task of visual reference resolution 3: given a visual scene (an image) with localized entities (bounding boxes) and an re, the grounder must estimate all its referents, as defined in SS3.3. The model learns its task by observing an image accompanied by a sequence of res, with each re paired with its designation in the image.

Footnote 3: Code available at [https://github.com/itl-ed/igre](https://github.com/itl-ed/igre)

We measure the performance of igre on the task after each observed re and its designation. Performance is measured using the precision **P**, recall **R**, and F1 score **F1** on the test set between: 1) estimated vs. ground-truth domain models, formed from the concept vectors (intrinsic evaluation) and 2) estimated vs. ground-truth referents for the re (extrinsic evaluation). These metrics are calculated only for those symbols/concepts that the teacher has mentioned so far (since the system is unaware that the remaining concepts exist). To obtain reliable results, we repeat the experiment 10 times: i.e., 10 different visual scenes, with a sequence of 5 different teacher utterances in each scene. We record in SS6 the average precision, recall and f-scores over those 10 trials.

Perhaps unusually, this training and testing regime uses very small data sets: that's because in itl it is the initial portions of the learning curve that matters. The learner must achieve decent performance on its task via only a few teacher utterances: human teachers won't tolerate repeating the same res many times and so the learner lacks the luxury of learning (and testing) symbol grounding on large data sets.

### Data: ShapeWorld

To generate training and test sets, we construct ShapeWorld domain models Kuhnle and Copestake (2017), each consisting of 3-12 entities, synthesized visual scenes \(X\) (64x64 pixels), and 5 res. Each domain model is describable using 7 shape symbols **S1** (square, circle, triangle, pentagon, cross, ellipse, semicircle), 6 colour symbols **C1** (red, green, blue, yellow, magenta, cyan) and 4 spatial relationships symbols **R2** (left, right, above, below).4 In scene synthesis, the image is created from the domain model, with variation on the hue of the colour category, variation on the size, position, rotation, and distortion of the shapes, and variation on the spatial positions of the entities related by each spatial term. Note that the colour categories are not mutually exclusive--e.g., there are RGB values that count as both red and magenta.

Footnote 4: Entities are non-overlapping, thus we omit on/behind.

To generate res, we sample Dependency Minimal Recursion Semantics (dmrs) Copestake (2009) graph templates, processed using ace (generation mode) 5 and the English Resource Grammar (erg) Flickinger (2000). Generated res are evaluated with respect to the domain model to guarantee an existing referent. In total we generated 30 such domain models for training and 10 for testing. The data statistics for the training set is given in Table 2 for the general categories of symbols, where _certain_ (\(\mathcal{C}^{n}\)) means that the designation is denoted by the symbol in the re, and _noisy_ (\(\mathcal{N}^{n}\)) means that the symbol is a part of the re but is not designated by it. Note that the first argument to the spatial relations **R2** is always denoted by the designation while its second argument is not. Note also there is high variance in the frequencies among the individual symbols. For instance, blue occurs 27 and 28 times in certain vs. noisy positions respectively, while triangle occurs 7 and 12 times respectively.

Footnote 5: [http://sweBesw.org/linguistics/ace/](http://sweBesw.org/linguistics/ace/)

### Implementation Details

#### 5.3.1 Feature Extraction

To extract visual features for individuals in the scene, we utilize bounding boxes \(\mathbf{b}=[x_{\mathit{left}},x_{\mathit{right}},y_{top},y_{bottom}]^{\top}\) for each entity \(e\in E\) in the visual scene by localizing them (cropping) and extracting the visual features using a pre-trained visual feature encoder (in our case,

\begin{table}
\begin{tabular}{c c c} \hline \hline Category & \(\mathcal{C}^{n}\) candidates & \(\mathcal{N}^{n}\) candidates \\ \hline
**C1** & 18.67 \(\pm\) 5.39 & 19.83 \(\pm\) 5.04 \\
**S1** & 14.67 \(\pm\) 3.98 & 16.50 \(\pm\) 5.32 \\
**R2** & 0 & 37.75 \(\pm\) 6.75 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Average symbol counts per word for colours (**C1**), shapes (**S1**), and spatial relationships (**R2**).

 

DenseNet161 (Huang et al., 2017)). Additionally, for the feature vector, we add each entity's bounding box coordinates for spatial information, lost in the localization process:

\[{\bf x}_{n}=\texttt{Concat}(\{[\texttt{DenseNet161}(X[{\bf b}_{i}]],{\bf b}_{ i})\}^{n}_{i=1})\]

#### 5.3.2 Grammar-based Semantic Parsing

To parse res to their logical forms, we use the English Resource Grammar (erg) and ace (parsing mode) to produce a representation in minimal recursion semantics (mrs) (Copestake et al., 1997), which we then simplify via hand-written rules (e.g., removing event arguments from predicate symbols corresponding to adjectives and prepositions). Underspecification of the mrs was resolved using utoolKoller and Thater, 2005) and the final logical form was selected based on the linear order of scope-bearing elements (quantifiers and negation): e.g. for the re "every circle above a square", _every_q outscopes_a_q.

#### 5.3.3 Axioms for R2

For \(|E|\) entities, there are \(|E|^{2}\) denotations to consider for each \(2\)-place predicate--a larger search space compared to \(|E|\) denotations for \(1\)-place predicates. Moreover, these predicates can only be acquired from the noisy component \(\mathcal{N}^{n}\) because the referent of the second argument to the relation is always latent.

To aid the learning process for **R2**, whenever a new symbol \(R\in\mathbf{R2}\) is observed, domain-level axioms are added to \(\Delta\) for it, making it irreflexive: \(\forall x.\neg R(x,x)\) (an entity cannot be in a spatial relationship to itself) and asymmetric: \(\forall x,y.R(x,y)\rightarrow\neg R(y,x)\) (reflecting the fact that entities in spatial relations take different roles (Miller and Johnson-Laird, 1976)). These axioms reduce the number of possible denotations for **R2** symbols from \(|E|^{2}\) to \(\frac{|E|^{2}}{2}-|E|\).

### Baselines

To test the benefit of using noisy training exemplars \(\mathcal{N}^{n}\) from the oblique symbols in the res--in other words, those symbols that are a part of the re but not designated by it--we implemented a head grounder baseline, which uses information only from \(\mathcal{C}^{n}\). That head uses only symbol-designation pairs that are acquired when the symbol denotes the referent (in our case, that's the head noun in the re and its pre-head modifier, if it exists).

To test the the benefit of using the precise formal semantic meanings of logical symbols (i.e., quantifiers and negation), we implemented an exist groundwater baseline. This utilizes the information from the symbols in the oblique positions, but it does _not_ utilize the precise symbolic interpretation of the logical symbols, instead simplifying the logical form of the re by replacing all quantifiers with the existential _a_q and removing negation (e.g., "every cross on the left of the one circle" is equivalent to "a cross on the left of a circle"). This baseline preserves the basic linguistic structure of the formal semantic representation of the re, but not its truth-functional interpretation.

## 6 Results and Discussion

Figure 2 shows the evolution of the performance of the igre grounder and the two baselines on the test set, as it gets exposed to more information (i.e., re-designation pairs) over time. In the intrinsic evaluation (domain model prediction), there is no significant difference between the three grounders considered. Yet, for extrinsic evaluation (reference resolution), we observe that igre outperforms the head and exists baselines over time (both a steeper and a smoother curve). By the end of the interaction, a t-test shows significant differences in igre's performance compared with both baselines (p-value of 0.01).

Table 3 shows the best performance that each grounder achieved over time. When analysing their performance on particular categories, we observe that **C1** and **S1** are equally hard to learn for grounders while **R2** is easier.

We suspect that the reason why the three models performed differently in extrinsic evaluation even though they don't with intrinsic evaluation is down to the fact that igre uses its complete and accurate knowledge of the meanings of closed class words like quantifiers and negation at test time as well as training time in the extrinsic evaluation, but not in the intrinsic evaluation. The igre model can use these meanings to constrain and correct error-prone estimates of referents for open class words at test time in the reference task (as well as using their meanings to boost the training sets). For example, the re "both squares" implies there exist exactly two squares; if the symbol grounding model has an uncertain belief that there are more (or fewer) squares than this, it will select the two most probably candidates (and infer that all other entities are 

non-squares). These experiments suggest that this sort of correction to confident but wrong estimates of the denotations of open-class symbols happens sufficiently often at test time in the reference task to make a difference in this low-data regime we are interested in, for addressing itl tasks.

### Error Analysis

The head and exists baselines never acquire negative exemplars: e.g., information that a particular individual is not red. Figure 2 shows that this severely impacts their performance, and error analysis showed that in some experiment runs it leads to model-collapse, with all denotations predicted to be in the extensions of all symbols. On the other hand, igre is able to acquire and use negative examples from the truth functional meanings of the logical symbols, specifically from: (a) negation ("not"); (b) the presupposition triggers"the \(N^{*}\), "\(N\) of the \(M\)", and "all but \(N\)" where \(N\), and \(M\) are numbers and "both"; and (c) the use of "every" when it modifies the head noun.

## 7 Conclusions

In this work, we presented igre--a grounder that supports incremental learning of the mapping from symbols to visual features whenever the teacher presents a linguistically complex re and its designation(s) in the visual scene. The grounder starts the learning process with no conceptualisation of the domain model, and so the learner must revise its hypothesis space of possible domain models as and when the teacher introduces new and unforeseen concepts via neologisms. We showed how exploiting the model-theoretic interpretation of the formal semantic representations of res, and in particular the truth conditions of 'logical' words like quantifiers and negation, can inform the acquisition of noisy training exemplars that in turn guide learning--igre reasons about the likely denotations of symbols within an re that aren't designated by that re, and when sufficiently confident it exploits them to update its grounding model. We showed that: 1) this grounding approach is more data efficient then a model that omits such observations and reasoning, using only the designated symbols; and 2) it is beneficial to exploit the logical consequences of the logical symbols, to gain even more data efficiency and training stability. In both cases, there was much to be gained from such reasoning because in contrast to the baselines, it contributes to acquiring _negative_ exemplars: in other words, objects that get associated with _not_ being red, for example.

Figure 2: Evolution of F1 scores for igre (ours), exists, head grounders over the course of interaction on domain model prediction (left) and reference resolution (right)





### Future Work

igre uses a single source of data augmentation by acquiring noisy exemplars from symbols in oblique positions. Further and parallel data gains may be obtained by exploring semi-supervised learning methods (Yarowsky, 1995; Delalleau et al., 2005).

In this work, converting \({\cal L}\)-sentences to conjunctive normal form, which is an NP-hard problem, was a computational bottleneck. Future work needs to address this by either considering lifted inference methods (e.g., den Broeck et al. (2011)) or defining model counters that use \({\mathfrak{L}}\)-sentences directly.

Finally, the purpose of igre is to aid itl: i.e., the (incremental) updates to beliefs about symbol grounding should enhance learning to solve domain-level planning problems. Future work needs to address this by using igre to learn planning tasks where the learner has the physical ability to execute certain actions but starts out unaware of domain concepts that define the goal and are critical to task success. The learner must not only use igre to interpret the teacher's feedback, but also learn decision making strategies, both on what to say (or ask) the teacher in their extended dialogue and what actions to perform in the environment. Furthermore, the static formal semantics that we used here should be replaced with a dynamic semantics (e.g., Groenendijk and Stokhof (1991); van der Sandt (1992); Asher and Lacarides (2003)), to account for how contextual salience influences truth and reference in dialogue. Following Batra et al. (2020), we plan to test the benefits of igre within a system that learns to solve planning problems that focus on rearrangement tasks.

## Acknowledgments

This work was supported in part by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1), UKRI Strategic Priorities Fund to the UKRI Research Node on Trustworthy Autonomous Systems Governance and Regulation (grant EP/V026607/1, 2020-2024), and the Turing 2.0 'Enabling Advanced Autonomy through Human-AI Collaboration' project funded by EPSRC and the University of Edinburgh through the Alan Turing Institute.

## References

* Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_, pages 2425-2433. IEEE Computer Society.
* Asher and Lascarides [2003] Nicholas Asher and Alex Lascarides. 2003. _Logics of Conversation_. Cambridge University Press.
* Azaria et al. [2016] Amos Azaria, Jayant Krishnamurthy, and Tom M. Mitchell. 2016. Invractable intelligent personal agent. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA_, pages 2681-2689. AAAI Press.
* Barwise and Cooper [1981] Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. _Linguistics and Philosophy_, 4:159-219.
* Batra et al. [2020] Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergeye Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. 2020. Rearrangement: A challenge for embodied AI. _CoRR_, abs/2011.01975.
* Bogin et al. [2021] Ben Bogin, Sanjay Subramanian, Matt Gardner, and Jonathan Berant. 2021. Latent compositional representations improve systematic generalization in grounded question answering. _Transactions of the Association of Computational Linguistcsl (TACL)_, 9.
* Santin et al. [2020] Jose Miguel Cano Santin, Simon Dobnik, and Mehdi Ghanimifard. 2020. Fast visual grounding in interaction: bringing few-shot learning with neural networks to an interactive robot. In _Proceedings of the Probability and Meaning Conference (PaM 2020)_, pages 53-61, Gothenburg. Association for Computational Linguistics.
* ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXXX_, volume 12375 of _Lecture Notes in Computer Science_, pages 104-120. Springer.
* April 3, 2009_, pages 1-9. The Association for Computer Linguistics


Ann A. Copestake, D. Flickinger, C. Pollard, and I. Sag. 1997. Minimal recursion semantics: An introduction. _Research on Language and Computation_, 3:281-332.
* Curran et al. (2007) James R. Curran, Stephen Clark, and Johan Bos. 2007. Linguistically motivated large-scale NLP with c&c and boxer. In _ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic_. The Association for Computational Linguistics.
* Dale and Reiter (1995) Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. _Cognitive science_, 19(2):233-263.
* November 2, 2019_, pages 2601-2610. IEEE.
* Delalleau et al. (2005) Olivier Delalleau, Yoshua Bengio, and Nicolas Le Roux. 2005. Efficient non-parametric function induction in semi-supervised learning. In _Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, AISTATS 2005, Bridgetown, Barbados, January 6-8, 2005_. Society for Artificial Intelligence and Statistics.
* den Broeck et al. (2011) Guy Van den Broeck, Nima Taghipour, Wannes Meert, Jesse Davis, and Luc De Raedt. 2011. Lifted probabilistic inference by first-order knowledge compilation. In _IJCAI 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, July 16-22, 2011_, pages 2178-2185. IJCAI/AAAI.
* Dudek et al. (2020) Jeffrey M. Dudek, Vu Phan, and Moshe Y. Vardi. 2020. ADDMC: weighted model counting with algebraic decision diagrams. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 1468-1476. AAAI Press.
* nonparametric discrimination: Consistency properties. _International Statistical Review_, 57:238.
* Flickinger (2000) Dan Flickinger. 2000. On building a more effcient grammar by exploiting types. _Nat. Lang. Eng._, 6(1):15-28.
* Fukui et al. (2016) Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 457-468. The Association for Computational Linguistics.
* Groenendijk and Stokhof (1991) Jeroen Groenendijk and Martin Stokhof. 1991. Dynamic predicate logic. _Linguistics and Philosophy_, 14:39-100.
* Harnad (1999) Stevan Harnad. 1999. The symbol grounding problem. _CoRR_, cs.AI/9906002.
* November 1, 2019, Proceedings_, volume 100 of _Proceedings of Machine Learning Research_, pages 870-884. PMLR.
* Hristov et al. (2018) Yordan Hristov, Alex Lascarides, and Subramanian Ramamoorthy. 2018. Interpretable latent spaces for learning from demonstration. In _2nd Annual Conference on Robot Learning, CoRL 2018, Zurich, Switzerland, 29-31 October 2018, Proceedings_, volume 87 of _Proceedings of Machine Learning Research_, pages 957-968. PMLR.
* Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_, pages 2261-2269. IEEE Computer Society.
* Jiang and Bansal (2019) Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2726-2736, Florence, Italy. Association for Computational Linguistics.
* modulated detection for end-to-end multimodal understanding. _CoRR_, abs/2104.12763.
* Karamcheti et al. (2020) Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. 2020. Learning adaptive language interfaces through decomposition. _CoRR_, abs/2010.05190.
* Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 787-798. ACL.
* Kennington et al. (2015) Casey Kennington, Livia Dia, and David Schlangen. 2015. A discriminative model for perceptually-grounded incremental reference resolution. In _Proceedings of the 11th International Conference on 

Computational Semantics, IWCS 2015, 15-17 April, 2015, Queen Mary University of London, London, UK_, pages 195-205. The Association for Computer Linguistics.
* Kenington and Schlangen (2017) Casey Kenington and David Schlangen. 2017. A simple generative model of incremental reference resolution for situated dialogue. _Comput. Speech Lang._, 41:43-67.
* Koller and Thater (2005) Alexander Koller and Stefan Thater. 2005. The evolution of dominance constraint solvers. In _Proceedings of Workshop on Software_, pages 65-76, Ann Arbor, Michigan. Association for Computational Linguistics.
* A new test methodology for multimodal language understanding. _CoRR_, abs/1704.04517.
* Liard et al. (2017) John E. Liaird, Kevin A. Gluck, John R. Anderson, Kenneth D. Forbus, Odest Chadwicke Jenkins, Christian Lebiere, Dario D. Salvucci, Matthias Scheutz, Andrea Thomaz, J. Gregory Trafton, Robert E. Wray, Shiwali Mohan, and James R. Kirk. 2017. Interactive task learning. _IEEE Intelligent Systems_, 32(4):6-21.
* ECCV 2020
- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXXX_, volume 12375 of _Lecture Notes in Computer Science_, pages 121-137. Springer.
* Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 13-23.
* Mao et al. (2019) Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Matuszek (2018) Cynthia Matuszek. 2018. Grounded language learning: Where robotics and NLP meet. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden_, pages 5687-5691. ijcai.org.
* Miller and Johnson-Laird (1976) George A. Miller and Philip N. Johnson-Laird. 1976. _Language and Perception_. Belknap Press Imprint.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn-transferable visual models from natural language supervision. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_, volume 139 of _Proceedings of Machine Learning Research_, pages 8748-8763. PMLR.
* Russell (1917) Bertrand Russell. 1917. Knowledge by acquaintance and knowledge by description. In _Myticism and Logic_, pages 152-167. London: Longmans Green.
* Samer and Zweider (2010) Marko Samer and Stefan Szefier. 2010. Algorithms for propositional model counting. _J. Discrete Algorithms_, 8(1):50-64.
* Shi et al. (2019) Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019. Visually grounded neural syntax acquisition. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 1842-1861. Association for Computational Linguistics.
* Shridhar and Hsu (2018) Mohit Shridhar and David Hsu. 2018. Interactive visual grounding of referring expressions for human-robot interaction. In _Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018_.
* Subramanian et al. (2020) Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020. Obtaining faithful interpretations from compositional neural networks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5594-5608, Online. Association for Computational Linguistics.
* Subramanian et al. (2019) Sanjay Subramanian, Sameer Singh, and Matt Gardner. 2019. Analyzing compositionality in visual question answering. In _ViGIL@NeurIPS_.
* Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019. LXMERT: learning cross-modality encoder representations from transformers. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 5099-5110. Association for Computational Linguistics.
* Valiant (1979) Leslie G. Valiant. 1979. The complexity of computing the permanent. _Theor. Comput. Sci._, 8:189-201.
* van Benthem (1984) Johan van Benthem. 1984. Questions about quantifiers. _J. Symb. Log._, 49(2):443-466.
* van der Sandt (1992) Rob van der Sandt. 1992. Presupposition projection as anaphora resolution. _Journal of Semantics_, 9(4):333-377.

 

Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016. Matching networks for one shot learning. In Advances in Neural Information Processing Systems _29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_, pages 3630-3638.
* August 4, Volume 1: Long Papers, pp. 929-938. Association for Computational Linguistics.
* Wang et al. (2016) Sida I. Wang, Percy Liang, and Christopher D. Manning. 2016. Learning language games through interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
* Whitney et al. (2016) David Whitney, Miles Eldon, John Oberlin, and Stefanie Tellex. 2016. Interpreting multimodal referring expressions in real time. In _2016 IEEE International Conference on Robotics and Automation (ICRA)_, pages 3331-3338.
* Williams (1992) Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256.
* Yarowsky (1995) David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In _33rd Annual Meeting of the Association for Computational Linguistics_, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
* Ye et al. (2019) Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. 2019. Cross-modal self-attention network for referring image segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 10502-10511. Computer Vision Foundation / IEEE.
* Yi et al. (2018) Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic VQA: disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems _31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada_, pages 1039-1050.
* Yu et al. (2021) Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 3208-3216. AAAI Press.
* Zhao and Titov (2020) Yanpeng Zhao and Ivan Titov. 2020. Visually grounded compound pcfgs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4369-4379. Association for Computational Linguistics.