SlowGuess/NAACL_2022_OCR huggingface dataset:
Concept vectors for  $S^n$  are built using information in  $\mathcal{C}^n$  and  $\mathcal{N}^n$ : namely each denotation  $e^n$  gets associated with its feature vector  $\mathbf{x}^n$ , and the  $z$ -dimension of  $\mathbf{y}$  corresponding to predicate  $\mathfrak{p}_z \in P^n$  is computed as follows:
$$
y _ {z} ^ {n} = \left\{ \begin{array}{l} y _ {z} ^ {n} \text {i f} (e ^ {n}, \mathbf {p} _ {z}, y _ {z} ^ {n}) \in \mathcal {C} ^ {n} \\ \tilde {y} _ {z} ^ {n} \text {i f} (e ^ {n}, \mathbf {p} _ {z}, \tilde {y} _ {z} ^ {n}) \in \mathcal {N} ^ {n} \wedge \mathbb {H} [ \mathcal {B} (\tilde {y} _ {z} ^ {n}) ] \leq \tau^ {n} \\ 0. 5 \text {o t h e r w i s e} \end{array} \right.
$$
where  $\mathbb{H}[\mathcal{P}]$  is the entropy of the probability distribution  $\mathcal{P}$ , and  $\tau^n$  is the confidence threshold for adding noisy exemplars: in our case, it's set to 0.6 for predicates of all arities.
5.1 Task: Visual Reference Resolution
To evaluate IGRE, we use a task of visual reference resolution<sup>3</sup>: given a visual scene (an image) with localized entities (bounding boxes) and an RE, the grounder must estimate all its referents, as defined in §3.3. The model learns its task by observing an image accompanied by a sequence of REs, with each RE paired with its designation in the image.
We measure the performance of IGRE on the task after each observed RE and its designation. Performance is measured using the precision  $\mathbf{P}$ , recall  $\mathbf{R}$ , and F1 score  $\mathbf{F1}$  on the test set between: 1) estimated vs. ground-truth domain models, formed from the concept vectors (intrinsic evaluation) and 2) estimated vs. ground-truth referents for the RE (extrinsic evaluation). These metrics are calculated only for those symbols/concepts that the teacher has mentioned so far (since the system is unaware that the remaining concepts exist). To obtain reliable results, we repeat the experiment 10 times: i.e., 10 different visual scenes, with a sequence of 5 different teacher utterances in each scene. We record in §6 the average precision, recall and f-scores over those 10 trials.
Perhaps unusually, this training and testing regime uses very small data sets: that's because in ITL it is the initial portions of the learning curve that matters. The learner must achieve decent performance on its task via only a few teacher utterances: human teachers won't tolerate repeating the same REs many times and so the learner lacks the luxury of learning (and testing) symbol grounding on large data sets.
5.2 Data: ShapeWorld
To generate training and test sets, we construct ShapeWorld domain models (Kuhnle and Copestake, 2017), each consisting of 3-12 entities, synthesized visual scenes  $X$  (64x64 pixels), and 5 RESs. Each domain model is describable using 7 shape symbols S1 (square, circle, triangle, pentagon, cross, ellipse, semicircle), 6 colour symbols C1 (red, green, blue,
yellow, magenta, cyan) and 4 spatial relationships symbols R2 (left, right, above, below). In scene synthesis, the image is created from the domain model, with variation on the hue of the colour category, variation on the size, position, rotation, and distortion of the shapes, and variation on the spatial positions of the entities related by each spatial term. Note that the colour categories are not mutually exclusive—e.g., there are RGB values that count as both red and magenta.
To generate REs, we sample Dependency Minimal Recursion Semantics (DMRS) (Copestake, 2009) graph templates, processed using ACE (generation mode) and the English Resource Grammar (ERG) (Flickinger, 2000). Generated REs are evaluated with respect to the domain model to guarantee an existing referent. In total we generated 30 such domain models for training and 10 for testing. The data statistics for the training set is given in Table 2 for the general categories of symbols, where certain  $(\mathcal{C}^n)$  means that the designation is denoted by the symbol in the RE, and noisy  $(\mathcal{N}^n)$  means that the symbol is a part of the RE but is not designated by it. Note that the first argument to the spatial relations R2 is always denoted by the designation while its second argument is not. Note also there is high variance in the frequencies among the individual symbols. For instance, blue occurs 27 and 28 times in certain vs. noisy positions respectively, while triangle occurs 7 and 12 times respectively.
Category	Cn candidates	Nn candidates
C1	18.67 ± 5.39	19.83 ± 5.04
S1	14.67 ± 3.98	16.50 ± 5.32
R2	0	37.75 ± 6.75
Table 2: Average symbol counts per word for colours (C1), shapes (S1), and spatial relationships (R2).
5.3.1 Feature Extraction
To extract visual features for individuals in the scene, we utilize bounding boxes  $\mathbf{b} = [x_{left}, x_{right}, y_{top}, y_{bottom}]^{\top}$  for each entity  $e \in E$  in the visual scene by localizing them (cropping) and extracting the visual features using a pre-trained visual feature encoder (in our case,
3Code available at https://github.com/itl-ed/ igre
4Entities are non-overlapping, thus we omit on/behind.
<sup>5</sup>http://sweaglesw.org/linguistics/ace/
DenseNet161 (Huang et al., 2017)). Additionally, for the feature vector, we add each entity's bounding box coordinates for spatial information, lost in the localization process:
$$
\mathbf {x} _ {n} = \mathrm {C o n c a t} (\{\left[ \mathrm {D e n s e N e t 1 6 1} (X [ \mathbf {b} _ {i} ] ], \mathbf {b} _ {i}) \right\} _ {i = 1} ^ {n})
$$
5.3.2 Grammar-based Semantic Parsing
To parse RES to their logical forms, we use the English Resource Grammar (ERG) and ACE (parsing mode) to produce a representation in minimal recursion semantics (MRS) (Copestake et al., 1997), which we then simplify via hand-written rules (e.g., removing event arguments from predicate symbols corresponding to adjectives and prepositions). Underspecification of the MRS was resolved using UTOOL (Koller and Thater, 2005) and the final logical form was selected based on the linear order of scope-bearing elements (quantifiers and negation): e.g. for the RE "every circle above a square", _every_q outscopes _a_q.
5.3.3 Axioms for R2
For  $|E|$  entities, there are  $|E|^2$  denotations to consider for each 2-place predicate—a larger search space compared to  $|E|$  denotations for 1-place predicates. Moreover, these predicates can only be acquired from the noisy component  $\mathcal{N}^n$  because the referent of the second argument to the relation is always latent.
To aid the learning process for R2, whenever a new symbol  $R \in \mathbf{R2}$  is observed, domain-level axioms are added to  $\Delta$  for it, making it irreflexive:  $\forall x.\neg R(x,x)$  (an entity cannot be in a spatial relationship to itself) and asymmetric:  $\forall x,y.R(x,y) \rightarrow \neg R(y,x)$  (reflecting the fact that entities in spatial relations take different roles (Miller and Johnson-Laird, 1976)). These axioms reduce the number of possible denotations for R2 symbols from  $|E|^2$  to  $\frac{|E|^2}{2} - |E|$ .
5.4Baselines
To test the benefit of using noisy training exemplars  $\mathcal{N}^n$  from the oblique symbols in the REs—in other words, those symbols that are a part of the RE but not designated by it—we implemented a HEAD grounder baseline, which uses information only from  $\mathcal{C}^n$ . That HEAD uses only symbol-designation pairs that are acquired when the symbol denotes the referent (in our case, that's the head noun in the RE and its pre-head modifier, if it exists).
To test the benefit of using the precise formal semantic meanings of logical symbols (i.e., quantifiers and negation), we implemented an EXIST groundeder baseline. This utilizes the information from the symbols in the oblique positions, but it does not utilize the precise symbolic interpretation of the logical symbols, instead simplifying the logical form of the RE by replacing all quantifiers with the existential _a_q and removing negation (e.g., "every cross on the left of the one circle" is equivalent to "a cross on the left of a circle"). This baseline preserves the basic linguistic structure of the formal semantic representation of the RE, but not its truth-functional interpretation.
Figure 2 shows the evolution of the performance of the IGRE grounded and the two baselines on the test set, as it gets exposed to more information (i.e., RE-designation pairs) over time. In the intrinsic evaluation (domain model prediction), there is no significant difference between the three grounders considered. Yet, for extrinsic evaluation (reference resolution), we observe that IGRE outperforms the HEAD and EXISTS baselines over time (both a steeper and a smoother curve). By the end of the interaction, a t-test shows significant differences in IGRE's performance compared with both baselines (p-value of 0.01).
Table 3 shows the best performance that each grounder achieved over time. When analysing their performance on particular categories, we observe that C1 and S1 are equally hard to learn for grounders while R2 is easier.
We suspect that the reason why the three models performed differently in extrinsic evaluation even though they don't with intrinsic evaluation is down to the fact that IGRE uses its complete and accurate knowledge of the meanings of closed class words like quantifiers and negation at test time as well as training time in the extrinsic evaluation, but not in the intrinsic evaluation. The IGRE model can use these meanings to constrain and correct error-prone estimates of referents for open class words at test time in the reference task (as well as using their meanings to boost the training sets). For example, the RE "both squares" implies there exist exactly two squares; if the symbol grounding model has an uncertain belief that there are more (or fewer) squares than this, it will select the two most probably candidates (and infer that all other entities are
Figure 2: Evolution of F1 scores for IGRE (ours), EXISTS, HEAD grounders over the course of interaction on domain model prediction (left) and reference resolution (right)
C1	S1	R2	Reference
P	R	F1	P	R	F1	P	R	F1	P	R	F1
HEAD	0.17	0.54	0.25	0.16	0.52	0.23	0.16	0.50	0.25	0.14	0.04	0.06
EXISTS	0.15	0.49	0.21	0.16	0.48	0.22	0.33	1.00	0.49	0.21	0.06	0.10
IGRE	0.17	0.51	0.23	0.17	0.56	0.25	0.33	1.00	0.50	0.42	0.10	0.16
Table 3: Precision P, recall R, and the F1 score for symbols of different category: colour C1, shape S1, and spatial relation R2 (intrinsic evaluation), as well as reference prediction (extrinsic evaluation). Reported metrics are averaged across the words in each category and in turn averaged across the 10 different test visual scenes.
non-squares). These experiments suggest that this sort of correction to confident but wrong estimates of the denotations of open-class symbols happens sufficiently often at test time in the reference task to make a difference in this low-data regime we are interested in, for addressing ITL tasks.
The HEAD and EXISTS baselines never acquire negative exemplars: e.g., information that a particular individual is not red. Figure 2 shows that this severely impacts their performance, and error analysis showed that in some experiment runs it leads to model-collapse, with all denotations predicted to be in the extensions of all symbols. On the other hand, IGRE is able to acquire and use negative examples from the truth functional meanings of the logical symbols, specifically from: (a) negation ("not"); (b) the presupposition triggers "the  $N$ ", "  $N$  of the  $M$ ", and "all but  $N$  " where  $N$ , and  $M$  are numbers and "both"; and (c) the use of "every" when it modifies the head noun.
In this work, we presented IGRE—a grounder that supports incremental learning of the mapping from symbols to visual features whenever the teacher presents a linguistically complex RE and its desig-
nation(s) in the visual scene. The grounder starts the learning process with no conceptualisation of the domain model, and so the learner must revise its hypothesis space of possible domain models as and when the teacher introduces new and unforeseen concepts via neologisms. We showed how exploiting the model-theoretic interpretation of the formal semantic representations of REs, and in particular the truth conditions of 'logical' words like quantifiers and negation, can inform the acquisition of noisy training exemplars that in turn guide learning—IGRE reasons about the likely denotations of symbols within an RE that aren't designated by that RE, and when sufficiently confident it exploits them to update its grounding model. We showed that: 1) this grounding approach is more data efficient than a model that omits such observations and reasoning, using only the designated symbols; and 2) it is beneficial to exploit the logical consequences of the logical symbols, to gain even more data efficiency and training stability. In both cases, there was much to be gained from such reasoning because in contrast to the baselines, it contributes to acquiring negative exemplars: in other words, objects that get associated with not being red, for example.
IGRE uses a single source of data augmentation by acquiring noisy exemplars from symbols in oblique positions. Further and parallel data gains may be obtained by exploring semi-supervised learning methods (Yarowsky, 1995; Delalleau et al., 2005).
In this work, converting  $\mathcal{L}$ -sentences to conjunctive normal form, which is an NP-hard problem, was a computational bottleneck. Future work needs to address this by either considering lifted inference methods (e.g., den Broeck et al. (2011)) or defining model counters that use  $\mathfrak{L}$ -sentences directly.
Finally, the purpose of IGRE is to aid ITL: i.e., the (incremental) updates to beliefs about symbol grounding should enhance learning to solve domain-level planning problems. Future work needs to address this by using IGRE to learn planning tasks where the learner has the physical ability to execute certain actions but starts out unaware of domain concepts that define the goal and are critical to task success. The learner must not only use IGRE to interpret the teacher's feedback, but also learn decision making strategies, both on what to say (or ask) the teacher in their extended dialogue and what actions to perform in the environment. Furthermore, the static formal semantics that we used here should be replaced with a dynamic semantics (e.g., Groenendijk and Stokhof (1991); van der Sandt (1992); Asher and Lascarides (2003)), to account for how contextual salience influences truth and reference in dialogue. Following Batra et al. (2020), we plan to test the benefits of IGRE within a system that learns to solve planning problems that focus on rearrangement tasks.
This work was supported in part by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1), UKRI Strategic Priorities Fund to the UKRI Research Node on Trustworthy Autonomous Systems Governance and Regulation (grant EP/V026607/1, 2020-2024), and the Turing 2.0 'Enabling Advanced Autonomy through Human-AI Collaboration' project funded by EPSRC and the University of Edinburgh through the Alan Turing Institute.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-2433. IEEE Computer Society.
Nicholas Asher and Alex Lascarides. 2003. *Logics of Conversation*. Cambridge University Press.
Amos Azaria, Jayant Krishnamurthy, and Tom M. Mitchell. 2016. Instructable intelligent personal agent. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2681-2689. AAAI Press.
Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. Linguistics and Philosophy, 4:159-219.
Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. 2020. Rearrangement: A challenge for embodied AI. CoRR, abs/2011.01975.
Ben Bogin, Sanjay Subramanian, Matt Gardner, and Jonathan Berant. 2021. Latent compositional representations improve systematic generalization in grounded question answering. Transactions of the Association of Computational Linguisticsl (TACL), 9.
Jose Miguel Cano Santín, Simon Dobnik, and Mehdi Ghanimifard. 2020. Fast visual grounding in interaction: bringing few-shot learning with neural networks to an interactive robot. In Proceedings of the Probability and Meaning Conference (PaM 2020), pages 53-61, Gothenburg. Association for Computational Linguistics.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: universal image-text representation learning. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 104-120. Springer.
Ann A. Copestake. 2009. Slacker semantics: Why superficiality, dependency and avoidance of commitment can be the right way to go. In EACL 2009, 12th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, Athens, Greece, March 30 - April 3, 2009, pages 1-9. The Association for Computer Linguistics.
Ann A. Copestake, D. Flickinger, C. Pollard, and I. Sag. 1997. Minimal recursion semantics: An introduction. Research on Language and Computation, 3:281-332.
James R. Curran, Stephen Clark, and Johan Bos. 2007. Linguistically motivated large-scale NLP with c&c and boxer. In ACL 2007, Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic. The Association for Computational Linguistics.
Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive science, 19(2):233-263.
Samyak Datta, Karan Sikka, Anirban Roy, Karuna Ahuja, Devi Parikh, and Ajay Divakaran. 2019. Align2ground: Weakly supervised phrase grounding guided by image-caption alignment. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 2601-2610. IEEE.
Olivier Delalleau, Yoshua Bengio, and Nicolas Le Roux. 2005. Efficient non-parametric function induction in semi-supervised learning. In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics, AISTATS 2005, Bridgetown, Barbados, January 6-8, 2005. Society for Artificial Intelligence and Statistics.
Guy Van den Broeck, Nima Taghipour, Wannes Meert, Jesse Davis, and Luc De Raedt. 2011. Lifted probabilistic inference by first-order knowledge compilation. In *IJCAI* 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, July 16-22, 2011, pages 2178-2185. IJCAI/AAAI.
Jeffrey M. Dudek, Vu Phan, and Moshe Y. Vardi. 2020. ADDMC: weighted model counting with algebraic decision diagrams. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 1468-1476. AAAI Press.
Evelyn Fix and Joseph L. Hodges. 1989. Discriminatory analysis - nonparametric discrimination: Consistency properties. International Statistical Review, 57:238.
Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Nat. Lang. Eng., 6(1):15-28.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical
Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 457-468. The Association for Computational Linguistics.
Jeroen Groenendijk and Martin Stokhof. 1991. Dynamic predicate logic. Linguistics and Philosophy, 14:39-100.
Stevan Harnad. 1999. The symbol grounding problem. CoRR, cs.AI/9906002.
Yordan Hristov, Daniel Angelov, Michael Burke, Alex Lascarides, and Subramanian Ramamoorthy. 2019. Disentangled relational representations for explaining and learning from demonstration. In 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 870-884. PMLR.
Yordan Hristov, Alex Lascarides, and Subramanian Ramamoorthy. 2018. Interpretable latent spaces for learning from demonstration. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 957-968. PMLR.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 2261-2269. IEEE Computer Society.
Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726-2736, Florence, Italy. Association for Computational Linguistics.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. 2021. MDETR - modulated detection for end-to-end multi-modal understanding. CoRR, abs/2104.12763.
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. 2020. Learning adaptive language interfaces through decomposition. CoRR, abs/2010.05190.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 787-798. ACL.
Casey Kennington, Livia Dia, and David Schlangen. 2015. A discriminative model for perceptually-grounded incremental reference resolution. In Proceedings of the 11th International Conference on