SlowGuess/ACL_2020_OCR huggingface dataset:
Alessandro Suglia<sup>1</sup>, Ioannis Konstas<sup>1</sup>, Andrea Vanzo<sup>1</sup>, Emanuele Bastianelli<sup>1</sup>, Desmond Elliott<sup>2</sup>, Stella Frank<sup>3</sup>, and Oliver Lemon<sup>1</sup>
<sup>1</sup>Heriot-Watt University, Edinburgh, UK
$^{2}$ University of Copenhagen, Copenhagen, Denmark
<sup>3</sup>University of Edinburgh, Edinburgh, UK
$^{1}\{as247,i.konstas,a.vanzo,e.bastianelli,o. lemon\} @hw.ac.uk$
2de@di.ku.dk
<sup>3</sup>stellla.frank@ed.ac.uk
Approaches to Grounded Language Learning typically focus on a single task-based final performance measure that may not depend on desirable properties of the learned hidden representations, such as their ability to predict salient attributes or to generalise to unseen situations. To remedy this, we present GROLLA, an evaluation framework for Grounded Language Learning with Attributes with three subtasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zeroshot evaluation. We also propose a new dataset CompGuessWhat?! as an instance of this framework for evaluating the quality of learned neural representations, in particular concerning attribute grounding. To this end, we extend the original GuessWhat?! dataset by including a semantic layer on top of the perceptual one. Specifically, we enrich the VisualGenome scene graphs associated with the GuessWhat?! images with abstract and situated attributes. By using diagnostic classifiers, we show that current models learn representations that are not expressive enough to encode object attributes (average F1 of 44.27). In addition, they do not learn strategies nor representations that are robust enough to perform well when novel scenes or objects are involved in gameplay (zero-shot best accuracy  $50.06\%$ ).
Several grounded language learning tasks have been proposed to capture perceptual aspects of language (Shekhar et al., 2017; Hudson and Manning, 2019; Suhr et al., 2019; Agrawal et al., 2018). However, the advances in this field have been primarily driven by the final performance measures and less on the grounding capability of the models. In fact, in some cases, high-performance models exploit dataset biases to achieve high scores on the final task (Zhang et al., 2016; Agrawal et al., 2016). In
Figure 1: Every gameplay in the CompGuess-What?! benchmark has a reference scene that is mapped to a scene graph composed of objects represented in terms of abstract and situated attributes.
the literature, several methods have been proposed to analyse what kind of information is captured by neural network representations (Kádár et al., 2017; Belinkov and Glass, 2019). Most of these works examine the hidden state representations learned by models trained on only textual data. However, many aspects of human semantic representations are grounded in perceptual experience (Andrews et al., 2009; Riordan and Jones, 2011). This paper explores the idea that visually grounded representations ought to be a result of systematic composition of grounded representations (Harnad, 1990). For instance, the understanding of the word "microwave" is grounded in perception of objects with specific attributes such as shape, colour, and size - see Figure 1 for an example. Therefore, investigating whether the representations learned by a model exhibit forms of attribute composition is beneficial for assessing model interpretability and generalisation.
In this work, we propose GROLLA - a multitask evaluation framework for Grounded Language Learning with Attributes that expands a goal
7625
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7625-7641 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: CompGuessWhat?!: Detailed description of the attributes of two different objects in the reference scene. Both the objects have a set of abstract attributes (indicated in blue) and a set of situated attributes (indicated in green).
oriented evaluation – based on the standard final task measure, with two auxiliary tasks: 1) Object attribute prediction (AP), and 2) Zero-shot evaluation (ZS). The attribute prediction task is designed to evaluate the extent to which the model's latent representations associated with objects are useful for predicting their attributes. The prediction performance on this task can be related to a degree of compositionality of the learned representations. We adopt a behavioural, i.e., task-driven, approach to assessing aspects of compositionality for visually grounded representations, whereby the extent to which a representation is compositional depends on: (a) its ability to predict object attributes, and (b) its ability to generalise to novel contributions of object attributes. To support (b), we design a zero-shot evaluation that measures the extent to which the learned representations can be reused in a task involving objects unseen during training. By optimising for both the final end-goal measure as well as the auxiliary tasks, we aim to drive the design of models that can solve the task more reliably and whose representations are easier to interpret as a result of being a composition of visual attributes.
This paper presents three main contributions: (1) We define GROLLA - a multi-task evaluation framework for grounded language learning that augments the final end-goal measure(s) with auxiliary tasks aimed at assessing the degree of attribute grounding of the model's representations; (2) We propose an instance of this multi-task evaluation framework, namely CompGuessWhat?!; and (3) We evaluate state-of-the-art models using the CompGuessWhat?! dataset. The evaluation shows that models with high performance in the end-goal task are not able to reliably predict the attributes of given objects and do not generalise to examples
with unseen object categories.
CompGuessWhat?! is a benchmark of 65,700 dialogues (see Section 3). It is based on GuessWhat?! (de Vries et al., 2017) dialogues and enhanced by including object attributes coming from resources such as VISA attributes (Silberer and Lapata, 2012), VisualGenome (Krishna et al., 2017) and ImSitu (Yatskar et al., 2016).
2 Evaluation Framework
Our evaluation framework for Grounded Language Learning tasks is based on three different sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; 3) Zero-shot evaluation.
Goal-oriented evaluation We evaluate the models according to the multi-modal task that they have to solve, which can generally be categorised as classification or generation. Classification tasks such as Visual Question Answering (Antol et al., 2015) or Visual Natural Language Inference (Suhr et al., 2019) involve predicting the correct label for a given example whose performance is measured in terms of predictive accuracy. In generative tasks, such as Image Captioning (Bernardi et al., 2016), the model has to learn to generate a sequence of labels for a given input data whose performance measure is BLEU (Papineni et al., 2002).
Object attribute prediction evaluation We support the goal-oriented evaluation with the attribute prediction auxiliary task related to assessing the degree of compositionality of the representations learned for a specific task.
With an attribute prediction task, we can assess whether the learned representations capture what we think they should, in terms of object attributes, rather than spurious correlations. The idea of using object attributes as an auxiliary task follows from the Characteristic Feature Hypothesis (Hampton, 1979) according to which every concept category has a set of defining features, which provide a criterion for judging which objects are category members, and which are not. Therefore, the higher the accuracy in the attribute prediction task, the more the representations learned by the model are composed of the set of attributes of the objects.
Zero-shot Evaluation Via the attribute prediction task, we can assess the ability of latent representations to recover some of the attributes associated with their object category. Assuming that the model has learned to represent these attributes, we
7626
hypothesise that it should solve the original task even when objects that have never been seen during training are involved.
In our evaluation framework, inspired by other multi-task evaluation frameworks (Wang et al., 2018; McCann et al., 2018; Wang et al., 2019; Shuster et al., 2019), we define Grounded Language Learning with Attributes (GROLLA) as the final score assigned to the model. It is computed as macro-average of the metrics over all tasks. We define the GROLLA score for convenience only and we underline the importance of having multiple scores for assessing different model abilities. In this work, we present CompGuessWhat?! as a dataset implementing this evaluation framework. Thanks to the high overlap between the image set of several datasets (Lu et al., 2019), future work will extend it to other grounded language learning tasks such as image captioning and visual navigation.
3 CompGuess What?! Benchmark
CompGuessWhat?! is an instance of our evaluation framework that is based on a guessing game (Steels, 2015), which can be viewed as a first step in a curriculum of language games for artificial agents. It involves two agents, a scene, and a target object: the Questioner asks questions in order to identify the target object in a scene, while the Oracle knows the target object and has to answer the questions. A multi-word guessing game requires two essential properties for grounded language learning: 1) the ability to generate discriminative questions aimed at narrowing down the search space (Natural Language Generation), and 2) the ability to understand the information provided so far during the game and exploit it to guess the target object (Natural Language Understanding).
3.2 Image Annotations Design
CompGuessWhat?! extends the GuessWhat?! dataset (de Vries et al., 2017) to promote the study of attribute-grounded language representations. The original GuessWhat?! dataset is extended with a semantic layer on top of the perceptual layer (i.e., images). This layer consists of a collection of intentional and extensional attributes of the objects in the reference image (Figure 2). We enrich the VisualGenome (Krishna et al., 2017) scene graphs associated with the GuessWhat?! images with several attributes coming from resources
such as VISA (Silberer and Lapata, 2012) and ImSitu (Yatskar et al., 2016). Unfortunately, not all the GuessWhat?! images are included in VisualGenome. We were able to reuse  $40.79\%$  of the original GuessWhat?! dialogues for a total of 65,700 dialogues (additional information can be found in the related Appendix A.1). By relying on this set of attributes, we define an attribute prediction evaluation to assess the extent to which the learned neural representations can encode the attributes specified during the dialogue. In order to determine the generalisation power of the learned representations and their ability to be transferred, we propose a novel zero-shot learning set of reference games involving target object belonging to an unseen object category. The dataset and the code associated with this paper can be found online<sup>1</sup>.
Psycholinguistically-motivated attributes We extend the set of attributes for every object category in MSCOCO with psycholinguistically-motivated semantic representations based on the McRae Norms (McRae et al., 2005) developed by Silberer and Lapata (2012). We use only the subset of so-called abstract attributes, and ignore attributes from the original set that can change depending on the reference image (e.g., "shape", "texture", etc.). We use the WordNet synset identifier (e.g., "person.n.01") associated with a given MSCOCO category (e.g., "person") to automatically associate its corresponding abstract attributes with a specific object instance. However, very often several VisualGenome objects have a synset associated with a class that is a hyponym of the MSCOCO category synset. Therefore, we rely on the Wu-Palmer similarity (Wu and Palmer, 1994) to find the best match between the VisualGenome synset and the MSCOCO category synset (with a similarity threshold of 0.75 chosen by using as reference the distance between the synset of person and woman). The intuition behind this heuristic is that we assume that a hyponym will inherit the abstract attributes of its hypernym.
Affordances & Behaviours We extract the semantic roles associated to specific object categories using the ImSitu dataset (Yatskar et al., 2016), in order to include affordances and behaviours associated with every object category. An object category is associated with a behaviour every time it appears as the agent of a given predicate. For in-
1https://compguesswhat.github.io
stance, "the food mixer [agent] blends fruit", where the behaviour is the food mixer's ability to blend something. We also consider affordances associated with a given category and divide them into two categories: 1) can be, every predicate having the object category as item, coagent, vehicle semantic role; 2) used to, every predicate having the object category as tool, heatsource, object. For example, in the statement "the person opens the oven [item]" an affordance can be intended as the fact that an oven can be opened. These attributes extend the set of abstract attributes. The abstract attributes do not depend on the reference image so they can be reused in other contexts as well.
Situated attributes Since the images contained in GuessWhat?! come from the MSCOCO dataset (see Figure 1 for an example), some of them are included in the VisualGenome (Krishna et al., 2017) dataset, which is composed of rich scene graphs for every image. In particular, we verified that 27,155 images from the GuessWhat?! dataset are also contained in VisualGenome. However, due to the presence of possible visual elements, the VisualGenome images are not the same as the MSCOCO ones. We use a heuristic approach based on both Intersection over Union (IoU) and language-only features to match the object bounding boxes between the two images. We report more details about the algorithm in Appendix A.2. The set of object attributes from VisualGenome (attribute types, colour, size, etc.) and location/positional attributes (one of top/bottom/left/right/centre, based on bounding box location) make up the situated attributes, which are specific to the reference image.
As a final step, due to the image mismatch, we decided to include the original GuessWhat?! object annotations in the VisualGenome graph in case a GuessWhat?! object cannot be mapped to a VisualGenome one. By doing this, we have access to the MSCOCO category of the object from which we can recover all its abstract attributes.
4 CompGuessWhat?! Evaluation
4.1 Guesser accuracy evaluation
We consider the guesser accuracy metric (in gameplay mode²) from the GuessWhat?! dataset for our goal-oriented evaluation. It measures how many
times the guesser model can select the correct target object among the candidate objects, given the dialogue generated so far. Due to the importance of this language game for NLU and NLG model skills, we decide to keep the guesser accuracy as a reference metric to assess the ability of the questioner to play the game. However, unlike the original dataset evaluation, we make sure that the score is evaluated ignoring duplicated dialogues.<sup>3</sup>
4.2 Attribute Prediction Evaluation
In a sequential guessing game like the one in Figure 1, we regard the representation for the last turn of the dialogue as a composition or aggregation of all the attributes specified so far. Therefore, we can use it to predict with high accuracy the attributes associated with a specific target object because it should encode the information needed to correctly discriminate the target from all the other objects in the scene. In the dialogue of Figure 1, when the model generates a representation for the last turn of the conversation (i.e., "Q: Is it the microwave? A: Yes"), it should encode the fact that "it is an appliance", "it is not the oven" and "it is the microwave", allowing the agent to guess the target object correctly.
By playing several guessing games that have a microwave as the target object, the agent should learn a representation of microwave that is expressive enough to correctly discriminate a microwave from all the other objects in a scene. In this setup we are not assuming that the model has a single representation for the concept of microwave; rather the concept of microwave develops from aggregating multimodal information related to microwaves across the situations in which the object is experienced (Barsalou, 2017). In the context of CompGuessWhat?!, every successful dialogue involving a microwave as the target object will be considered as an experience.
We are interested in understanding whether the dialogue state representation generated by a neural model for the last turn of the dialogue can encode the attributes of the target object specified during the dialogue. To do so, we define four attribute prediction tasks. For every target object we predict the corresponding vector composed of: 1) abstract attributes only (A); 2) situated attributes only (S),
3In the test dataset multiple conversations are associated with the same (image, target object) pair. Therefore, we want the pair (image, target object) to be considered only once in the accuracy evaluation.
2A gameplay involves three trained models that generate dialogues given a pair of (image, target object).
7628
3) the union of abstract and situated attributes (AS), and 4) location attributes  $(L)$  such as center, top, bottom, right and left. After training the model on the original GuessWhat?! dataset, we can generate dialogue representations corresponding to all the CompGuessWhat?! successful games. Then, we can train a diagnostic classifier that predicts the attributes associated with a given object category using the dialogue hidden representation generated for a given game as features. We hypothesise that a model that has learned grounded representations that are expressive enough to correctly guess the target object should retain the relevant features to predict its attributes.
We treat the attribute-prediction problem as a multi-label classification task. We implement our diagnostic classifier  $\Phi$  as a linear transformation parameterised by a weight matrix  $\mathbb{R}^{d_d\times d_a}$  (where  $d_{d}$  is the dialogue hidden state size and  $d_{a}$  is the number of attributes to be predicted) followed by a sigmoid activation function. We use a sigmoid activation function because it models a Bernoulli distribution. The diagnostic classifier outputs  $d_{a}$  logits where each of them models the probability  $P(y_{k} = 1|\mathbf{d})$  (where  $\mathbf{d}$  is dialogue state representation), one for each attribute  $y_{k}$  to be predicted. To mitigate a possible class-imbalance problem, we apply a filtering strategy to remove underrepresented attributes from our attribute set, which is a similar technique used to deal with out-of-vocabulary words. We also decided to avoid using class-weighting so that we could evaluate the power of the learned representations with simple linear classifiers as done in previous work using probing classifiers (Belinkov and Glass, 2019). Please refer to Appendix A.3 for details about the procedure to derive the reference set of attributes.
We use the CompGuessWhat?! dataset split as the reference for our training and evaluation setup: we train the diagnostic classifiers on CompGuessWhat?! gold training dialogues and evaluate their performance on the test dialogues using the validation set dialogues for early stopping. We consider Precision, Recall, and F1-measure for multi-label classification (Sorower, 2010) (computed as macro-average) and evaluate them with 0.5 as the threshold value for the sigmoid activation function (selected after considering the models performance using threshold values of 0.75 and 0.9). We report additional details in Appendix A.3.
4.3 Zero-shot Evaluation
Assuming that the model has learned to compose concepts during the turns of the dialogue, we hypothesise that it should also be able to use these representations to play games involving target objects that belong to categories that have never been seen before. For example, humans can discriminate between a dolphin and a dog even though they might not know what it is called. The measure presented in this section has the potential to demonstrate whether current models lack the ability to systematically generalise to new instances that are composed of attributes learned during training.
In order to assess the true generalisation power of the trained agents, we define a zero-shot learning scenario based on the nocaps dataset images (Agrawal et al., 2018). The nocaps dataset is composed of 3 evaluation splits: 1) in-domain: annotated objects belong to MSCOCO categories only; 2) near-domain: contains a mixture of MSCOCO and OpenImages objects; 3) out-of-domain: contains only OpenImages object categories. Since the number of categories in the original GuessWhat?! dataset (80) is lower than the number of categories in the Open Images dataset (660) – contained in nocaps – there are many categories that are never seen during training. Therefore, we can create zero-shot learning games by considering a target object for the game whose category has never been seen during training. We define an automatic procedure to generate the set of reference games for the zero-shot learning setup using the nocaps images. We split the nocaps images into near-domain or out-of-domain. An image is considered near-domain if it contains at least one object whose category belongs to MSCOCO. In contrast, we consider the image out-of-domain if it does not contain any MSCOCO category objects. This procedure generates a dataset of 19, 179 near-domain reference games and 18, 672 out-of-domain reference games. More details about the automatic procedure as well as the resulting reference set of games can be found in Appendix A.4. As a last step of our evaluation framework, we evaluate the performance of the state-of-the-art models in the zero-shot gameplay setup. For this task, the trained models need to interact with each other and generate dialogues given the pair (image, target object). As an evaluation metric for this task, we consider gameplay guesser accuracy for the near-domain (ND-Acc) and out-of-domain (OD-Acc) reference games.
7629
Gameplay Accuracy	Attribute Prediction	Zero-shot Gameplay	GroLLA
A-F1	S-F1	AS-F1	L-F1	ND-Acc	OD-Acc
Random	15.81%	15.1	0.1	7.8	2.8	16.9%	18.6%	13.3
GloVe	-	34.6	29.7	36.4	33.6	-	-	-
ResNet	-	24.5	31.7	27.9	43.4	-	-	-
GDSE-SL-text	-	57.0	45.3	57.5	46	-	-	-
GDSE-CL-text	-	56.9	45.0	57.3	45	-	-	-
DeVries-SL	41.5%	46.8	39.1	48.5	42.7	31.3%	28.4%	38.5
DeVries-RL	53.5%	45.2	38.9	47.2	42.5	43.9%	38.7%	46.2
GDSE-SL	49.1%	59.9	47.6	60.1	48.3	29.8%	22.3%	43.0
GDSE-CL	59.8%	59.5	47.6	59.8	48.1	43.4%	29.8%	50.1
Table 1: Results for state-of-the-art models on the CompGuessWhat?! suite of evaluation tasks. We assess model quality in terms of gameplay accuracy, the attribute prediction quality, measured in terms of F1 for the abstract (A-F1), situated (S-F1), abstract+situated (AS-F1) and location (L-F1) prediction scenario, as well as zero-shot learning gameplay. The final score GROLLA is a macro-average of the individual scores. We use the models GloVe, ResNet and GDSE-  $\star$  text only as a baseline for the attribute prediction tasks.
5 Results: Model Evaluation using CompGuessWhat?!
Guesser accuracy We evaluate the GDSE and DeVries models in gameplay mode using the set of reference games provided in CompGuessWhat?!. As shown in Table 1, the results are in line with the performance of the models on the original GuessWhat?! dataset (de Vries et al., 2017; Shekhar et al., 2019) confirming that our filtering strategy did not affect the complexity of the task.
Attribute Prediction We use the CompGuess-What?! benchmark to compare several dialogue state representations: