SlowGuess/ACL_2020_OCR huggingface dataset:
DeVries-SL: the representation learned by the Questioner model presented in (de Vries et al., 2017) that generates the question tokens conditioned on the image features and is trained using Supervised Learning (SL).
DeVries-RL: the representations learned by the Questioner model presented in (de Vries et al., 2017), fine-tuned using the Reinforcement Learning procedure proposed in (Strub et al., 2017).
GDSE-SL: the grounded dialogue state learned by a seq2seq model trained using the multi-task Learning procedure in (Shekhar et al., 2019).
GDSE-CL: the grounded dialogue state learned by the Questioner model used in GDSE-SL, finetuned with the Collaborative Learning procedure presented in (Shekhar et al., 2019).
GDSE-SL-text: the learned LSTM (Hochreiter
and Schmidhuber, 1997) dialogue encoder of the GDSE-SL model.
GDSE-CL-text:4 the learned dialogue encoder of the GDSE-CL model.
In order to control for possible bias in our task, we consider unimodal (Thomason et al., 2019a) as well as random attribute predictors:
GloVe: a dialogue is represented as the average of the GloVe embeddings associated with each word (Pennington et al., 2014).
ResNet: uses the latent representation of the reference scene generated by a ResNet152 as proposed in Shekhar et al. (2019).
Random: samples  $d_{a}$  scores from  $U(0,1)$  where samples are independent from each other. We incorporate this baseline as a lower bound performance on the attribute prediction task.
With the AP task, we try to answer the following question: "Do the representations associated with the target object encoding provide useful information that can be exploited to predict the object attributes correctly?" We assume that, due to the nature of the CompGuessWhat?! games, the final dialogue state representation should encode relevant features of the target object. So, a high gameplay accuracy should correlate with a high AP score. Table 1 summarises the results of the attribute prediction task evaluated on the CompGuessWhat?!
4We could use the dialogue encoder of the GDSE models only due to their modular architecture. It was not possible to properly separate the dialogue encoder from the visual representation in the DeVries models.
7630
test games. As the average best model performance was only 44.27, far from ceiling, our hypothesis is only partially supported. In particular, the models having the highest guesser accuracy, GDSE-CL and GDSE-SL, seem to learn better representations than unimodal baselines GloVe and ResNet, confirming the importance of multi-modal training for this task. There is also a gap in performance between the GDSE and DeVries models. This might be related to the multi-task learning strategy used by GDSE models that favours the emergence of more expressive representations than the ones learned by DeVries models which are trained in isolation. By comparing the enhanced versions GDSE-CL and DeVries-RL with the less sophisticated ones, GDSE-SL and DeVries-SL, respectively, we observe that, despite their higher guesser accuracy, these models do not have any advantage in terms of the AP task. We believe that this is because the Collaborative training strategy (for GDSE-CL) and Reinforcement Learning (for DeVries-RL) are optimising end-goal performance while sacrificing the expressiveness of the representations. Finding a way to encode task-specific representations and generalise them to learn abstract representations becomes an important research direction to improve on this task.
As an additional ablation, we compared the representations learned by the LSTM module used by GDSE to encode the dialogue (GDSE-  $\star$ -text) with their grounded dialogue state counterpart. Differences in terms of F1 are minimal, confirming that the heavy lifting is done by the textual representations and it is not clear how well the grounded dialogue state retains the visual information. Another confirmation of this issue is provided by the results in terms of location attributes prediction. Performance in this task for all the models is around 40 meaning that both VGGNet and ResNet features (used for DeVries and GDSE, respectively) are not able to recover fine-grained object information. This result sheds light on the ability of these models to ground the textual data in perceptual information of the reference scene. We believe that models should be able to co-ground one modality with the other and, as a result, learn more expressive grounded representations.
Zero-shot Evaluation Results are summarised in Table 1; the most striking observation is that all models struggle with this dataset (guesser accuracy is barely above 40), although arguably humans
would be able to solve the task despite their unfamiliarity with a specific object. Indeed, in this zero-shot scenario, reusing previously learned attributes that are shared among the objects or leveraging mutual exclusivity (Markman and Wachtel, 1988) would result in a successful gameplay.
Even the most accurate model in the CompGuess-What?! guesser evaluation performs poorly in this zero-shot setup (see Figure 3 for an example). We attribute this drop in performance to the way that these models represent objects. In particular, they all rely on category embeddings, i.e., latent representations associated to specific object categories (refer to (Shekhar et al., 2019; de Vries et al., 2017) for more details). In the case of ZS evaluation, when an object is unknown, its category embedding is also not available. This is true for both DeVries and GDSE models; it seems that GDSE models suffer more than DeVries models possibly due to overfitting. On the other hand, we aim to learn object representations which are not associated with manually-provided categories but are obtained by playing the game and that encode both abstract and situated attributes.
Once again, we find that models optimised using Reinforcement Learning seem to learn a better game strategy that results in higher performance on both near-domain and out-of-domain games. To better understand the quality of the generated dialogues, we classify each type of question according to a pre-defined set of types based on (Shekhar et al., 2019) (please refer to Appendix A.5 for a detailed description and a detailed summary of the evaluation results). We noticed that the Devries models generate dialogues with  $70\%$  of their turns comprising "location" questions (e.g., "is it the person on the right?") compared to  $20\%$  for GDSE models. We argue that to tackle zero-shot scenes, a model should instead learn features useful to discriminate the target object without relying on locations. Of course, in some reference scenes, location questions are still useful attributes used by humans when playing the game. In addition, asking location questions is an effective strategy because the Oracle has access to positional information that can be used to provide reliable answers but does not have any category embeddings for the target object.
Text-only Multi-task Evaluation In recent years progress in NLP has been driven by multi-
7631
Target object: crocodile
GDSE-CL
Failure
DeVries-RL
Failure
Figure 3: Example gameplay in the out-of-domain scenario of the two best performing systems GDSE-RL and DeVries-RL. The models have to play the game considering the crocodile as target object. This is a zero-shot scenario because the object category crocodile is not among the MSCOCO categories.
Question	Answer
is it a bird?	no
is it a vehicle?	no
is it a person?	no
is it wood?	no
is it a light?	no
is it near the person?	yes
is it to the right of the person?	no
is person wearing it?	no
is the person holding it?	yes
Question	Answer
is it an animal?	no
is it a picture?	no
is it a toilet?	no
is it tie?	no
is it a person?	no
is it a person?	yes
is it in the left?	no
is it in the middle?	yes
is it in the top?	yes
task evaluation datasets proposed to mitigate the biases of task-specific datasets (Wang et al., 2018; McCann et al., 2018; Wang et al., 2019). Despite their multi-task nature, these datasets focus on text-only data making the resulting models unable to learn meaning representations which are grounded in perceptual experience (Andrews et al., 2009; Riordan and Jones, 2011). Another downside is that these benchmarks focus only on end-goal metrics, i.e., are not informative on what the model has learned. Going beyond the end-goal metric is fundamental for designing models that are more generalisable and interpretable. By introducing the attribute prediction task in our framework, we assess whether the learned representations are expressive enough to predict the attributes of relevant objects in the scene. Also, we propose a zero-shot evaluation where the model has to generate predictions for examples that have never been seen during training, thus providing a way to understand the generalisation power of the learned representations.
Grounded Language Learning Evaluation
Several grounded language learning tasks have been proposed in the literature that can be divided into discriminative (Shekhar et al., 2017; Hudson and Manning, 2019; Suhr et al., 2019) and generative grounded language learning tasks (Xu et al., 2015; Agrawal et al., 2018). Recent works proposed models trained in a multi-task fashion by exploiting several language/vision tasks. The dodecaDialogue task (Shuster et al., 2019) proposes twelve dialogue tasks, among which there are two language/vision tasks in which the agent has to generate a response for a given context. Other works try to exploit multi-task learning to improve on single-task model performance in discriminative tasks (Pramanik et al., 2019;
Lu et al., 2019). Unfortunately, implementing multi-task learning using different datasets results is cumbersome (Subramanian et al., 2018). We propose an evaluation framework that can be applied in the context of a single task and dataset (e.g. GuessWhat?!) that allows to understand the extent to which the model can learn useful representations for the task at hand.
Inspecting the learned representations is important because, due to biases in the datasets, models might learn spurious correlations between input and output rather than actual grounding capabilities. For instance, in Visual Question Answering, questions starting with "What colour are" have "white" as a correct answer  $23\%$  of the time; models learn to memorise this sort of association rather than using the visual information (Zhang et al., 2016; Agrawal et al., 2016). This issue calls for a model evaluation aimed at inspecting the model representations as well as how these representations are used. The GQA (Hudson and Manning, 2019) dataset goes in this direction. It presents a Visual Question Answering dataset where images are supported by rich semantic annotations in the form of scene graphs. The GQA task requires the model to select an answer among a set of candidates.
However, we advocate the importance of tasks that involve both Natural Language Understanding (NLU) and Natural Language Generation (NLG) skills in a curriculum for grounded language learning. There are significant differences concerning the proposed auxiliary tasks as well. First of all, GQA's tasks are specifically designed around the VQA tasks to make sure that the model is consistent and plausible. It does not however tell us what the model's learned representations are encoding.
We propose the AP task as a diagnostic task
aimed at better understanding the learned neural representations (Belinkov and Glass, 2017; Conneau et al., 2018; Peters et al., 2018; Tenney et al., 2019). In addition, going beyond simple object classification is considered beneficial for vision systems (Farhadi et al., 2009) because it allows generalisation across object categories, not just across instances within a category. However, we believe that to truly assess the generalisation ability of a model, object attributes have to be used for the downstream task, which is not necessarily needed in object classification tasks. With the ZS evaluation, we investigate the ability of the models to exploit more fine-grained visual attributes which is important for models able to learn from few examples and easily transfer to new domains.
Compositionality Evaluation Andreas (2019) presents a method to estimate the degree of compositionality of neural representations by using an oracle compositional model aware of the compositional structure (i.e., a derivation) of the input data. Building a reference oracle is easy for synthetic scenes (as in Andreas (2019)) but is a significant challenge for real-world scenes. Previous work has studied compositionality in real-world scenes for visual concept composition (Misra et al., 2017) and image captioning (Nikolaus et al., 2019). In our benchmark CompGuessWhat?!, we use realworld scenes from the MSCOCO (Lin et al., 2014) and OpenImages (Kuznetsova et al., 2018) datasets. Our AP task is related to measuring compositionality. It relies on image annotations in the form of intensional and extensional attributes as a reference structure for the objects in the scene.
7 Conclusions & Future Work
We proposed CompGuessWhat?! as an implementation of GROLLA, a multi-task evaluation framework for Grounded Language Learning with Attributes. We found that the best performing model achieves a GROLLA score of  $50.06\%$ ; notably this model's out-of-domain accuracy is under  $30\%$ , as compared to the human performance on the original GuessWhat?! dataset of  $90.2\%$  (de Vries et al., 2017). Clearly, even models with high in-domain gameplay success rates still have difficulty generalising to new scenarios. In the following, we discuss insights gained from the evaluation and new research directions for this task.
The attribute prediction task shows that model representations are not able to accurately recover
attribute representations. We argue that this result calls for new approaches to exploiting and representing textual and visual data. We believe that models should be equipped with a co-grounding operator that fuses the textual and visual modalities. For instance, in the context of CompGuessWhat?! it would be used to learn a representation for the current turn that is influenced by both the language and visual modality. CompGuessWhat?! requires models to learn to combine the co-grounded information provided for every turn. Therefore, we propose that CompGuessWhat?! represents a benchmark dataset for evaluating the design of such an attribute compositionality operator that would be a possible implementation of compositionality à la Barsalou (2017).
In this work, we have shown how our multi-task evaluation framework can be applied to Guess-What?!. However, the same framework could be applied to other multi-modal tasks. For example, in image captioning, the goal-oriented evaluation would be the textual similarity metrics (e.g. BLEU); the attribute-prediction task would use the decoder representation to predict the attributes of the objects in the image (Elliott and Kádár, 2017, e.g.); and the zero-shot setting could leverage the nocaps dataset (Agrawal et al., 2018). Likewise, in the Vision-and-Dialog navigation task (Thomason et al., 2019b), the goal-oriented evaluation is the navigation task; attribute prediction is based on predicting the attributes of the hidden object when the agent decides it is in the correct room, and the zero-shot setting could evaluate model performance on novel combinations of rooms and object types.
Finally, from the evaluation presented here, it emerges that these models learn task-specific representations that do not generalise to unseen object categories. We hope that GROLLA and the CompGuessWhat?! data will encourage the implementation of learning mechanisms that fuse task-specific representations with more abstract representations to encode attributes in a more compositional manner. In addition, we will use the CompGuessWhat?! image annotations to design a visual grounding evaluation to assess the ability of the model to attend to the correct objects during the turns of the dialogue.
We thank Arash Eshghi and Yonatan Bisk for fruitful discussions in the early stages of the project.
7633
Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016. Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1955-1960.
Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2018. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658.
Jacob Andreas. 2019. Measuring compositionality in representation learning. arXiv preprint arXiv:1902.07181.
Mark Andrews, Gabriella Vigliocco, and David Vinson. 2009. Integrating experiential and distributional data to learn semantic representations. *Psychological review*, 116(3):463.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Lawrence W Barsalou. 2017. Cognitively plausible theories of concept composition. In Compositionality and concepts in linguistics and psychology, pages 9-30. Springer, Cham.
Yonatan Belinkov and James Glass. 2017. Analyzing hidden representations in end-to-end automatic speech recognition systems. In Advances in Neural Information Processing Systems, pages 2441-2451.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from images: A survey of models, datasets, and evaluation measures. Journal of Artificial Intelligence Research, 55:409-442.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136.
Desmond Elliott and Ákos Kádár. 2017. Imagination improves multimodal translation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 130-141.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1778-1785. IEEE.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. Allennlp: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640.
James A Hampton. 1979. Polymorphous concepts in semantic memory. Journal of verbal learning and verbal behavior, 18(4):441-461.
Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-346.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700-6709.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. 2018. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00982.
Ákos Kádár, Grzegorz Chrupa, and Afra Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43(4):761-780.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2019. 12-in-1: Multi-task vision and language representation learning. arXiv preprint arXiv:1912.02315.
7634
Ellen M Markman and Gwyn F Wachtel. 1988. Children's use of mutual exclusivity to constrain the meanings of words. Cognitive psychology, 20(2):121-157.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
Ken McRae, George S Cree, Mark S Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior research methods, 37(4):547-559.
Ishan Misra, Abhinav Gupta, and Martial Hebert. 2017. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1792-1801.
Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliott. 2019. Compositional generalization in image captioning. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 87-98, Hong Kong, China.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509.
Subhojeet Pramanik, Priyanka Agrawal, and Aman Hussain. 2019. Omninet: A unified architecture for multi-modal multi-task learning. arXiv preprint arXiv:1907.07804.
Brian Riordan and Michael N Jones. 2011. Redundancy in perceptual and linguistic experience: Comparing feature-based and distributional models of semantic representation. Topics in Cognitive Science, 3(2):303-345.
Ravi Shekhar, Sandro Pezzelle, Yauhen Klimovich, Aurélie Herbelot, Moin Nabi, Enver Sangineto, and Raffaella Bernardi. 2017. Foil it! find one mismatch between image and language caption. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 255-265.