SlowGuess/ACL_2020_OCR huggingface dataset:
Ravi Shekhar, Aashish Venkatesh, Tim Baumgartner, Elia Bruni, Barbara Plank, Raffaella Bernardi, and Raquel Fernandez. 2019. Beyond task success: A closer look at jointly learning to see, ask, and guess what. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2578-2587.
Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. 2019. The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents. arXiv preprint arXiv:1911.03768.
Carina Silberer and Mirella Lapata. 2012. Grounded models of semantic representation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1423-1433. Association for Computational Linguistics.
Mohammad S Sorower. 2010. A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis, 18:1-25.
Luc Steels. 2015. The Talking Heads experiment: Origins of words and meanings, volume 1. Language Science Press.
Florian Strub, Harm De Vries, Jeremie Mary, Bilal Piot, Aaron Courvile, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 2765-2771. AAAI Press.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, Florence, Italy. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanian Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316.
Jesse Thomason, Daniel Gordon, and Yonatan Bisk. 2019a. Shifting the baseline: Single modality performance on visual navigation & qa. In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1977-1983.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. 2019b. Vision-and-dialog navigation. arXiv preprint arXiv:1907.04957.
Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503-5512.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Zhibiao Wu and Martha Palmer. 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 133-138. Association for Computational Linguistics.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057.
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In Conference on Computer Vision and Pattern Recognition.
Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5014-5022.
7636
A.1 CompGuessWhat?! Dataset
We extend the GuessWhat?! dataset (de Vries et al., 2017) to promote the study of compositional grounded language representations. The original GuessWhat?! dataset has been enhanced by including a semantic layer on top of the purely perceptual one (i.e., images). In particular, we enrich the VisualGenome (Krishna et al., 2017) scene graphs associated with the GuessWhat?! images with several attributes coming from resources such as VISA (Silberer and Lapata, 2012) and ImSitu (Yatskar et al., 2016). As shown in Table 2 not all the GuessWhat?! images are included in VisualGenome. We were able to reuse  $40.79\%$  of the original GuessWhat?! dialogues for a total of 65,700 dialogues as summarised in Table 3.
Split	GuessWhat?! images	Mapped images
Train	46794	19117
Validation	9844	4049
Test	9899	3989
Table 2: Statistics of the mapping between Guess-What?! images and VisualGenome images.
A.2 VisualGenome object mapping
VisualGenome images are not exactly the same in terms of shape and content as the ones in MSCOCO. This is due to the presence of possible visual elements (i.e., banners) that are in the VisualGenome version of the image and are not in the MSCOCO one. This complicates the object mapping procedure used to link together abstract attributes and attributes coming from VisualGenome. As a first step, the procedure finds the largest VisualGenome bounding box with an IoU greater than 0.5. If there is not one, it looks for the largest VisualGenome bounding box with an IoU which is not close to 0 (with a tolerance of 0.05) and whose category is similar to the one of the reference MSCOCO one (where 'similar' is measured according to the Jaccard index between the corresponding category tokens). Whenever the MSCOCO object bounding box cannot be mapped to one of the VisualGenome bounding boxes, we assume that we do not have access to the situated attributes and we use the abstract attributes associated to its MSCOCO category only.
A.3 Diagnostic Classifiers for Attribute Prediction
For the attribute prediction task we apply a filtering procedure on the attribute set that will be used for training. In particular, we ignore all the attributes that belong to the abstract attribute category whose frequency is less than 100 (resulting in a set of attributes equal to 1997) and we ignore all the situated attributes whose frequency is less than 2 (resulting in a set of attributes equal to 4085).
For the attribute-prediction task we define a probing classifier  $\Phi$  as a linear transformation parameterised by a weight matrix  $\mathbb{R}^{d_d\times d_a}$  (where  $d_{d}$  is the dialogue hidden state size and  $d_{a}$  is the number of attributes to be predicted) followed by a sigmoid activation function. The number of input dimensions  $d_{d}$  depends on the model hidden state representations. We report in Table 4 the corresponding hidden state sizes for all the evaluated models. The output size  $d_{a}$  depends on the attribute set that we intend to consider. When situated attributes are considered  $d_{a} = 6082$ ,  $d_{a} = 1997$  for abstract attributes,  $d_{a} = 5$  for location attributes and  $d_{a} = 4085$  for situated-only attributes.
We consider the CompGuessWhat?! splits as reference for our experimental evaluation. We generate an hidden state for every successful dialogue and we use the classifier  $\Phi$  to predict the target object attributes. We train the classifier  $\Phi$  by minimising the binary cross-entropy loss computed between the model prediction and the reference set of attributes. We use ADAM (Kingma and Ba, 2014) as optimiser for our training procedure. To prevent overfitting, we perform early stopping on the validation set using the multi-label F1-measure (with threshold 0.75) as reference metric and we apply a learning rate scheduler to gradually reduce the learning rate. The model training has been implemented using AllenNLP (Gardner et al., 2018). We report the full set of metrics evaluated for this task in Table 5.
For the GDSE models we used a modified version of the code provided by the author via personal correspondence. On the other hand, for the DeVries model, we use the pretrained models and code that is available on the official webpage  $^{5}$ . The GloVe representations have been generated considering the dialogue as a long sequence of tokens and averaging the corresponding word embed
5https://github.com/GuessWhatGame/
7637
Split	# Guess What?!
dialogues	# CompGuess What?!
dialogues	Vocab. size	Avg. dialogue length	Successful dialogues	Failed dialogues	Incomplete dialogues
Train	113221	46277
(40.92%)	7090	5.128	84.06%
(38901)	10.35%
(4790)	5.59%
(2586)
Valid	23739	9716
(41.02%)	3605	5.106	83.97%
(8159)	11.03%
(1069)	5.03%
(488)
Test	23785	9619
(40.44%)	3552	5.146	84.10%
(8090)	10.74%
(1034)	5.14%
(495)
Table 3: Comparison between the original GuessWhat?! dataset and CompGuessWhat?! dataset. We report the percentage of dialogues that we retain after the filtering procedure based on the VisualGenome images.
Model	Hidden size
DeVries-SL	512
DeVries-RL	512
GDSE-SL	512
GDSE-CL	512
GDSE-SL-text	1024
GDSE-CL-text	1024
GloVe	300
ResNet	2048
Table 4: Summary of hidden state sizes for all the models considered in the attribute prediction evaluation.
dings. We used  $\mathrm{SpaCy}^6$  to obtain the representation of the entire dialogue. For the ResNet features we used the ones used by (Shekhar et al., 2019) based on a pretrained ResNet-152 model<sup>7</sup>.
Models such as GDSE adopt during training a specific constraint on the dialogue length. Particularly, they ignore dialogues having dialogue length greater than 10. This means that the model is never exposed to dialogues whose length is greater than 10. So for this family of models, for all those reference dialogues in GuessWhat?! having more than 10 turns, we consider only the last 10 turns and we generate the hidden state for the last turn. In general, we also assume that, whenever a model is not able to generate an hidden state representation for a given dialogue, we generate a zero vector. We did not change the behaviour in any way to avoid possible conflicts with the pretrained model. In addition, in this way a model that is not able to generate a representation for the dialogue would be penalised in the evaluation phase.
A.4 Zero-shot Evaluation Reference Games Generation
We define an automatic procedure to generate the set of reference games for the zero-shot learning setup. Specifically, for all the images in nocaps validation and test sets we first extract all the bounding boxes that satisfy the following conditions: 1) bounding box area should be greater than 500 pixels; 2) bounding box region should not be occluded; 3) bounding box region should not be truncated; 4) bounding box should not be associated with human body parts. An additional inclusion condition for the image is that the number of valid bounding boxes should be between 3 and 20. This 'sanity check' step is inspired by the procedure adopted in the original Guess What?! dataset (de Vries et al., 2017) and is used in order to guarantee that the gameplay reference images are not too crowded or composed of really small objects. Finally, we split the valid images in near-domain or out-of-domain. An image is considered near-domain if it contains at least an object whose category belongs to MSCOCO; we consider the image out-of-domain if it does not contain any MSCOCO category.
All the valid images resulting from the sanity check step can be considered as reference scene for the game. In order to define a fair comparison between all the agents, we define a reference set of games by sampling a fixed number of target objects for every image (e.g., 5 objects). In order to make sure that the sampling procedure is not biased by the frequency of the classes in the dataset, we sample an object according to the inverse of its category frequency in the dataset. As a result of this procedure, as shown in Table 6, we generated a dataset of 19, 179 near-domain reference games and 18, 672 out-of-domain reference games. In Figure 4 and 5 show the object category distribution in the near-domain and out-of-domain reference games, respectively.
$^{6}$ https://spacy.io/  
 $^{7}$ https://pytorch.org/docs/master/torchvision/models.html
7638
Figure 4: Object category distribution in the near-domain reference set of games.
Figure 5: Object category distribution in the out-of-domain reference set of games.
7639
Abstract	Situated-only	Abstract+situated	Location
Models	F1	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall
DeVries-SL	46.8	46.2	53.4	39.1	34.8	51.2	48.5	50.8	57.8	42.7	42.8	42.9
DeVries-RL	45.2	44.4	52.5	38.9	34.4	51	47.2	49.4	57.4	43.5	43.6	43.6
GDSE-SL	59.9	59.8	64.1	47.6	44	58.3	60.1	63.8	65.9	48.3	48.6	48.6
GDSE-CL	59.5	59.3	63.6	47.6	43.8	58.6	59.8	63.3	65.6	48.1	48.1	48.6
GDSE-SL-text	57	56.7	61.5	45.3	41.3	56.5	57.5	60.6	60.6	46	46.1	46.4
GDSE-CL-text	56.9	56.9	61.4	45	40.9	56.4	57.3	60.5	60.5	45	45	45.4
GloVe	34.6	33.6	45.9	29.7	25.1	42.1	36.4	37.4	52.9	33.6	33.6	33.7
ResNet	24.5	24.3	37.9	31.7	27.5	43.8	27.9	30.3	47.1	43.4	43.5	43.6
Random	15.1	40.8	16.3	0.1	50.6	0.1	7.8	50.3	5.4	27.5	49.7	20.3
Table 5: Full set of attribute prediction task metrics. We evaluate F1, Precision and Recall for all the tasks. All the metrics are computed as macro-average.
# images	# games
Near-domain validation	1208	5343
Out-of-domain validation	1306	5372
Near-domain test	3097	13836
Out-of-domain test	3212	13300
Table 6: Statistics for the CompGuessWhat?! zero-shot scenario. We provide near-domain and out-of-domain splits using specific nocaps images as reference scenes.
results of these analysis for the models Devries and GDSE analysed in this paper.
A.5 Generated Dialogue Evaluation
In order to evaluate to provide a more fine-grained evaluation of the generated dialogues, we adapt the quality evaluation script presented by (Shekhar et al., 2019) and extend it with additional metrics. First of all, it relies on a rule-based question classifier that classifies a given question in one of seven classes: 1) super-category (e.g., "person", "uten-sil", etc.), 2) object (e.g., "car", "oven", etc.), 3) "color", 4) "size", 5) "texture", 6) "shape" and "location". The question classifier is useful to evaluate the dialogue strategy learned by the models. In particular, we look at two types of turn transitions: 1) super-category  $\rightarrow$  object/attr, it measures how many times a question with an affirmative answer from the Oracle related to a super-category is followed by either an object or attribute question (where "attribute" represents the set {color, size, texture, shape and location}; 2) object  $\rightarrow$  attr, it measures how many times a question with an affirmative answer from the Oracle related to an object is followed by either an object or attribute question. We compute the lexical diversity as the type-token ratio among all games, question diversity and the percentage of games with repeated questions. We also evaluate the percentage of dialogue turns involving location questions. Table 7 and 8 show the
7640
Model	Lexical diversity	Question diversity	% games repeated questions	Super-cat -> obj/attr	Object -> attribute	% turns location questions	Vocab. size	Accuracy
DeVries-SL	0.76	44.64	12.54%	97.33%	73%	29.34%	2668	31.33%
DeVries-RL	0.13	1.77	99.48%	96.43%	98.63%	78.07%	702	43.92%
GDSE-SL	0.13	6.10	92.38%	95.60%	52.35%	15.74%	862	29.78%
GDSE-CL	0.17	13.74	66.76%	99.48%	67.25%	31.23%	1260	43.42%
Table 7: Gameplay quality analysis on Near-domain zero-shot reference games.
Model	Lexical diversity	Question diversity	% games repeated questions	Super-cat -> obj/attr	Object -> attribute	% turns location questions	Vocab. size	Accuracy
DeVries-SL	0.83	45.86	11.58	97.87%	76.50%	29.64%	2604	28.37%
DeVries-RL	0.24	2.96	98.49%	91.83%	98.58%	75.84%	1275	38.73%
GDSE-SL	0.09	1.31	97.19%	100%	67.45%	7.90%	519	22.32%
GDSE-CL	0.14	7.86	66.32%	100%	71.14%	26.03%	1002	29.83%
Table 8: Gameplay quality analysis on Out-of-domain zero-shot reference games.