SlowGuess/ACL_2020_OCR huggingface dataset:
ing ResNeXt-IG-3.5B, and TRANSRESNET $_{RET}$  using ResNet152 features. As in automatic evaluations, ResNet152 features performed more poorly. The retrieval model outperformed the generative model, a result that has been observed in other (text-only) dialogue tasks (Dinan et al., 2019; Zhang et al., 2018). In turn 1, TRANSRESNET $_{RET}$  (ResNeXt-IG-3.5B) has a win rate against humans of  $49.4\%$  (difference not significant using a binomial two-tailed test,  $p > 0.5$ ), while both other models are significantly outperformed by humans ( $p < 2 \times 10^{-7}$  compared to ResNet152 features), showing the importance of our retrieval architecture and image feature choices. We thus compare only TRANSRESNET $_{RET}$  (ResNeXt-IG-3.5B) to humans in all three turns (Fig. 4, right). That model performs well, with an overall win rate against humans of  $47.7\%$  (difference is significant,  $p < 7 \times 10^{-5}$ ). Example predictions of TRANSRESNET $_{RET}$  (ResNeXt-IG-3.5B) are given in Figure 3.
5.3 Transfer to the IGC Task
To test the strength of our task and models we consider transfer to the IGC of task of Mostafazadeh et al. (2017). In particular, we focus on their response task, which provides an image and a dialogue history of two utterances: a context utterance, followed by a question. The task is to then pro
Image	Style	Conversation
Turn 1 examples
A: Artful	Model predictions:This looks like a painting. Something out of a Norman Rockwell.Some wretched town!.Yeah, we have finally arrived at Grandpa's old barnhouse! I can't wait to get started with the restoration!
A: Monstrous
A: Earnest
A: Wishful	Model predictions:I hope one day to have a dog this majestic.
A: Opinionated	This puppy looks cold get him a blanket.
A: Imaginative	Puppies are just the universe's way of telling us everything will be okay.
A: Respectful	Model predictions:What an honor to have beautiful places like these to contemplate natures rocks at their best.
A: Shy	While the area looks nice, I'd stay away, fear of tsunamis leaves me paralyzed at the beach.
A: Confused	I don't understand why would you put rocks there?
Turn 2 examples
A: Empathetic	I understand that an open field full of beautiful foliage is a testament to our environment.Model predictions:
B: Aloof	It's just a regular old field.
B: Frivolous	So is every grass in the country
A: Playful	i would jump inside that waterfall
B: Questioning	Model predictions:
B: Amusing	Do you really think it is feasible?
Well I would try to top you with a backflip.
A: Maternal	It must be very dangerous if children play there.
B: Whimsical	Model predictions:
B: Blunt	Nonsense, this would be a great place to play!
I would rather find some lawn. Those rocks look uncomfortable
Turn 3 examples
A: Happy	This looks like something fun and exciting to do on vacation!
B: Anxious	oh dear what happens if he gets taken with the wind
A: Happy	Model prediction:
Ooh, the wind looks perfect to fly this awesome kite!
A: Pompous	This archway is so dull and boring. I could have done a much better job with the garden in my own back yard.
B: Offhand	Its about as boring as you!!
A: Pompous	Model prediction:Please. I could show you much prettier places in my own back yard.
A: Sweet	This is so beautiful, I love nature and would love to see this place and meet the people who live there!
B: Spontaneous	Then lets go there right now! Just you and me.
A: Sweet	I wish you could take me there, I would love to go with you!
Figure 3: Example predictions from our  $\mathrm{TRANSRESNET}_{RET}$  (MM-Sum) model on the evaluation set using all candidates for turns 1-3. Two speakers A & B with given style traits discuss a photo. The dialogue context before the model prediction is completed by humans, followed by one or more possible model responses, given different style conditioning. The model clearly uses the image, given style and dialogue history in formulating its response.
Figure 4: Human evaluations on IMAGE-CHAT. Engagingness win rates of pairwise comparisons between human utterances and  $\mathrm{TRANSRESNET}_{\mathrm{RET}}$  (ResNet152 or ResNeXt-IG-3.5B) or  $\mathrm{TRANSRESNET}_{\mathrm{GEN}}$ , comparing over the rounds of dialogue.
duce a response. This is clearly related to our task, except it focuses on answering questions, which our task does not. Our task is more varied as it was collected in an unconstrained way, unlike in IGC where they were asked to write a question. Nevertheless, assuming a question contains a ? or starts with who, what, when, where, why or how, our dataset contains 40,076 training utterances that are questions (11.3% of the data) and so it could be possible to produce responses to them. Without any fine-tuning at all, we thus simply took exactly the same best trained models and used them for their question response task as well.
Unfortunately, after contacting the authors of Mostafazadeh et al. (2017) they no longer have the predictions of their model available, nor have they made available the code for their human evaluation setup. However, the test set is available. We therefore attempted to reproduce the same setup as in their experiments, which we will also make publicly available upon acceptance.
Automatic Evaluation We measure our best  $\mathrm{TRANSRESNET}_{GEN}$  model's performance on the IGC test set in terms of BLEU-4. The results are shown in Fig. 5 (right). We find that our model outperforms the model from Mostafazadeh et al. (2017), achieving a score of 2.30 compared to 1.49.
Human Evaluation We compare the provided human response (from the test set) with 7 variants of our TRANSRESNET  $\mathsf{RET}$  model (mimicking their setup), whereby we have our model condition on 7 styles for which it performed well on evaluations in section 5.2. Annotators rated the quality of responses on a scale from 1 to 3, where 3 is the highest, reporting the mean over  $\sim 2\mathrm{k}$  questions. We then scale that by the score of human authored
Figure 5: IGC Evaluations. The best model from Mostafazadeh et al. (2017) is compared to our best  $\mathrm{TRANSRESNET}_{\mathrm{RET}}$  and  $\mathrm{TRASNRESNET}_{\mathrm{GEN}}$  models. On the left, annotator's ratings of responses from the models are shown as a percentage of the annotator's ratings of human responses. On the right, BLEU-4 scores on the response task are shown.
responses, to give a percentage. The results are shown in Fig. 5 (left). Our model narrows the gap between human and model performance, yielding a higher percentage of the human score (62.9% vs. 54.2%). More detailed results and example predictions of our model can be found in Appendices E and F, including examples of highly rated and poorly rated outputs from our model.
This paper presents an approach for improving the way machines can generate grounded conversations that humans find engaging. Focusing on the case of chit-chatting about a given image, a naturally useful application for end-users of social dialogue agents, this work shows that our best proposed model can generate grounded dialogues that humans prefer over dialogues with other fellow humans almost half of the time (47.7%). This result is made possible by the creation of a new dataset IMAGE-CHAT<sup>3</sup>.
Our work shows that we are close to having models that humans can relate to in chit-chat conversations, which could set new ground for social dialogue agents. However, our retrieval models outperformed their generative versions; closing that gap is an important challenge for the community. While our human evaluations were on short conversations, initial investigations indicate the model as is can extend to longer chats, see Appendix G, which should be studied in future work. The next challenge will also be to combine this engagingness with other skills, such as world knowledge (Antol et al., 2015) relation to personal interests (Zhang et al., 2018), and task proficiency.
3http://parl.ai/projects/image chatting
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and vqa. CVPR.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Dan Bohus and Eric Horvitz. 2009. Models for multi-party engagement in open-world dialog. In Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-234. Association for Computational Linguistics.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dólár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020. The second conversational intelligence challenge (convai2). In *The NeurIPS'18 Competition*, pages 187-208. Springer.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR).
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. Vse++: Improving visual-semantic embeddings with hard negatives.
Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. 2017. Stylenet: Generating attractive visual captions with styles. In Proc IEEE Conf on Computer Vision and Pattern Recognition, pages 3137-3146.
J. Gu, J. Cai, S. Joty, L. Niu, and G. Wang. 2018. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7181-7189.
Longteng Guo, Jing Liu, Peng Yao, Jiangwei Li, and Hanqing Lu. 2019. Mscap: Multi-style image captioning with unpaired stylized text. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4204-4213.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263-272.
Yuheng Hu, Lydia Manikonda, and Subbarao Kambhampati. 2014. What we instagram: A first analysis of instagram photo content and user types. In *Eighth International AAAI Conference on Weblogs and Social Media*.
Bernd Huber, Daniel McDuff, Chris Brockett, Michel Galley, and Bill Dolan. 2018. Emotional dialogue generation using image-grounded language models. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, page 277. ACM.
Margaret Li, Jason Weston, and Stephen Roller. 2019. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons. arXiv preprint arXiv:1909.03087.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. 2018. Exploring the limits of weakly supervised pretraining. In Computer Vision - ECCV 2018, pages 185-201, Cham. Springer International Publishing.
Sebastien Marcel and Yann Rodriguez. 2010. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM International Conference on Multimedia, MM '10, pages 1485-1488. ACM.
Alexander Mathews, Lexing Xie, and Xuming He. 2018. Semstyle: Learning to generate stylised image captions using unaligned text. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8591-8600.
Alexander Patrick Mathews, Lexing Xie, and Xuming He. 2016. Senticap: Generating image descriptions with sentiments. In AAAI, pages 3574-3580.
Pierre-Emmanuel Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium. Association for Computational Linguistics.
A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. Parlai: A dialog research software platform. In Empirical Methods in Natural Language Processing (EMNLP), pages 79-84.
Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios Spithourakis, and Lucy Vanderwende. 2017. Image-grounded conversations: Multimodal context for natural question and response generation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 462-472, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. 2016. Dual attention networks for multimodal reasoning and matching. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2156-2164.
Ramakanth Pasunuru and Mohit Bansal. 2018. Game-based video-context dialogue. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 125-136, Brussels, Belgium. Association for Computational Linguistics.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252.
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In *The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. Yfcc100m: The new data in multimedia research. Commun. ACM, 59(2):64-73.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of the 31st International Conference on Machine Learning, Deep Learning Workshop, Lille, France.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition.
S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He. 2017. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057.
Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 164-174, Melbourne, Australia. Association for Computational Linguistics.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Zhou Yu, Leah Nicolich-Henkin, Alan W Black, and Alexander Rudnicky. 2016. A wizard-of-oz study on a non-task-oriented dialog systems that reacts to user engagement. In Proceedings of the 17th annual meeting of the Special Interest Group on Discourse and Dialogue, pages 55–63.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
A More Details of IGC Evaluations
In this section we describe a few choices we made and implementation details regarding the IGC human evaluation in the section regarding Transfer to the IGC Task.
Multiple Traits In the IGC human evaluation setup from (Mostafazadeh et al., 2017), human annotators were shown eight choices when rating the quality of responses to questions: seven responses from various models, and one human response. To mirror this setup as closely as possible, we chose seven of our highest performing style traits to condition on to display in addition to the human response. We show the results of each trait in Table 4.
Automatic Evaluation In (Mostafazadeh et al., 2017), the authors provide BLEU scores for their models in an attempt to evaluate their effectiveness via automated metrics. The authors note that the scores are very low, "as is characteristic for tasks with intrinsically diverse outputs." Additionally, it has been shown in (Shuster et al., 2019) that BLEU scores for image captioning retrieval models are generally far lower than those of generative models (as retrieval models do not optimize for such a metric), and yet human evaluations can show the complete opposite results. In fact, in that work retrieval models were shown to be superior to generative models in human evaluations, which is why we adopted them here. For these reasons we omit BLEU scores of our retrieval models on the IGC test set as uninteresting. We do however compare BLEU scores with our generative model in the main paper.
Test Set Size The IGC test set provides the urls to all 2591 images for which (context, question, response) tuples were collected. We were only able to recover 2195 images from this initial set, as some of the URLs provided are no longer associated with the corresponding images. Thus, our human evaluations are conducted on this subset.
Style	Score
Neutral	1.55
Charming	1.55
Extravagant	1.55
Calm	1.57
Sweet	1.58
Spirited	1.60
Enthusiastic	1.61
Human	2.55
Table 4: IGC Human Evaluation on responses from our TRANSRESNET MM-SUM model conditioned on various personalities. Responses were rated on a quality scale from 1 to 3, where 3 is the highest.
B IMAGE-CHAT Human Annotation Setup
Respond to a Comment on an Image Description
In this task, you will be shown 5 images, each of which has a comment about the image. The goal of this task is to write an engaging response to this comment as if you were continuing a dialog about the image.
With each new photo, you will be given a personality trait that you will try to emulate in your response to the comment on the image. For example, you might be given "snarky" or "sentimental". The personality describes YOU, not the picture. It is you who is snarky or sentimental, not the contents of the image nor the original comment about the image.
You will then be shown an image and a comment that goes with the image, for which you will write a response in the context of your given personality trait. Please make sure your response has at least three words. Note that these are responses to the comments on the image, and not simply image captions.
Someone wrote the following comment on this image:
Peace and tranquility should be more abundant. This greenery evokes those feelings for me and I'm very thankful.
Write your response as if you were: Profound
Continue a Dialog on an Image
In this task, you will imagine that you are speaking with your friend about 5 separate images. For each image, you will be shown "your" initial comment on the image, and your friend's response to the comment. The goal of this task is to write an engaging response to your friend as if you were continuing a dialog about the image.
With each new photo, you will be given a personality trait that you will try to emulate in your response. For example, you might be given "adventurous". The personality describes YOU, not the picture. It is you who is adventurous, not the contents of the image.
You will then be shown an image, "your" initial comment that goes with the image, and your friend's response. You will continue the dialog by responding to your friend's response in the context of your given personality trait. Please make sure your response has at least three words. Note that these are not simply image captions, but engaging responses.
Figure 6: Instructions pane for crowdworkers when collecting the second round of dialogue.
YOU wrote the following comment on this image: I would be worried about getting cold out there.
Figure 7: Instructions pane for crowdworkers when collecting the third round of dialogue.
YOUR FRIEND responded:  
It's nice to just sit out in the snow and watch it fall. It's like being a whole different world.
Write your response as if you were: Maternal (Mother-like)
C IMAGE-CHAT Human Evaluation Setup
Rank Responses to Comments on an Image
In this task, you will be shown 5 images, and a short discussion about each image. The goal of this task is to pick which of two responses is the most engaging (interesting, captivating, attention-grabbing).
You will be shown an image, a short discussion, and a response. Additionally, you may be shown the personality of the person who wrote the response.
E.g., you may be shown an image of a tree, and the following discussion:
1. Comment: "What an absolutely beautiful tree! I would put this in my living room it's so extravagant!"
2. Response: "I bet I could climb that tree"
3. Responses to Evaluate: (a) I don't think you could, (b) Let me help you try!
And, you may be shown a personality, e.g. 'Cheerful'.
You will choose which response is more engaging.
E.g. in the example above, the second response (b) is more engaging than the first.
Person 1: This is so beautiful, I love nature and would love to see this place and meet the people who live there!  
Person 2: Then lets go there right now! Just you and me.
Personality of Person 1: Sweet
Person 1 Response:
Aww that's nice. I want to go on your boat with you!
Aww that would be nice just the two of us.
D IGC Human Evaluation Setup
Rate Quality of Responses
You will be shown an image, some textual context, a questions in response to the textual context, and a set of candidate responses to the question.
E.g., you may be shown an image of a tree; some textual context, i.e. "An amazing tree for climbing."; a question, "Do you think you could really climb that tree?"; and, a set of candidate responses:
1. "Are you kidding? I could climb that tree in my sleep."
2. "Is it time for dinner yet?"
You will rate each candidate response on a scale from 1 to 3, where 3 is the highest quality and 1 is the lowest quality.
E.g. in the example above, you might give the first question a "3" rating and the second question a "1" rating.
Figure 8: Instructions pane for crowdworkers when collecting the IMAGE-CHAT Evaluations.
Textual Context: I so love lightening storms.
Question: What is your favorite thing about them?
There's always something cosy and romantic about watching a thunderstorm from inside.
That lightning, striking its destination with amazing force and tenacity.
Whoa! Lightning is awesome!
Lightning is very dangerous, especially when you are outside.
How there is no end to how the lightening can fork out from itself.
Ratings (1 is lowest quality, 3 is highest quality)
Figure 9: Instructions pane for crowdworkers when collecting the IGC Evaluations.
Submit Choice