SlowGuess/ACL_2020_OCR huggingface dataset:
Kurt Shuster, Samuel Humeau, Antoine Bordes, Jason Weston
{kshuster,samuelhumeau,abordes,jase}@fb.com
To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the ImageChat test set (preferred  $47.7\%$  of the time).
A key way for machines to exhibit intelligence is for them to be able to perceive the world around them – and to be able to communicate with humans in natural language about that world. To speak naturally with humans it is necessary to understand the natural things that humans say about the world they live in, and to respond in kind. This involves understanding what they perceive, e.g. the images they see, what those images mean semantically for humans, and how mood and style shapes the language and conversations derived from these observations.
In this work we take a step towards these goals by considering grounded dialogue involving open
ended discussion of a given image, a setting that is naturally fun for humans (Hu et al., 2014), and study neural conversational models for task. In particular, we explore both generative and retrieval models that handle multimodal dialogue by fusing Transformer architectures (Vaswani et al., 2017) for encoding dialogue history and responses and ResNet architectures (He et al., 2016) for encoding images. We propose ways to fuse those modalities together and perform a detailed study including both automatic evaluations, ablations and human evaluations of our models using crowdworkers.
To train and evaluate such models, we collect a large set of human-human crowdworker conversations, with the aim of training a model to engage a human in a similar fashion, consisting of  $202\mathrm{k}$  diverse images and 401k utterances over the images, with 215 different style traits (e.g., optimistic, skeptical or frivolous) to promote engaging conversation. The dataset is made publicly available in ParlAI (Miller et al., 2017)
Our results show that there is a significant gap between state-of-the-art retrieval and generative models on this task. Our best fused retrieval models set a strong baseline, being preferred to human conversationalists  $47.7\%$  of the time. We show that both large-scale image and text pre-training, and utilization of style traits, are critical for best results. We then consider transfer to the existing Image Grounded Conversations (IGC) task of Mostafazadeh et al. (2017), where we obtain state-of-the-art results.
The majority of work in dialogue is not grounded in perception, e.g. much recent work explores sequence-to-sequence models or retrieval models for goal-directed (Henderson et al., 2014) or chit
http://parl.ai/projects/image chatting
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2414-2429 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
chat tasks (Vinyals and Le, 2015; Zhang et al., 2018). While these tasks are text-based only, many of the techniques developed can likely be transferred for use in multimodal systems, for example using state-of-the-art Transformer representations for text (Mazare et al., 2018) as a sub-component.
In the area of language and vision, one of the most widely studied areas is image captioning, whereby a single utterance is output given an input image. This typically involves producing a factual, descriptive sentence describing the image, in contrast to producing a conversational utterance as in dialogue. Popular datasets include COCO (Chen et al., 2015) and Flickr30k (Young et al., 2014). Again, a variety of sequence-to-sequence (Vinyals et al., 2015; Xu et al., 2015; Anderson et al., 2018) and retrieval models (Gu et al., 2018; Faghri et al., 2018; Nam et al., 2016) have been applied. These tasks measure the ability of models to understand the content of an image, but not to carry out an engaging conversation grounded in perception. Some works have extended image captioning from being purely factual towards more engaging captions by incorporating style while still being single turn, e.g. (Mathews et al., 2018, 2016; Gan et al., 2017; Guo et al., 2019; Shuster et al., 2019). Our work also applies a style component, but concentrates on image-grounded dialogue, rather than image captioning.
Visual question answering (Antol et al., 2015) and visual dialogue (Das et al., 2017) are another set of tasks which employ vision and language. They require the machine to answer factual questions about the contents of the image, either in single turn or dialogue form. They do not attempt to model natural conversation, but rather assess whether the machine can perform basic perception over the image via a series of questions.
There are some works which directly address dialogue grounded with vision. The work of Pasunuru and Bansal (2018) assesses the ability to execute dialogue given video of computer soccer games. The work of Huber et al. (2018) investigates the use of sentiment-based visual features and facial expressions for emotional image-based dialogue. Perhaps the most related work to ours is Mostafazadeh et al. (2017). Their work considers (visual context, textual context, question, response) tuples, and builds validation and test sets based on 4k eventful images called Image Grounded Conversations (IGC). No training data is provided, but instead the authors
use Twitter for that in their experiments. In contrast, we provide training, validation and testing sets over 202k images for our task (that do not overlap with IGC), and consider a general set of images and dialogues, not just events and questions plus responses. In our experiments we also show strong transfer ability of our models to the IGC task.
While there are many ways to measure dialogue quality, human engagement is a popular metric. Engagement itself can be measured in many ways (Bohus and Horvitz, 2009; Yu et al., 2016) but here we adopt the common approach of simply asking humans which speaker they find more engaging, following other works (Li et al., 2019; Dinan et al., 2020).
3 Image-Chat
The IMAGE-CHAT dataset is a large collection of (image, style trait for speaker A, style trait for speaker B, dialogue between A & B) tuples that we collected using crowd-workers. Each dialogue consists of consecutive turns by speaker A and B. No particular constraints are placed on the kinds of utterance, only that we ask the speakers to both use the provided style trait, and to respond to the given image and dialogue history in an engaging way. The goal is not just to build a diagnostic dataset but a basis for training models that humans actually want to engage with.
Style Traits A number of works have shown that style traits for image captioning help provide creative captions (Mathews et al., 2018, 2016; Gan et al., 2017; Shuster et al., 2019). We apply that same principle to image grounded dialogue, considering a set of 215 possible style traits, using an existing set from Shuster et al. (2019). The traits are categorized into three classes: positive (e.g., sweet, happy, eloquent, humble, witty), neutral (e.g., old-fashioned, skeptical, solemn, questioning) and negative (e.g., anxious, childish, critical, fickle, frivolous). We apply these to both speakers A and B, who will be assigned different style traits for each given conversation.
Images The images used in our task are randomly selected from the YFCC100M Dataset2 (Thomee et al., 2016).
Dialogue For each image, we pick at random two style traits, one for speaker A and one for speaker
2https://multimediacommons.wordpress.com/yfcc100m-core-dataset/
A: Peaceful
B: Absentminded
Figure 1: Some samples from the IMAGE-CHAT training set. For each sample we asked humans to engage in a conversation about the given image, where the two speakers, A and B, each have a given provided style.
A: I'm so thankful for this delicious food.
B: What is it called again?
A: Not sure but fried goodness.
A:Fearful
B: Miserable
A: I just heard something out there and I have no idea what it was.
B: It was probably a Wolf coming to eat us because you talk too much.
A: I would never go camping in the woods for this very reason.
A: Erratic
B: Skeptical
A: What is the difference between the forest and the trees? Oh look, dry pavement.
B: I doubt that's even a forest, it looks like a line of trees.
A: There's probably more lame pavement on the other side!
B, and collect the dialogue using crowdworkers who are asked to both assume those roles, and to be engaging to the other speaker while doing so. It was emphasized in the data collection instructions that the style trait describes a trait of the speaker, not properties of the content of the image they are discussing. Some examples from the training set are given in Figure 1.
Data Quality During data collection crowdsourceders were manually monitored, checking to ensure they were following the instructions. Poor performers were banned, with comments discarded. A verification process was also conducted on a subset of the data, where separate annotators were asked to choose whether the utterance fit the image, style, or both, and found that  $92.8\%$  of the time it clearly fit the image, and  $83.1\%$  the style, and  $80.5\%$  both. Note, given that not all utterances should directly reference an image property or invoke the style, we do not expect  $100\%$ .
Overall Dataset The overall dataset statistics are given in Table 1. This is a fairly large dialogue dataset compared to other existing publicly available datasets. For example, PersonaChat (Zhang et al., 2018) (which is not grounded in images) consists of  $162\mathrm{k}$  utterances, while IGC (Mostafazadeh et al., 2017) (grounded in images) consists of  $4\mathrm{k}$  of validation and test set examples only, compared to over  $400\mathrm{k}$  utterances in IMAGE-CHAT.
Split	train	valid	test
Number of Images	186,782	5,000	9,997
Number of Dialogues	186,782	5,000	9,997
Number of Utterances	355,862	15,000	29,991
Style Types	215	215	215
Vocabulary Size	46,371	9,561	13,550
Tokens per Utterance	12.3	12.4	12.4
Table 1: IMAGE-CHAT dataset statistics.
We consider two major types of dialogue model: retrieval and generative. Both approaches make use of the same components as building blocks. We use three sub-networks for the three modalities of input: (i) an image encoder, (ii) a dialogue history encoder; and (iii) a style encoder. In the retrieval model these are then fed into a combiner module for combining the three modalities. Finally, there is a response encoder for considering candidate responses and this is scored against the combined input representations. An overview of the retrieval architecture is shown in Figure 2. For the generative model, the three encoders are used as input, and a further decoder Transformer is used for outputting a token sequence; beam search is applied.
Image Encoder We build our models on top of pretrained image features, and compare the performance of two types of image encoders. The first is a residual network with 152 layers described in He et al. (2016) trained on ImageNet (Russakovsky et al., 2015) to classify images among 1000 classes, which we refer to in the rest of the pa
Figure 2: The TRANSRESNET $_{RET}$  multimodal architecture for grounded dialogue. There are several options: different image encoders (ResNet152 or ResNeXt-IG-3.5B), text encoders (shared or separate Transformers for history and response), and different multimodal combiners (sum or attention-based).
per as ResNet152 features. We used the implementation provided in the torchvision project (Marcel and Rodriguez, 2010). The second is a ResNeXt  $32 \times 48\mathrm{d}$  (Xie et al., 2017) trained on 3.5 billion Instagram pictures following the procedure described by Mahajan et al. (2018), which we refer to in the rest of the paper as ResNeXt-IG-3.5B. The representation  $r_I$  of an image  $I$  is obtained by using the 2048-dimensional output of the image encoder as input to a feed-forward network: a multi-layer perceptron with ReLU activation units and a final layer of 500 dimensions in the retrieval case, and a linear layer in the generative case.
Style Encoder To condition on a given style trait, we embed each trait to an  $N$ -dimensional vector to obtain its representation  $r_{S}$ . We used  $N = 500$  for retrieval and  $N = 300$  for generation.
Dialogue Encoder The entire dialogue history  $D$  is encoded into a fixed size vector  $r_D$  using a Transformer architecture (Vaswani et al., 2017), followed by a linear layer. Such Transformers have been shown to perform strongly on a variety of dia
logue tasks previously (Yang et al., 2018; Mazare et al., 2018). We use a Transformer with 4 layers, 300 hidden units, and 6 attention heads. The outputs are pooled (mean) to give a final vectorial encoding.
We pretrain the entire encoder following the setup described in Mazare et al. (2018): we train two encoders on a next-utterance retrieval task on a Reddit dataset of dialogues containing 1.7 billion pairs of utterances, where one encodes the context and another the candidates for the next utterance; their dot product indicates the degree of match, and they are trained with negative log-likelihood and  $k$ -negative sampling. We then initialize our system using the weights of the candidate encoder only, and then train on our task in either generative or retrieval mode.
4.1 Retrieval Models
Multimodal combiner module We consider two possible combiner modules for the inputs:
Multimodal sum combiner (MM-sum): Given an input image, style trait and dialogue  $(I,S,D)$ , together with a candidate response  $C$ , the score of the final combination is computed as  $s(I,S,D,C) = (r_I + r_S + r_D)\cdot r_C$
Multimodal attention combiner (MM-att): A more sophisticated approach is to use an attention mechanism to choose which modalities are most relevant for each example by stacking Transformers. We concatenate the three representation vectors  $r_I$ ,  $r_S$  and  $r_D$  and feed them to a second Transformer (4 attention heads, 2 layers, 500 hidden units) which performs self-attention over them. The three modalities are thus reweighted by the corresponding attention weights to give the final input representation vector  $r_T$ , which is used to compute the score for a given candidate using  $r_T \cdot r_C$ .
Response encoder We employ the same Transformer architecture as in the dialogue encoder for encoding candidate responses. We tried two variants: either sharing or not sharing the weights with the input dialogue encoder.
Training and Inference Given a tuple  $I, S, D$ , and a set of candidates  $(c_1,.., c_N)$ , at inference time the predicted utterance is the candidate  $c_i$  that maximizes the score  $s(I, S, D, c_i)$ . At training time we pass a set of scores through a softmax and train to maximize the log-likelihood of the correct responses. We use mini-batches of 500 training
examples; for each example, we use the gold responses of the other examples of the batch as negatives. During final human evaluation all candidates from the training set are considered to produce a response (356k candidates in our experiments).
4.2 Generative Models
Dialogue Decoder The encoding from the image encoder has a final linear layer of dimension 2048  $\times 300$ . This projects it to the same size of the token encoding of the dialogue decoder. We thus add it as an extra token at the end of the Transformer's encoder output. For style, we simply prepend the style to the beginning of the dialogue history, and it is thus encoded in the dialogue encoder. We then treat this as a standard seq2seq Transformer in order to generate dialogue responses.
Training and Inference We train with a batch size of 32 and learning rate of .0001 using adam, and apply beam search with a beam of size 2 and tri- gram blocking at inference time. Hyperparameters are chosen on the validation set.
We test our models on the IMAGE-CHAT and IGC datasets using automatic metrics and human evaluations. We analyze the performance of the different module and architecture choices, as well as ablation studies to determine the importance of each of the model's inputs.
5.1 Automatic Evaluation on IMAGE-CHAT
Module Choices We first compare various module configurations of our TRANSRESNET $_{RET}$  model, and additionally show the results for a simple information retrieval baseline, in which the candidates are ranked according to their weighted word overlap to the input message. We measure recall at 1 and 5 (R@1/100 and R@5/100) retrieval metrics, where for each sample there are 100 candidates to rank: 99 random candidates chosen from the test set, and the true label. Note that in human evaluations we use all the train set candidates.
The results are shown in Table 2. We report the average metrics for the total task, as well as the breakdown of the performance on each turn of dialogue (turns 1, 2 and 3). The average metrics indicate that using the ResNeXt-IG-3.5B image encoder features improves performance significantly across the whole task, as we obtain  $50.3\%$  R@1 for our best ResNeXt-IG-3.5B model and only  $40.6\%$
for our best ResNet152 model. When broken down by turn, it appears that the ResNeXt-IG-3.5B features are particularly important in the first round of dialogue, in which only the image and style are considered, as the difference between their best models increases from  $9.7\%$  in the full task to  $19.5\%$  in the first turn. Our baseline multimodal sum combiner (MM-Sum) outperforms the more sophisticated self-attention (MM-Att) combiner, with the latter scoring  $49.3\%$  on the full task. Having separate candidate and dialogue history text encoders also works better than sharing weights.
In subsequent experiments we use the best performing system for our retrieval model. As ResNeXt-IG-3.5B performs best we use that for our generative model going forward as well.
Full & Ablation Study We now perform experiments for both retrieval and generative models for the full system, and additionally we remove modalities (image, style, and dialogue history). For the generative models we report the ROUGE-L metric. The results are shown in Table 3, which we now analyze.
Turn 1: In the first round of dialogue the models produce utterances given the image and style only, as there is no dialogue history yet. For both models, image is more important than style, but using both together helps.
Turn 2: In the second turn, in which a model produces a response to a first utterance, the models perform similarly when using only the image or only the dialogue history, while performing poorly with just the style. Any combination of two modalities improves the results, with the style + dialogue combination performing slightly higher than the other two. Using all modalities works best.
Turn 3: By the third turn of dialogue, the conversation history proves to be by far the most important in isolation compared to the other two modalities in isolation. Conditioning on the style+dialogue is the most effective of any combination of two modalities. Again, using all modalities still proves best.
5.2 Human Evaluations on IMAGE-CHAT
We test our final models using human evaluation.
Evaluation Setup We use a set of 500 images from YFCC-100M that are not present in IMAGECHAT to build a set of three-round dialogues pairing humans with models in conversation. We then
Model	Combiner	Text Encoders R@1	Image Encoder R@1	Turn 1 R@1	Turn 2 R@1	Turn 3 R@1	All
R@1	R@5
IR Baseline	n/a	n/a	n/a	-	-	-	2.15	5.86
TRANSRESNETRET	MM-Att	Separate	ResNet152	35.7	44.5	40.5	40.2	67.0
TRANSRESNETRET	MM-Sum	Separate	ResNet152	34.5	46.0	41.3	40.6	67.2
TRANSRESNETRET	MM-Sum	Shared	ResNeXt-IG-3.5B	53.6	47.0	41.3	47.3	73.1
TRANSRESNETRET	MM-Att	Shared	ResNeXt-IG-3.5B	54.4	49.0	43.3	48.9	74.2
TRANSRESNETRET	MM-Att	Separate	ResNeXt-IG-3.5B	53.5	50.5	43.8	49.3	74.7
TRANSRESNETRET	MM-Sum	Separate	ResNeXt-IG-3.5B	54.0	51.9	44.8	50.3	75.4
Table 2: Module choices on IMAGE-CHAT. We compare different module variations for TRANSRESNET  ${}_{\text{RET }}$  .
Modules	TRANSRESNETRET (R@1/100)	TRANSRESNETGEN (ROUGE-L)
Turn 1	Turn 2	Turn 3	All	Turn 1	Turn 2	Turn 3	All
Image Only	37.6	28.1	20.7	28.7	21.1	21.9	22.4	21.8
Style Only	18.3	15.3	17.0	16.9	20.2	20.9	22.0	21.0
Dialogue History Only	1.0	33.7	32.3	22.3	18.9	22.7	23.7	21.8
Style + Dialogue (no image)	18.3	45.4	43.1	35.4	20.4	24.1	24.8	23.1
Image + Dialogue (no style)	37.6	39.4	32.6	36.5	21.3	22.8	23.6	22.6
Image + Style (no dialogue)	54.0	41.1	35.2	43.4	23.7	23.2	23.8	23.5
Style + Dialogue + Image (full model)	54.0	51.9	44.8	50.3	23.7	24.2	24.9	24.3
Table 3: Ablations on IMAGE-CHAT. We compare variants of our best TRANsRESNET generative and retrieval models (ResNeXt-IG-3.5B image encoder, and MM-Sum + separate text encoders for retrieval) where we remove modalities: image, dialogue history and style conditioning, reporting R@1/100 for retrieval and ROUGE-L for generation for dialogue turns 1, 2 and 3 independently, as well as the average over all turns.
conduct evaluations at each round of dialogue for each example in the evaluation set; we have a separate set of human evaluators look at the provided conversation turns, and ask them to compare two possible utterances for the next turn of conversation, given the image, dialogue history and relevant style (which is the same for both human author and model, so there is no advantage). We ask the evaluators in a blind test to choose the "more engaging" of the two possible utterances: one from a human, and the other from a model.
Human annotation vs. TRANSRESNET model We compare human-authored utterances to those produced by our models. The human conversations are collected in the same fashion as in IMAGE-CHAT but on test images. As for humans, the model outputs are conditioned on the image, style and previous dialogue history.  $\mathrm{TRANSRESNET}_{GEN}$  simply generates a response, whereas  $\mathrm{TRANSRESNET}_{RET}$  retrieves candidate utterances from the IMAGE-CHAT training set. The latter is given a separate set of candidates corresponding to the round of dialogue – e.g. when producing a response to turn 1, the model retrieves from all possible round 1 utterances from the train set (in that case 186,858 possible choices).
The results are shown in Fig. 4, comparing all models on the first round (left):  $\mathrm{TRANSRESNET}_{GEN}$  and  $\mathrm{TRANSRESNET}_{RET}$  us