SlowGuess/ACL_2020_OCR huggingface dataset:
Shubham Agarwal $^{1*}$ , Trung Bui $^{2}$ , Joon-Young Lee $^{2}$ , Ioannis Konstas $^{1}$  and Verena Rieser $^{1}$
<sup>1</sup>The Interaction Lab, Heriot-Watt University, Edinburgh, UK  
<sup>2</sup>Adobe Research, San Jose, CA, US  
{sa201, i.konstas, v.t.rieser}@hw.ac.uk  
{bui, jolee}@adobe.com
Visual Dialog involves "understanding" the dialog history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct response. In this paper, we show that co-attention models which explicitly encode dialog history outperform models that don't, achieving state-of-the-art performance (72% NDCG on val set). However, we also expose shortcomings of the crowd-sourcing dataset collection procedure by showing that history is indeed only required for a small amount of the data and that the current evaluation metric encourages generic replies. To that end, we propose a challenging subset (VisDialConv) of the VisDial val set and provide a benchmark of 63% NDCG.
Recently, there has been an increased interest in visual dialog, i.e. dialog-based interaction grounded in visual information (Chattopadhyay et al., 2017; De Vries et al., 2017; Seo et al., 2017; Guo et al., 2018; Shekhar et al., 2018; Kottur et al., 2019; Haber et al., 2019). One of the most popular test beds is the Visual Dialog Challenge (VisDial) (Das et al., 2017), which involves an agent answering questions related to an image, by selecting the answer from a list of possible candidate options. According to the authors, nearly all interactions  $(98\%)$  contain dialog phenomena, such as co-reference, that can only be resolved using dialog history, which makes this a distinct task from previous Visual Question Answering (VQA) challenges, e.g. (Antol et al., 2015). For example, in order to answer the question "About how many?" in Figure 1, we have to infer from what was previously said, that the conversation is about the skiers.
Figure 1: Visual Dialog task according to (Das et al., 2017) as a ranking problem, where for the current question (blue), the agent ranks list of 100 candidate answers (yellow). Relevance weights for each candidate were collected via crowd-sourcing. Previous dialog history (red) together with the caption (green) forms the contextual information for the current turn.
In the original paper, Das et al. (2017) find that models which structurally encode dialog history, such as Memory Networks (Bordes et al., 2016) or Hierarchical Recurrent Encoders (Serban et al., 2017) improve performance. However, "naive" history modelling (in this case an encoder with late fusion/concatenation of current question, image and history encodings) might actually hurt performance. Massiceti et al. (2018) take this even further, claiming that VisDial can be modeled without taking history or even visual information into account. Das et al. (2019) rebutted by showing that both features are still needed to achieve state-of-the
*This work was carried out during the internship at Adobe Research.
8182
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8182-8197 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
art (SOTA) results and an appropriate evaluation procedure has to be used.
In this paper, we show that competitive results on VisDial can indeed be achieved by replicating the top performing model for VQA (Yu et al., 2019b) – and effectively treating visual dialog as multiple rounds of question-answering, without taking history into account. However, we also show that these results can be significantly improved by encoding dialog history, as well as by fine-tuning on a more meaningful retrieval metric. Finally, we show that more sophisticated dialog encodings outperform naive fusion on a subset of the data which contains “true” dialog phenomena according to crowd-workers. In contrast to previous work on the VisDial dataset, e.g. (Kottur et al., 2018; Agarwal and Goyal, 2018; Gan et al., 2019; Guo et al., 2019; Kang et al., 2019), we are the first to conduct a principled study of dialog history encodings. Our contributions can thus be summarized as follows:
- We present SOTA results on the VisDial dataset using transformer-based Modular CoAttention (MCA) networks. We further show that models encoding dialog history outperform VQA models on this dataset.
- We show that curriculum fine-tuning (Bengio et al., 2009) on annotations of semantically equivalent answers further improves results.
- We experiment with different dialog history encodings and show that early fusion, i.e. dense interaction with visual information (either via grounding or guided attention) works better for cases where conversational historical context is required.
- We release a crowd-sourced subset containing verified dialog phenomena and provide benchmark results for future research.
2 Visual Dialog Models
In this section, we extend Modular Co-Attention Networks, which won the VQA challenge 2019 (Yu et al., 2019b) and adapt it to visual dialog. Different from previous co-attention networks (Kim et al., 2018; Nguyen and Okatani, 2018), MCA networks use guided attention to model dense relations between the question and image regions for better visual grounding. In the following, we explore MCA networks with different input encodings following a '[model]-[input]' convention to refer to our MCA model variants; see Figure 3 for an overview. Whenever unspecified, images
are represented as a bag of bottom-up features, i.e. object level representations (see Section 3).
2.1 Modular Co-Attention networks
The MCA module with multi-modal fusion as depicted in Figure 2, is common to all our architectures. Inspired by the transformers (Vaswani et al., 2017), the MCA network (Yu et al., 2019b) is a modular composition of two basic attention units: self-attention and guided attention. These are arranged in an encoder-decoder composition in the MCA module (Figure 2), which performed best for VQA (Yu et al., 2019b).
2.1.1 Self-Attention and Guided-Attention
The Self-Attention (SA) unit in transformers (Vaswani et al., 2017) is composed of a multi-head attention layer followed by a feed-forward layer. When applied to vision, the SA unit can be viewed as selecting the most relevant object-level image features for the downstream task. Specifically, the scaled dot product attention takes as input key, query and value (usually same modality's embedded representations) and outputs a self-attended vector (Eq.1). Multi-head attention provides multiple representation spaces to capture different linguistic/grounding phenomena, which are otherwise lost by averaging using a single head.
$$
\begin{array}{l} A t t (Q, K, V) = s o f t m a x (\frac {Q K ^ {T}}{\sqrt {d _ {K}}}) V \\ M H A t t (Q, K, V) = C o n c a t \left(\text {h e a d} _ {1}, \dots \text {h e a d} _ {n}\right) W ^ {O} \\ h e a d _ {i} = A t t \left(Q W _ {i} ^ {Q}, K W _ {k} ^ {K}, V W _ {i} ^ {V}\right) \tag {1} \\ \end{array}
$$
The Guided-Attention (GA) unit conditions the attention on different sequences. The key and value come from one modality, while the query comes from a different modality similar to the decoder architecture in Transformers (Vaswani et al., 2017). Similar to Eq. 1, the GA unit outputs features  $f_{i} = Att(X,Y,Y)$  where  $X\in \mathcal{R}^{m\times d_x}$  comes from one modality and  $Y\in \mathcal{R}^{n\times d_y}$  from the other. Residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) are applied to the output of both the attention and feed-forward layers similar to (Vaswani et al., 2017; Yu et al., 2019b) in both the SA and GA units.
2.1.2 Modular Co-Attention Module
The following description of the MCA module is based on the question and the image, but can be
8183
Figure 2: Modular Co-Attention (MCA) module with MCA-I (Section 2.1) as an example.
extended analogously to model the interaction between the question and history. First, the input (i.e. the question) is passed through multiple multi-head self-attention layers  $L$ , in order to get self-aware representations before acting as conditional signal to different modalities (visual or contextual history) similar to the auto-encoding procedure of Transformers. Then the final representation  $X^{L}$  is used as the input for GA units to model cross-modal dependencies and learn the final conditioned representation  $Y^{L}$ .
2.1.3 Multi-modal fusion
The learned representations  $X^L \in \mathcal{R}^{m \times d}$  and  $Y^L \in \mathcal{R}^{n \times d}$  contain the contextualized and conditioned representations over the word and image regions, respectively. We apply attention reduction (Yu et al., 2019b) with a multi-layer perceptron (MLP) for  $X^L$  (analogously for  $Y^L$ ). We obtain the final multi-modal fused representation  $z$ :
$$
\begin{array}{l} \alpha^ {x} = \operatorname {s o f t m a x} \left(M L P ^ {x} \left(X ^ {L}\right)\right) \\ \tilde {x} = \sum_ {m} ^ {i = 1} \alpha_ {i} ^ {x} x _ {i} ^ {L} \tag {2} \\ z = L a y e r N o r m (W _ {x} ^ {T} \tilde {x} + W _ {y} ^ {T} \tilde {y}) \\ \end{array}
$$
where  $\alpha^x = [\alpha_1^x \dots \alpha_m^x] \in \mathcal{R}^m$  are learned attention weights (same process for  $\alpha^y$  and  $\tilde{y}$ ) and  $W_x \in \mathcal{R}^{d \times d_z}$ ,  $W_y \in \mathcal{R}^{d \times d_z}$  are linear projection matrices (dimensions are the same for simplicity).
We call this model MCA with Image component only; (MCA-I), since it only encodes the question and image features and therefore treats each question in Visual Dialog as an independent
instance of VQA, without conditioning on the historical context of the interaction.
2.2 Variants with Dialog History
In the following, we extend the above framework to model dialog history. We experiment with late/shallow fusion of history and image (MCA-I-H), as well as modelling dense interaction between conversational history and the image representation (i.e. MCA-I-VGH, MCA-I-HGuidedQ).
History guided Question (MCA-I-HGuidedQ): The network in Figure 3a is designed to model coreference resolution, which can be considered as the primary task in VisDial (Kottur et al., 2018). We first enrich the question embedding by conditioning on historical context using guided attention in the MCA module. We then use this enriched (coreference resolved) question to model the visual interaction as described in Section 2.1.
Visually grounded history with image representation (MCA-I-VGH): Instead of considering conversational history and the visual context as two different modalities, we now ground the history with the image first, see Figure 3b. This is similar in spirit to maintaining a pool of visual attention maps (Seo et al., 2017), where we argue that different questions in the conversation attend to different parts of the image. Specifically, we pass the history to attend to object-level image features using the MCA module to get visually grounded contextual history. We then embed the question to pool the relevant grounded history using another MCA module.
8184
Figure 3: All models incorporating dialog history described in Section 2.2
In parallel, the question embedding is also used to ground the current visual context. At the final step, the respective current image and historical components are fused together and passed through a linear layer before decoding. Note, this model is generic enough to potentially handle multiple images in a conversation and thus could be extended for tasks e.g. conversational image editing, which is one of the target applications of visual dialog (Kim et al., 2017; Manuvinakurike et al., 2018a,b; Lin et al., 2018; El-Nouby et al., 2018).
Two-stream Image and History component (MCA-I-H): Figure 3c shows the model which maintains two streams of modular co-attention networks - one for the visual modality and the other for conversational history. We follow a similar architecture for the visual component as MCA-I and duplicate the structure for handling conversational history. At the final step, we concatenate both the embeddings and pass them through a linear layer.
2.3 Decoder and loss function
For all the models described above, we use a discriminative decoder which computes the similarity between the fused encoding and RNN-encoded answer representations which is passed through a softmax layer to get the probability distribution
over the candidate answers. We train using cross entropy over the ground truth answer:
$$
L (\theta) = \frac {1}{N} \sum_ {n = 1} ^ {N = 1 0 0} y _ {n} \log P \left(x _ {n}, \theta\right) \tag {3}
$$
$N$  denotes the number of candidate answers which is set to 100 for this task,  $y_{n}$  is the (ground truth) label which is 0 or 1 during the training procedure, or a relevance score of the options during fine-tuning (casting it as multi-label classification).
We use PyTorch $^{1}$  (Paszke et al., 2017) for our experiments $^{2}$ . Following Anderson et al. (2018), we use bottom-up features of 36 proposals from images using a Faster-RCNN (Ren et al., 2015) pre-trained on Visual Genome (Krishna et al., 2017) to get a bag of object-level 2048-d image representations. Input question and candidate options are tokenized to a maximum length of 20 while the conversational history to 200. Token embeddings in text are initialized with 300-d GloVe vectors (Pennington et al., 2014) and shared among all text-based encoders. The RNN encodings are implemented using LSTMs (Hochreiter and Schmidhuber, 1997).
1https://pytorch.org/
$^{2}$ Code available at https://github.com/shubhamagarwal92/visdial_conv
8185
We use the Adam optimizer (Kingma and Ba, 2015) both for training and fine-tuning. More training details can be found in Appendix A.
4 Task Description
We use VisDial v1.0 for our experiments and evaluation. $^3$  The dataset contains 123K/2K/8K dialogs for train/val/test set respectively. Each dialog is crowd-sourced on a different image, consisting of 10 rounds of dialog turns, totalling approx. 1.3M turns. Each question has also been paired with a list of 100 automatically generated candidate answers which the model has to rank. To account for the fact that there can be more than one semantically correct answer (e.g. "Nope", "No", "None", "Cannot be seen"), "dense annotations" for  $2\mathrm{k} / 2\mathrm{k}$  turns of train/val of the data have been provided, i.e. a crowd-sourced relevance score between 0 and 1 (1 being totally relevant) for all 100 options.
4.2 Evaluation protocol
As the Visual Dialog task has been posed as a ranking problem, standard information retrieval (IR) metrics are used for evaluation, such as Recall@{1,5,10} to measure performance in the top N results (higher better), mean reciprocal rank (MRR) of the Ground-Truth (GT) answer (higher better), and Mean rank of the GT answer (lower better). Normalized Discounted Cumulative Gain (NDCG) is another measure of ranking quality, which is commonly used when there is more than one correct answer (provided with their relevance).
4.3 Training details
Sparse Annotation Phase: We first train on sparse annotations, i.e. only 1 provided ground-truth answer, which is available for the whole training set. Here the model learns to select only one relevant answer.
Curriculum Fine-tuning Phase: Dense annotations, i.e. crowd-sourced relevance weights, are provided for  $0.16\%$  of training set, which we use to fine-tune the model to select multiple semantically equivalent answers. This acts like a curriculum learning setup (Elman, 1993; Bengio et al., 2009),
where selecting one answer using sparse annotation is an easier task and fine-tuning more difficult. $^4$
MCA-I-HConcQ and MCA-H: MCA-I-HConcQ is a naive approach of concatenating raw dialog history to the question while keeping the rest of the architecture the same as MCA-I. MCA-H on the other hand considers this task as only conversational (not visual) dialog with MCA module on history instead of image.
RvA: We reproduce the results of Niu et al. (2019)'s Recursive Visual Attention model (RvA), which won the 2019 VisDial challenge. Their model browses the dialog history and updates the visual attention recursively until the model has sufficient confidence to perform visual co-reference resolution. We use their single model's open-source implementation and apply our fine-tuning procedure on the val set in Table 1. When reporting on the test set results in Table 2, we use the leaderboard scores published online which contains further unpublished enhancements based on ensembling (MReaL-BDAI).
In the following, we report results on the VisDial v1.0 val set, (Table 1), as well as the test-standard set, (Table 2). For measuring significance (reported on  $p \leq 0.05$ ), we use Kruskal-Wallis (Kruskal and Wallis, 1952) and Wilcoxon signed rank test (Wilcoxon, 1992) with Bonferroni correction (Bonferroni, 1936). We report results in terms of NDCG, which is the main metric of the challenge.
MCA-I-H is our best performing model. It achieves state-of-the-art performance: It outperforms the RvA baseline by almost 5 NDCG points on the val set and by over 7 points on the test set. On the official challenge test set, MCA-I-H ranks  $2^{\text{nd}}$ : it improves over 7 NDCG over the best single model but loses by 2 points against a 6-strong RvA ensemble model (2019 winning entry).
4While 'instance-level' curriculum learning is defined in terms of 'harder dialogs', in our work, we used 'dataset/task-level' curriculum finetuning. Our suggested method is a combination of curriculum learning and fine tuning (pre-training and adjusting to a specific downstream task). As such, we use the term 'curriculum fine-tuning' i.e. adaptation by NDCG aware curriculum during fine-tuning.
5We only report results for our best preforming models as the number of allowed submissions to the challenge is limited.
3Following the guidelines on the dataset page we report results only on v1.0, instead of v0.9. VisDial v1.0 has been consistently used for Visual Dialog Challenge 2018 and 2019.
8186
Model	Sparse annotation Phase	Curriculum Fine-tuning
NDCG ↑	MRR ↑	R@1 ↑	R@5 ↑	R@10 ↑	Mean ↓	NDCG ↑	MRR ↑	R@1 ↑	R@5 ↑	R@10 ↑	Mean ↓
RvA (Challenge winners; single model)	55.86	64.42	50.71	81.50	90.15	4.06	67.90	51.92	36.57	70.69	83.61	5.85
MCA-H	51.67	59.65	45.21	77.01	86.79	4.92	64.06	38.16	22.86	54.99	71.24	9.19
MCA-I	59.94	59.67	45.95	76.15	86.24	5.24	70.82	37.34	21.22	56.13	72.74	9.23
MCA-I-HConcQ	60.65	64.08	50.83	80.74	89.62	4.22	70.81	40.75	24.53	60	75.11	8.13
MCA-I-HGuidedQ	60.17	64.36	50.99	80.95	89.93	4.17	71.32	44.1	28.44	61.74	76.53	7.83
MCA-I-VGH	62.44	61.25	47.5	78.16	87.8	4.74	72.0	40.22	24.38	58.8	73.77	8.44
MCA-I-H	60.27	64.33	51.12	80.91	89.65	4.24	72.22	42.38	26.94	60.17	75.2	8.2
MCA-I-H-GT	60.27	64.33	51.12	80.91	89.65	4.24	72.18	46.92	32.09	63.85	78.06	7.37
Table 1: Results on VisDial v1.0 val set. Here 'I' denotes image modality while 'H' refers to the use of dialog history. Our baseline models are defined in Section 2.1 and 4.4. MCA variants with dialog history follow the same order as Section 2.2. MCA-I-H-GT refers to the model with corrected dense annotations (see Section 6.2)
Model	NDCG ↑	MRR ↑	R@1 ↑	R@5 ↑	R@10 ↑	Mean ↓
RvA	55.59	63.03	49.03	80.40	89.83	4.18
MS-D365-AI (Ensemble-2nd)	64.78	54.23	42.88	65.38	76.12	6.50
MReal-BDAI (Ensemble-1st)	74.57	53.37	40.96	66.45	79.70	6.60
MCA-I	70.97	35.65	19.32	54.57	71.39	9.51
MCA-I-VGH	71.33	38.92	22.35	58.42	74.5	8.69
MCA-I-H	72.47	37.68	20.67	56.67	72.12	8.89
Table 2: Evaluation on test-standard set with results from the online leaderboard. Winners are picked on NDCG. MReaL-BDAI (2019 winning entry) is an ensemble of 6 RvA models. Runner-up MS-D365AI (unpublished) also used ensembling. Note all our submitted MCA models use curriculum fine-tuning and no ensembling.
Compared to MCA-I, which treats the task as multiple rounds of VQA, encoding history improves results, but only significantly for MCA-I-VGH in the sparse annotation phase. After fine-tuning, MCA-I-VGH and MCA-I-H perform equally. MCA-I-H implements a late/shallow fusion of history and image. Architectures which model dense interaction between the conversational history and the image representations (i.e. MCA-I-VGH, MCA-I-HGuidedQ) perform comparably; only MCA-HConcQ performs significantly worse. Note that MCA-I also outperforms the baselines and current SOTA by a substantial margin (both in the sparse annotation phase and curriculum finetuning phase), while, counter-intuitively, there is not a significant boost by adding conversational history. This is surprising, considering that according to Das et al. (2017),  $38\%$  of questions contain a pronoun, which would suggest that these questions would require dialog history in order to be "understood/grounded" by the model.
Furthermore, curriculum fine-tuning significantly improves performance with an average improvement of 11.7 NDCG points, but worsens performance in terms of the other metrics, which only consider a single ground truth (GT) answer.
In the following, we perform a detailed error analysis, investigating the benefits of dialog history en
coding and the observed discrepancy between the NDCG results and the other retrieval based metrics.
6.1 Dialog History
We performed an ablation study whereby we did not include the caption as part of historical context and compare with the results in Table 1. The performance dropped from (NDCG 72.2, MRR 42.3) to (NDCG 71.6, MRR 40.7) using our best performing MCA-I-H model after finetuning. Since the crowd-sourced conversation was based on the caption, the reduced performance was expected.
In order to further verify the role of dialog history, we conduct a crowd-sourcing study to understand which questions require dialog history, in order to be understood by humans. We first test our history-encoding models on a subset (76 dialogs) of the recently released VisPro dataset (Yu et al., 2019a) which focuses on the task of Visual Pronoun Resolution. Note that VisPro also contains non-referential pleonastic pronouns, i.e. pronouns used as "dummy subjects" when e.g. talking about the weather ("Is it sunny?").
We thus create a new crowd-sourced dataset<sup>7</sup>, which we call VisDialConv. This is a subset of the VisDial val-set consisting of 97 dialogs, where the crowd-workers identified single turns (with dense annotations) requiring historical information. In particular, we asked crowd-workers whether they could provide an answer to a question given an image, without showing them the dialog history, and select one of the categories in Table 4 (see further details in Appendix B).
In order to get reliable results, we recruited 3 crowd-workers per image-question pair and only kept instances where at least 2 people agreed. Note that we only had to discharge  $14.5\%$  of the origi
<sup>6</sup>We use the intersection of dialogs in VisDial val set and VisPro to create this subset.
7Data collection code available at https://github. com/shubhamagarwal92/visdialconv-amt
8187
Model	Sparse annotation Phase	Curriculum Fine-tuning
NDCG ↑	MRR ↑	R@1 ↑	R@5 ↑	R@10 ↑	Mean ↓	NDCG ↑	MRR ↑	R@1 ↑	R@5 ↑	R@10 ↑	Mean ↓
VisPro subset dataset
MCA-I	59.80	57.88	45.39	72.24	82.76	5.84	69.82	36.2	20	54.08	70.92	10.02
MCA-I-HConcQ	61.08	61.79	48.95	77.5	86.58	4.72	68.44	38	22.24	55.79	71.71	9.17
MCA-I-HGuidedQ	61.35	60.13	47.11	75.26	86.18	5.23	68.29	36.59	21.05	53.29	70.13	9.76
MCA-I-VGH	61.68	59.33	46.18	75.53	86.71	5.07	68.97	39.21	23.68	57.11	70.53	8.83
MCA-I-H	61.72	59.62	45.92	77.11	86.45	4.85	70.87	39.8	25.39	55.13	70.39	9.42
VisDialConv (Crowd-sourced subset) dataset
MCA-I	52.07	55.55	41.65	72.47	83.81	5.92	58.65	36.2	20.52	53.3	68.25	10.32
MCA-I-HConcQ	54.84	62.06	47.42	80.1	88.87	4.37	61.42	37.92	21.86	55.67	73.3	9.01
MCA-I-HGuidedQ	53.81	62.29	48.35	80.1	88.76	4.42	62.92	38.07	22.58	54.74	70.82	9.5
MCA-I-VGH	55.48	58.45	44.54	74.95	86.19	5.18	60.63	38.1	22.89	53.71	70.31	9.49
MCA-I-H	53.01	61.24	47.63	79.07	87.94	4.77	59.89	39.73	25.15	56.49	71.86	9.53