Venue: NAACL 2021

# Ensemble of MRR and NDCG models for Visual Dialog

Idan Schwartz

Technion

NetApp

idansc@cs.technion.ac.il

###### Abstract

Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (_e.g._, 'yeah' and 'yes'). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as 'I don't know.' Crafting a model that excels on both MRR and NDCG metrics is challenging [23]. Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% _vs._ 71.24%) and the NDCG state-of-the-art performance (72.16% _vs._ 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at [https://github.com/idansc/mrr-ndcg](https://github.com/idansc/mrr-ndcg).

## 1 Introduction

Das et al. DBLAN introduced the task of Visual Dialog, which requires an agent to converse about visual input. Evaluating visually aware conversation should examine both linguistic properties and visual reasoning. Analysis of generative metrics for dialog often shows no correlation with human judgments [9]. Hence, to evaluate the correctness of the candidate answers, a retrieval approach is preferred. Two metrics are standard, MRR and NDCG. The MRR metric focuses on a single human-derived ground-truth answer. Despite preferring the more human-like answer, the metric ignores many correct candidate answers. Differently, the NDCG considers the rank of all the correct answers. The metric relies on dense annotation, where three annotators were asked to mark all the correct candidate answers. However, the candidate answers are generated plausible answers. The analysis shows that the NDCG metric favors uncertain, generally correct answers, such as "not sure" [28, 10].

Prior work in visual dialog focused on a single metric. Ideally, an AI agent should answer human-like and detailed reply (the MRR metric) and be able to validate the correctness of any answer (the NDCG metric). However, crafting a model that excels in both metrics is challenging [23]. To this end, we propose principals to ensemble the rankings of strong MRR and NDCG models. Our approach is to find a minimal set that is likely

Figure 1: A visual dialog interaction. The question asks, “what is the nightstand made of ”. We show our final ranking, created by the ensemble of an MRR/NDCG models’ rankings. The MRR/NDCG models are trained to optimize the MRR/NDCG metric. The MRR metric measures the number of retrievals to retrieve the human-derived answer. Hence, the MRR model favors human-like and detailed answers. On the other hand, the NDCG metric measures the rank of all the correct candidates based on dense annotation, which are often general and uncertain. Our ensemble approach seeks a minimal candidate set that is likely to contain the human-derived answer. The remaining candidates are ranked according to the NDCG model.



to hold the human-derived answer. This permits ranking the rest of the candidates according to the NDCG model. Our approach won the recent Visual Dialog 2020 challenge and achieved strong performance on both the MRR and the NDCG metrics simultaneously.

## 2 Related Work

**Visual conversation evaluation:** Early attempts to marry conversation with vision used street scene images, and binary questions (Geman et al., 2015). While binary answers are easy to verify, such an approach is limiting for an AI agent. On the other hand, analysis of generative metrics for dialog often show no correlation with human judgements Liu et al. (2016). Intuitively, metrics like BLEU-scores rely on corresponding words with the ground-truth answer and often miss synonyms or the subjective nature. More importantly, generative metrics are geared toward textual assessment rather than visual reasoning, which results in models mainly relying on textual cues Schwartz et al. (2019). Malinowski and Fritz (2014) suggest Wu-Palmer similarity metric that calculates similarity based on the depth of two words based on the WordNet taxonomy Miller (1995). A different approach suggested in the VQA dataset focus only on brief, mostly 1-word answers Antol et al. (2015). In this setup, the task turns into popular answers classification, alleviating many text-generation challenges. Notably, VQA requires 3 out of 10 annotators to agree on the answer, which is robust to inter-person variation. Still, accuracy ignores the reasoning process Hudson and Manning (2019) propose GQA, which extends the accuracy metric and uses a scene graph for both question generation and evaluation. Following, Das et al. (2017) propose the VisDial dataset for the visual dialog task, which formulates multiple image-language interactions via a dialog. Concurrently, de Vries et al. (2017) propose GuessWhat, a goal-driven dialog dataset for object identification. Different from VQA and goal-driven dialogs, the VisDial answers are detailed and more human-like. For instance, in Fig. 1, the answer is "Can't tell...cloth", while a VQA answer would be "cloth". Therefore, metrics that require exact matching are no longer suitable. Instead, each question is accompanied with 100 candidate answers. Consequently, the metric has been shifted from accuracy to retrieval-based metrics, _e.g._, MRR and NDCG. Prior works focus on optimizing a single metric Guo et al. (2019); Jiang et al. (2020); Hu et al. (2017); Gan et al. (2019). Differently, Murahari et al. (2020) attempt to optimize both metrics with a joint loss. Still, a dedicated single metric model is superior. Instead, we propose principals to ensemble two dedicated models, one for NDCG and one for MRR. Our approach allows most of the MRR and NDCG to be preserved simultaneously.

**Visual dialog models:** Various approaches were proposed to solve the Visual Dialog task. Most of them focus on dialog history reasoning per interaction. Serban et al. (2017) propose history hierarchical encoding. Seo et al. (2017) introduce a memory network based on attention, which also addressed co-referential issues. Kottur et al. (2018) focus on visual co-reference. Jain et al. (2019) concatenate representations of all the cues (_e.g._, image, question, history, and caption) per candidate answer. Zheng et al. (2019) employ a graph structure learning. Schwartz et al. (2019) propose a model, namely Factor Graph Attention (FGA), that lets all entities (_e.g._, question-words, image-regions, answer-candidate, and caption-words) interact to infer an attention map for each modality. An ensemble of five FGA models achieves the state-of-the-art MRR performance. However, FGA optimizes using the sparse annotations, _i.e._, the human-derived answer. Murahari et al. (2020) recently propose Large-Scale(LS) model, which pre-trains on related vision-language datasets, _e.g._, Conceptual Captions and Visual Question AnsweringSharma et al. (2018); Antol et al. (2015). Concurrently, Wang et al. (2020) leverage the pretrained BERT language models, and Nguyen et al. (2020) propose a lightweight Transformer that handles the interplay between many modalities. The three methods mentioned above finetune using the dense annotation (_i.e._, human assessment of all the candidates), resulting in a substantial improvement on the NDCG metric. Importantly, Murahari _et al._ find that finetuning a model for NDCG hurts MRR performance. This work demonstrates that re-ranking MRR model (_e.g._, FGA) and NDCG model (_e.g._, LS) with simple principles keeps most MRR and NDCG performance.

## 3 Two-step Rank Ensemble

The MRR metric depends on a single human-derived answer. Hence, given that this answer is ranked highly, the remaining candidates can be ranked according to the NDCG model. In the following, we describe two steps: (i) the MRR step responsible for preserving the human-derived rank 

high, and (ii) the NDCG step responsible for ranking the remaining candidates based on the NDCG model.

### Setup

We are given a set of dialog questions \(\{(q,\mathcal{C}_{q})_{i}\}_{i=1}^{d}\), where \(d\) is the dataset size, \(q\) is a dialog question, and \(\mathcal{C}_{q}=\{c_{q,j}\}_{j=1}^{100}\) are the corresponding candidates. The MRR metric, _i.e_., the inverse harmonic mean of rank, is defined as:

\[\mathrm{MRR}=\frac{1}{d}\sum_{i=1}^{d}\frac{1}{r_{\rm i}}, \tag{1}\]

where \(r_{i}\) is the rank of the human response for the \(i\)-th dialog question. The DCG, _i.e_., discounted cumulative gain over the \(K\) correct answers, is defined as:

\[\mathrm{DCG}_{\rm K}=\sum_{i=1}^{K}\frac{s_{i}}{\log_{2}(i+1)}, \tag{2}\]

where \(s_{i}\) is a binary score, representing the fraction of annotators that marked the candidate at position as correct. We normalize by the ideal \(\mathrm{DCG}_{\rm K}\) score (\(\mathrm{IDCG}_{\rm K}\)), _i.e_., \(\mathrm{NDCG}_{\rm K}=\frac{\mathrm{DCCG}_{\rm K}}{\mathrm{IDCG}_{\rm K}}\). We denote the set of MRR models as \(\mathcal{M}=\{M_{1},\ldots,M_{n_{m}}\}\) where \(n_{m}\) is the number of MRR models. Each MRR model is built by altering the initial conditions. We denote the NDCG model as \(N\). We define an operator \(\mathrm{T}(M,n,q)\) that returns the model \(M\)'s top \(n\) responses given a question \(q\). Next, we describe the MRR step that aims to keep the MRR score.

### MRR step

The purpose of the MRR step is to find a minimal candidate set \(\mathcal{C}_{\text{MRR},q}\) that is likely to contain the human-derived answer given a question \(q\). We build this set as a union of three sets, as follows:

\[\mathcal{C}_{\text{MRR},q}=\mathcal{T}_{q}\cup\mathcal{N}_{q}\cup\mathcal{H} _{q}, \tag{3}\]

where \(\mathcal{T}_{q}\) is a set of first ranked candidates according to MRR models, \(\mathcal{N}_{q}\) is a set of high ranked candidates by both MRR and NDCG models, \(\mathcal{H}_{q}\) is a set of high-certainty candidates agreed by all the MRR models. All sets are conditioned by the question \(q\). In the following, we formally define those sets.

#### 3.2.1 High-certainty answers

One of the most significant signals to be the human-derived answer is being a top MRR-model's answer. However, in many subjective questions, the MRR model is not certain. We found that in those cases, the top answers often varies between different MRR models. Thus, to verify the top candidate's certainty, we require an agreement of MRR-models. Let \(q\) be a dialog question, we define the _high-certainty_ set as follows:

\[\mathcal{H}_{q}=\{c|\left(\forall M\in\mathcal{M};c\in\mathrm{T}(M,\rho_{h}, q)\right)\}, \tag{4}\]

where \(\rho_{h}\in\mathbb{R}\) is an hyperparameter. Intuitively, a low \(\rho_{h}\) results in higher certainty. We Next, we add the MRR-models' answer at first retrieval.

#### 3.2.2 Top answers

The MRR metric prioritizes the first-ranked answer (see Eq.(1)). This property suits the nature of dialog models that reply with a single response. Consequently, we keep the first responses of the MRR models. Let \(q\) be a dialog question, the _top-answers_ set is defined as:

\[\mathcal{T}_{q}=\{c|\left(\exists M\in\mathcal{M};c\in\mathrm{T}(M,\rho_{t},q )\right)\}, \tag{5}\]

where \(\rho_{t}\in\mathbb{R}\) is an hyperparameter. We note that \(\rho_{t}\) should be low to maintain candidates' certainty. In the next step, we consider top NDCG candidates.

#### 3.2.3 NDCG-agreement answers

When the NDCG model and the MRR model agree that a candidate is likely to be correct, it implies that both the NDCG and MRR metrics gain by ranking this candidate high. Thus, we want to rank it high. We note that the MRR set is ranked first, so we include these candidates in the MRR set. Let \(q\) be a dialog question, the _ndcg-agreement_ set is defined as:

\[\mathcal{N}_{q}=\{c|\exists M\in\mathcal{M};c\in\mathrm{T}(N,\rho_{n}^{n},q )\cap\mathrm{T}(M,\rho_{m}^{n},q)\}, \tag{6}\]

where \(\rho_{n}^{n},\rho_{m}^{n}\in\mathbb{R}\) are hyperparameters that indicate relevancy to NDCG and MRR, respectively. _I.e_., as \(\rho_{nn}\) increases, we may include more relevant candidates according to the NDCG model.

Up until this stage we have built a minimal set \(C_{\text{MRR},q}\) that is likely to hold the human-derived answer. In the following we describe how we rank this set.

#### 3.2.4 MRR ranking

Let \(r_{M_{i},c,q}\) denote the rank according to \(M_{i}\in\mathcal{M}\) of candidate \(c\) for a question \(q\). We compute the MRR rank of candidate \(c\in\mathcal{C}_{\text{MRR},q}\) via geometric mean: \(r_{\text{MRR},c,q}=\prod_{i=1}^{n_{m}}r_{M_{i},c,q}\).

 



### NDCG step

In this step, we rank the remaining candidates \({\cal C}_{\text{NDCG},q}={\cal C}_{q}\setminus{\cal C}_{\text{MRR},q}\). We assume the correct MRR answer is in \({\cal C}_{\text{MRR}}\). Thus, we rank the remaining candidates, according to the NDCG model via geometric mean: \(r_{\text{NDCG},c,q}=(r_{N,c,q})^{p}\cdot r_{M,c,q}\), where \(M\in\mathcal{M}\) is the most accurate MRR model, and \(p\in\mathbb{R}\) is a calibration hyperparameter which controls the trade-off between MRR and NDCG.

To conclude, let \(q\) be a dialog question and \({\cal C}_{q}\) the corresponding candidates. We first find \({\cal C}_{\text{MRR},q}\), and rank the set according to \(r_{\text{MRR},c,q}\). We then rank the remaining candidates, according to \(r_{\text{NDCG},c,q}\).

## 4 Results

We show our results on the VisDial v1.0 dataset, where 123,287 images are used for training, 2,000 images for validation, and 8,000 images for testing Das et al. (2017). Each image is associated with ten questions, and each question has 100 corresponding answer candidates. We use two MRR models (_i.e_., \(n_{m}=2\)), FGA Schwartz et al. (2019b) and an ensemble of LS Murahari et al. (2020) with FGA. We use LS(CE) as the NDCG model. We set \(\rho_{h}=3\), \(\rho_{t}=1\), \(\rho_{n}^{n}=5\), \(\rho_{m}^{n}=10\), and \(p=3\). We tune these parameters using the validation set.

**Comparison to state-of-the-art**: In Tab. 1 we compare our method to naive ensembles and previous baselines. We first ensemble the LS's output with the FGA's output. By combining them, we achieve the new MRR state-of-the-art (71.24% _vs_. 69.37%). Next, we build a naive ensemble of the MRR model and the NDCG model. We do so by adding the MRR ensemble scores (denoted by \({\cal S}_{M}\)) and LS(CE) scores (denoted by \({\cal S}_{N}\)), as follows: \(\alpha\cdot{\cal S}_{M}+(1-\alpha){\cal S}_{N}\), where \(\alpha\in\mathbb{R}\) calibrates the trade-off between MRR and NDCG performance. We show in Fig. 2 an analysis of different \(\alpha\) values on the validation set. In Tab. 1, we report results for \(\alpha=0.8\). Our two-step method outperforms the MRR (70.41% _vs_. 68.78%) and NDCG (72.16% _vs_. 69.22%) metrics, despite lacking the output scores and only requiring rankings.

We also compare our approach to previous baselines. Most methods use the sparse annotations, _i.e_., the human-derived answer, while MReal-BDAI, VD-BERT, and LS(CE) finetune using the dense annotations. Finetuning with the dense annotations tremendously boosts the NDCG performance but loses MRR performance. The MRR performance decline can be attributed to NDCG being biased toward uncertain answers. We also note that LS leverages large-scale image-text corpora. LS(CE+NSP) optimizes both the dense and sparse annotations but still suffers from a performance drop compared to metric-dedicated LS models, _i.e_., MRR (63.92% _vs_. 67.50%) and NDCG (68.08% _vs_. 74.47%). Unlike the method mentioned above, our method re-rank the candidates based on t

\begin{table}
\begin{tabular}{l l c c c c c c c} Model & MRR & R/I & R/I & R/I & R/I & R/I & Mean; NDCG \\ \hline NNX (Gattar et al., 2019a) & 88.00 & 14.05 & 88.00 & 86.38 & 4.61 & 58.00 & 1.001 \\ SNN Zhang et al. (2019) & 63.07 & 42.73 & 7.398 & 79.85 & 4.57 & 52.82 \\ CNCPNN Clarkson et al. (2018) & 61.07 & 73.55 & 71.36 & 71.80 & 87.40 & 4.49 & 74.90 \\ RNA Quacki et al. (2019) & 63.03 & 60.93 & 80.40 & 89.53 & 4.18 & 55.59 \\ HACAN Yuge et al. (2019) & 64.22 & 60.88 & 80.80 & 89.45 & 4.20 & 57.17 \\ MIRL-BBIaiai et al. (2020) & 55.22 & 60.01 & 65.88 & 79.13 & 6.76 & 76.20 \\ RDAN Chen et al. (2019) & 55.13 & 41.38 & 68.67 & 74.50 & 5.91 & 61.86 \\ ReDNJ* Graau et al. (2019) & 53.14 & 42.45 & 66.48 & 75.60 & 6.64 & 64.47 \\ DauDD Jiang et al. (2020) & 63.23 & 42.95 & 80.23 & 87.90 & 41.1 & 56.32 \\ DL-Hot Chao et al. (2019) & 62.02 & 63.20 & 79.00 & 88.49 & 49.57 & 41.71 & 57.32 \\ DL-Hot Chao et al. (2019) & 64.22 & 90.04 & 80.77 & 90.68 & 59.77 & 57.88 \\ DNN Gao et al. (2019) & 63.20 & 63.20 & 79.35 & 87.95 & 35.34 & 30.57 \\ DAN Yang et al. (2019) & 64.29 & 51.28 & 81.60 & 90.88 & 39.52 & 93.66 \\ Laird Gkiox et al. (2020) & 64.20 & 50.86 & 89.63 & 49.55 & 49.58 & 99.83 \\ LDNI Nguyen et al. (2020) & 63.20 & 89.53 & 86.60 & 86.05 & 65.53 & 74.88 \\ FGAA Schuster et al. (2019) & 63.75 & 95.80 & 88.55 & 45.1 & 52.12 \\ SGUA Schwartz et al. (2019) & 63.97 & 55.65 & 86.73 & 94.05 & 31.44 & 57.29 \\ ICSFD Marabier et al. (2020) & 50.34 & 59.55 & 41.80 & 60.06 & 62.48 & 60.86 \\ LSCFSDNetkar et al. (2020) & 50.22 & 50.78 & 79.53 & 89.66 & 42.58 & 60.88 \\ LiS-Mahani et al. (2020) & 67.50 & 55.58 & 55.88 & 68.32 & 53.22 & 63.87 \\ VDB-Bertrandt* Wang et al. (2020) & 51.77 & 38.90 & 60.22 & 77.98 & 66.99 & **73.38** \\ SVRGAL K-Eyt* & **71.34** & **73.48** & **58.75** & **69.44** & **74.95** & **24.66** & 69.22 \\ SMCA I+LS LCECF+1 & 68.57 & 55.22 & 60.31 & 93.55 & 32.62 & 69.22 \\ Ours*11 & 50.44 & 51.81 & 43.85 & 40.85 & 90.33 & 3.66 & 72.16 \\  & Visual Dialog Challenge 300  & **70.16** & **70.15** & **70.15** & **70.16** & **70.17** & **70.17** \\ \hline LS & 68.75 & 55.50 & 86.12 & 70.88 & 33.22 & 63.43 \\ VB-BER & 51.84 & 59.41 & 68.76 & 65.67 & 57.22 \\ MRRL-LDI (Price) & 64.12 & 50.81 & 80.08 & 90.92 & 33.83 & 75.70 \\ SDS-INQM (2nd) & 63.84 & 56.22 & 70.83 & 53.70 & 54.84 & 75.86 \\ Ours (1st* Pfater) & 20.42 & 55.92 & 62.85 & 88.84 & 39.6

two distinct steps, to keep the human-derived answer high. In doing so, we achieve a good MRR performance (70.41% _vs._ 71.24%), yet notably with limited NDCG drop (72.25% _vs._ 75.35%). This property comes in handy in the recent Visual Dialog challenge, where the winners were picked based on both the NNDG and MRR evaluation metrics. Our method performs well on both metrics simultaneously and won the challenge.

**Ablation analysis**: The MRR candidate set consists of different subsets. In Tab. 2 we show the influence of each of subset independently on the retrieval metrics. Further, omitting a subset harms the performance, _i.e_. each component is essential to preserve both the MRR and NDCG metrics. We also report the average size of the MRR-candidate set, and the validation performance of the MRR model (_i.e_., 5xFGA) and the NDCG model (_i.e_., LS(CE)). In addition we provide the results of the MRR ensemble, and the naive NDCG and MRR ensemble for \(\alpha=0.8\).

In Fig. 3, we examine how the NDCG and MRR metrics are affected by modifying one hyperparameter while maintaining the others. On the first figure from the left, we alter \(\rho_{c}\). The higher \(\rho_{c}\), we require higher agreement between the MRR models, resulting in higher certainty for elements in the MRR set. Because the MRR models are responsible for the MRR set ranking, an MRR set that is too large hurts the NDCG metric. For the same reason, in the second image from the left, increasing \(\rho_{t}\), significantly harms the NDCG performance. In the third figure from the left, we show that considering more candidates that both NDCG and MRR models agree upon (_i.e_., increasing \(\rho^{n}_{n}\)) helps both metrics' performance. However, adding too many candidates harms the NDCG metric. In the fourth image from the left, we show that the performance remains stable when \(\rho^{n}_{m}\) is larger than three. Last, on the fifth image from the left, we show the effect of changing \(p\), which calibrates the trade-off between MRR and NDCG during the NDCG ranking step.

**Qualitative analysis**: In Fig. 4, we show two sample visual dialogs from test-std. For each sample, we provide the ranked MRR candidate set and the next 4 NDCG candidates. The analysis reveals the answers' ambiguity and that the MRR candidate set mostly consists of certain responses. In addition, we highlight the candidates within each MRR candidate subset with different colors. Additional samples can be found in the appendix.

## 5 Conclusions

We describe a non-parametric method to ensemble the candidate ranks of two strong MRR and NDCG models into a single ranking that excels on both NDCG and MRR. Intuitively, we use the MRR-model for non-ambiguous questions with certain answers. The dense-annotations cue is more applicable in ambiguous questions than the sparse annotations. Thus, in the case of low certainty, our method relies almost entirely on the NDCG model. We hope the proposed principles can guide the community towards a parametric model that can employ answers' semantics to measure certainty.

## Acknowledgements

We thank Yftah Ziser, Itai Gat, Alexander Schwing and Tamir Hazan for useful discussions.

Figure 4: An illustration of two visual dialog samples. Each sample includes the MRR candidate set and four answers from the remaining NDCG candidates. We find that the MRR candidate set has more certain answers. We colorize the _high-certainty_ candidates (\(\mathcal{H}\)) with orange, the _NDCG-agreement_ candidates (\(\mathcal{N}\)) with purple, and the _top-answers_ subset (\(\mathcal{T}\)) with red. Note, if a candidate belongs to more than one set, we sketch the colors in the following order: \(\texttt{orange}\!\rightarrow\!\texttt{red}\!\rightarrow\!\texttt{purple}\).

Figure 3: MRR and NDCG scores for different hyperparameter values.

 



## References

* Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In _ICCV_.
* Das et al. (2017) Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In _CVPR_.
* Gan et al. (2019) Zhe Gan, Yu Cheng, Ahmed EI Kholy, Linjie Li, Jingjing Liu, and Jianfeng Gao. 2019. Multi-step reasoning via recurrent dual attention for visual dialog. _ACL_.
* Geman et al. (2015) Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. 2015. Visual turing test for computer vision systems. _NAS_.
* Guo et al. (2019) Dalu Guo, Chang Xu, and Dacheng Tao. 2019. Image-question-answer synergistic network for visual dialog. In _CVPR_.
* Hu et al. (2017) Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. _ICCV_.
* Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In _CVPR_.
* Jain et al. (2019) Unnat Jain, Svetlana Lazebnik, and Alexander Schwing. 2019. Two can play this game: Visual dialog with discriminative question generation and answering. _CVPR_.
* Jiang et al. (2020) Xiaoze Jiang, Jing Yu, Zengchang Qin, Yingying Zhuang, Xingxing Zhang, Yue Hu, and Qi Wu. 2020. Dualvd: An adaptive dual encoding model for deep visual understanding in visual dialogue. _AAAI_.
* Kang et al. (2019) Gi-Cheon Kang, Jaeseo Lim, and Byoung-Tak Zhang. 2019. Dual attention networks for visual reference resolution in visual dialog. _ACL_.
* Kottur et al. (2018) Satwik Kottur, Jose MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018. Visual coreference resolution in visual dialog using neural module networks. In _ECCV_.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. _arXiv preprint arXiv:1603.08023_.
* Malinowski and Fritz (2014) Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In _NIPS_.
* Miller (1995) George A Miller. 1995. Wordnet: a lexical database for english. _Communications of the ACM_.
* Murahari et al. (2020) Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. 2020. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. _ECCV_.
* Nguyen et al. (2020) Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. 2020. Efficient attention mechanism for handling all the interactions between many inputs with application to visual dialog. _ECCV_.
* Niu et al. (2019) Yulei Niu, Hanwang Zhang, Manli Zhang, Jianhong Zhang, Zhiwu Lu, and Ji-Rong Wen. 2019. Recursive visual attention in visual dialog. In _CVPR_.
* Qi et al. (2020) Jiaxin Qi, Yulei Niu, Jianqiang Huang, and Hanwang Zhang. 2020. Two causal principles for improving visual dialog. _CVPR_.
* Schwartz et al. (2019a) Idan Schwartz, Alexander G Schwing, and Tamir Hazan. 2019a. A simple baseline for audio-visual scene-aware dialog. In _CVPR_.
* Schwartz et al. (2019b) Idan Schwartz, Seunghak Yu, Tamir Hazan, and Alexander G Schwing. 2019b. Factor graph attention. In _CVPR_.
* Seo et al. (2017) Paul Hongsuck Seo, Andreas Lehrmann, Bohyang Han, and Leonid Sigal. 2017. Visual reference resolution using attention memory for visual dialog. In _NIPS_.
* Serban et al. (2017) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In _AAAI_.
* Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In _ACL_.
* de Vries et al. (2017) Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron C Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In _CVPR_.
* Wang et al. (2020) Yue Wang, Shafiq Joty, Michael R Lyu, Irwin King, Caiming Xiong, and Steven CH Hoi. 2020. Vd-bert: A unified vision and dialog transformer with bert. _EMNLP_.
* Yang et al. (2019) Tianhao Yang, Zheng-Jun Zha, and Hanwang Zhang. 2019. Making history matter: History-advantage sequence training for visual dialog. In _ICCV_.
* Zheng et al. (2019) Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. 2019. Reasoning visual dialogs with structural and partial observations. In _CVPR_.

## Appendix A Qualitative Analysis

In the following, we show additional 200 randomly picked visual dialog samples from test-std. For each sample, we provide the ranked MRR candidate set and the next 10 NDCG candidates. We

colorize the _high-certainty_ candidates (\({\cal H}\)) with orange, the _NDCG-agreement_ candidates (\({\cal N}\)) with purple, and the _top-answers_ subset (\({\cal T}\)) with red. Note, if a candidate belongs to more than one set, we sketch the colors in the following order: orange\(\rightarrow\)red\(\rightarrow\)purple.

**Question:**what color is the light ?

**MRR candidate set**

1. red
2. green
3. the light is currently green
4. the lights are n't visible

**Top 10 from the remaining NDCG candidates**

1. yellow
2. not sure
3. i ca n't tell
4. it 's kinda dark out so i am unsure
5. i can not tell
6. ca n't tell
7. black and red
8. UNK
9. black
10. it 's dark

**Question:**what ethnicity do they appear to be ?

**MRR candidate set**

1. all white
2. white
3. i can not tell
4. ca n't tell
5. not sure

**Top 10 from the remaining NDCG candidates**

1. black
2. i ca n't tell
3. both have brown hair
4. there are 2 teens , and 1 adult
5. i think like 40 's maybe
6. maybe younger seniors
7. dark brown
8. no , the tv is not that visible
9. 1 looks older the other is quite young
10. she appears to be Caucasian 

1. no
2. not that i see
3. nope
4. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no other animals
2. 0 that i can see
3. 0
4. 0 visible
5. lol , no
6. non
7. i do n't think so
8. i can not see
9. no , only animals
10. i do not see any animals

**Question:**how old is the woman ?

**MRR candidate set**

1. she 's probably 60
2. middle aged
3. late UNK 's
4. i can not tell
5. unable to tell

**Top 10 from the remaining NDCG candidates**

1. ca n't tell
2. looks middle aged
3. mid sixties
4. i ca n't tell
5. late 40 's
6. about middle aged maybe 40 something
7. 70s
8. not sure
9. about 30
10. at least 30s

**Question:do they all have any spots on them ? MRR candidate set**

1. yes , all but 1 has spots
2. no
3. yes
4. not that i can see
5. it 's too hard to tell
6. nope

**Top 10 from the remaining NDCG candidates**

1. i do n't think so
2. i can not tell
3. very few
4. some do , yes
5. not really
6. some do
7. can not tell
8. i ca n't tell
9. ca n't tell 



**Question:**see any store names ?

**MRR candidate set**

1. no
2. yes
3. i can not tell
4. unable to tell
5. not that i can tell
6. not that i can see

**Top 10 from the remaining NDCG candidates**

1. nope
2. not really
3. i ca n't see it
4. not sure
5. i ca n't tell
6. ca n't tell
7. probably just 1
8. not at all
9. yes 1
10. id o n't think so

**Question:**what color is their hair ?

**MRR candidate set**

1. i can not tell
2. black
3. ca n't see them
4. ca n't tell
5. it doesn ' t have hair it 's a light
6. i ca n't tell

**Top 10 from the remaining NDCG candidates**

1. not sure
2. dark is all i can tell
3. brown i believe
4. hard to tell , but it looks very blackened
5. brown
6. it looks brown
7. no people
8. dark brown
9. it is dark brown
10. it 's brown

5. it looks like it 's daytime
6. it 's daytime

**Top 10 from the remaining NDCG candidates**

1. it is day time
2. day time
3. it is day
4. it is during the day
5. hard to tell probably day
6. looks like early morning
7. daytime maybe dusk
8. hello , fellow turkey : yes , it is day time
9. in the even or morning
10. i ca n't tell

**Question:is this photo in color ? MRR candidate set**

1. yes
2. yes it is
3. yes , it is
4. yes it is in color
5. yes , it 's in color

**Top 10 from the remaining NDCG candidates**

1. yes the picture is in color
2. yes in color
3. it is
4. it is in color
5. it 's in color
6. yes , it 's very colorful !
7. i think so
8. looks like it
9. it is , but there is not much color in the photo due to the objects
10. yes it is well lit 



**Question:are they all facing the camera ? MRR candidate set**

1. yes
2. no
3. slightly
4. yes they are
5. nope

**Top 10 from the remaining NDCG candidates**

1. not really
2. i don ' t think so
3. yes they all are
4. yes i believe so
5. no side view
6. i do n't think so
7. i think so
8. all but 1
9. not all of them
10. not that i can see
11. no
2. i do n't think so
3. nope

**Top 10 from the remaining NDCG candidates**

1. does not look like it
2. i ca n't tell
3. i can not tell
4. not sure
5. i can not say
6. ca n't tell
7. not that i can see
8. i do n't think so , but it 's hard to tell
9. i 'm not sure what that is
10. no i am unable to tell



**Question:**how old is the man ?

**MRR candidate set**

1. i can not tell
2. can not tell
3. unable to tell
4. ca n't see his face
5. i ca n't tell
6. ca n't tell

**Top 10 from the remaining NDCG candidates**

1. i ca n't tell i ca n't see their face
2. twenties maybe , hard to tell
3. hard to tell but i would say mid 20 's
4. hard to say but i guess around thirty
5. not sure
6. i can not see their face
7. maybe 25ish
8. he looks to be about 30ish
9. it 's hard to tell , it 's from a distance
10. he looks to be maybe 20 or so

**Question:**is there anything near it ?

**MRR candidate set**

1. no
2. grass and tree 's
3. nothing
4. not that i can see
5. hope
6. ca n't see anything

**Top 10 from the remaining NDCG candidates**

1. yes
2. i do n't think so
3. i don ' t see any
4. not really
5. you can see a pitcher and maybe a wall
6. a few things
7. no , it 's just dirt
8. in the distance
9. no but it has a lot of stuff on it
10. not sure there may be , but it does n't show enough to tell



**Question:is there anything on the wallpaper of the laptop ? MRR candidate set**

1. no
2. nope
3. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no there is not
2. i can not tell
3. i do n't see any
4. ca n't tell
5. i ca n't tell
6. 0
7. i do n't think so
8. just a screen over it
9. not that i can UNK for sure
10. not sure

**Question:is it date stamped ?MRR candidate set**

1. no
2. no it 's not
3. no it is n't
4. nope
5. no , it 's not

**Top 10 from the remaining NDCG candidates**

1. not that i can see
2. i do n't think so
3. i do n't know
4. do n't know
5. i ca n't tell
6. ca n't tell
7. not sure
8. i can not tell
9. not readable



**Question:is this inside ? MRR candidate set**

1. yes
2. yes it is
3. yes , it is
4. yes i think so

**Top 10 from the remaining NDCG candidates**

1. yes , it seems to be
2. i think so , yes
3. this is inside
4. it seems to be
5. it is indoors
6. i believe so
7. i think so
8. looks like it
9. no
10. uses Question:does the food look appetizing ? MRR candidate set**

1. yes
2. no
3. not really
4. not at all
5. nope

**Top 10 from the remaining NDCG candidates**

1. yes it does
2. no , he does n't
3. yes , it does
4. i can not tell
5. ca n't tell
6. a little
7. not that i can see
8. sort of
9. kinda
10. i ca n't tell



**Question:is the image in color ? MRR candidate set**

1. yes
2. yes it is
3. yes , it is in color
4. yes it is in color
5. yes , it 's in color
6. yes , the photo is in color

**Top 10 from the remaining NDCG candidates**

1. yes , it is
2. yes the photo is in color
3. yes , full color photo
4. yes it 's color
5. it is
6. yes in color
7. it is yes
8. yes
9. yea
10.yses

1. yes
2. yes it is
3. yes , it is daytime
4. yes it is daytime
5. yes it is daytime

**Top 10 from the remaining NDCG candidates**

1. it is
2. yes , it looks to be
3. it is daytime
4. yup
5. it is day time
6. yes , i think so
7. i believe so
8. yes it is , it looks sunny
9. daytime
10. i think so

1. yes
2. yes it is
3. yes , it is in color
4. yes it is in color
5. yes it 's in color

**Top 10 from the remaining NDCG candidates**

1. yes , it is
2. yes the picture is in color
3. yeah , this photo is in color
4. yes in color
5. it is
6. it is in color
7. yup
8. it is a beautiful color picture
9. ye
10. y

**Question:any buildings visible ? MRR candidate set**

1. yes a few
2. yes
3. yes , a few buildings
4. yes
5. i think so

**Top 10 from the remaining NDCG candidates**

1. yeah in the distance
2. lots
3. in the distance
4. yes there are some small ones
5. yes , 1 across the street
6. yes in the far background
7. yes , but i ca n't tell what kind
8. yes , it is
9. very far in the background
10. no 



**Question:**how old is the woman ?

**MRR candidate set**

1. she looks like in her 30 's
2. in her twenties i believe
3. not sure

**Top 10 from the remaining NDCG candidates**

1. 20 's
2. early 20 's
3. i ca n't tell
4. ca n't tell
5. 20 's maybe
6. looks to be late 20s
7. i can not tell
8. young adult
9. 25

**Question:**is the image in color ?

**MRR candidate set**

1. yes
2. yes it is
3. yes, it is
4. yes , it is in color
5. yes it is in color

**Top 10 from the remaining NDCG candidates**

1. yes this is a color image
2. yes , it 's in color
3. it is yes
4. yes this picture is in color
5. yes the photo is in color
6. yes in color
7. yep
8. it is in color
9. yes it 's color



**Question:is anyone in the bathroom ? MRR candidate set**

1. no
2. no the bathroom
3. not that i can see
4. no people
5. nope
6. i do n't think so

**Top 10 from the remaining NDCG candidates**

1. no it appears empty
2. not it 's not visible
3. no he 's not
4. i would say no
5. no , there is no 1 in the kitchen
6. UNK
7. don ' t know
8. 0
9. not really
10. no people this is the sky

**Question:is he wearing a hat ? MRR candidate set**

1. no
2. no he is not
3. hope
4. i do n't see 1

**Top 10 from the remaining NDCG candidates**

1. i do n't think so
2. no , no hats
3. no baseball hat
4. not that i can see
5. no ca n't see it
6. no he is not wearing a helmet
7. ca n't tell
8. i ca n't tell
9. can not tell
10. not really

1. no
2. yes
3. not that i can see
4. id o n't see any
5. hope

**Top 10 from the remaining NDCG candidates**

1. i ca n't see any , no
2. no people
3. no people are visible
4. no , there are n't any people in the image
5. no there are n't
6. no people in the photo
7. 0 i can see
8. not that you can see
9. i ca n't see
10. on the screens there are

**Question:do they have shoes on ? MRR candidate set**

1. yes
2. i can not tell
3. i see 1 shoe the other person is to blurry to tell
4. it looks like it
5. it 's hard to tell
6. i ca n't tell

**Top 10 from the remaining NDCG candidates**

1. looks like it
2. i guess so , it is hard to tell
3. yes i think so , you ca n't really see
4. ca n't tell
5. i think so
6. they are
7. not sure
8. no , they do not
9. ca n't see
10. yes they are green 

1. not really
2. i ca n't tell
3. no i ca n't
4. id do n't think so but it 's hard to tell
5. not that i can see
6. not sure
7. no , there is n't
8. maybe , i ca n't really tell
9. possibly
10. not in the picture
9. somewhat , yes
10. looks it

**Question:is she wearing a blazer ? MRR candidate set**

1. no
2. no , sweater
3. hope
4. not that i can tell
5. i do n't think so

**Top 10 from the remaining NDCG candidates**

1. yeah , this photo is in color
2. yes in color
3. yes , it is fully colored
4. dyes
5. full blown color
6. i think so
7. looks like it
8. it is , but there is not much color in the photo due to the objects
9. somewhat , yes
10. looks it 



**Question:**does he look angry or otherwise UNK ?

**MRR candidate set**

1. no
2. ca n't tell
3. you can n't really see his face
4. i ca n't tell

**Top 10 from the remaining NDCG candidates**

1. i ca n't see that
2. i can not tell
3. not real sure
4. hope
5. not really
6. not sure
7. i do n't think so
8. looks to be sleeping if anything
9. not that i can see
10. it looks determined , but not angry



**Question:**what color is the reindeer ?

**MRR candidate set**

1. brown , with a red nose
2. red
3. brown
4. brown and white
5. mostly white

**Top 10 from the remaining NDCG candidates**

1. white
2. golden
3. white and tan
4. brown , i think
5. white with a red stripe
6. brown , white and black
7. dark brown
8. gold
9. it is brownish gray
10. mostly white with some black

**Question:**can you tell what type they are , stone or laminate ?

**MRR candidate set**

1. they are stone
2. i can not
3. i can not tell
4. i ca n't tell
5. no i 'm not sure
6. no , i can not

**Top 10 from the remaining NDCG candidates**

1. ca n't tell
2. i can ' tell , it 's just white
3. hope
4. no
5. not sure
6. not really
7. no , i 'm bad at that
8. i ca n't te
9. no , i can only see the top
10. not that i can see



**Question:any horses ? MRR candidate set**

1. no
2. yes
3. yes , l
4. 1
5. nope

**Top 10 from the remaining NDCG candidates**

1. 2
2. i just see 2
3. yes , it looks like it
4. not that i can see
5. ye , l if not more
6. not that i see
7. i think so
8. no there is n't
9. not visible
10. barely part of 1

**Question:are they male ?MRR candidate set**

1. yes
2. no
3. yes they are
4. i think so
5. i believe so
6. it seems like it

**Top 10 from the remaining NDCG candidates**

1. 1 is
2. yes 1 is
3. hard to tell
4. yes !
5. i ca n't tell
6. ca n't tell
7. i can not tell
8. not sure
9. no idea not visible

10. ca n't tell they are far away in background

**Question:is the bench in front of a building ?MRR candidate set**

1. yes , it is
2. yes
3. yes in front of glass
4. no it is n't
5. nope



**Top 10 from the remaining NDCG candidates**

1. no
2. yes it is
3. yes looks like it
4. not that i can see
5. looks like it
6. appears so
7. it think so
8. i see a building on the left side , but only a bit of
9. i can not tell
10. i think so , it's hard to tell

**Question:**how many pizzas ?

**MRR candidate set**

1. i can see 2 kinds of pizzas
2. 2
3. there are 3
4. there are 2
5. i see 2

**Top 10 from the remaining NDCG candidates**

1. 2 in the scene
2. 3
3. i think 4
4. just 1
5. i can not tell
6. 4
7. it looks like a box of them and 2 sitting out
8. they are like 4
9. 1
10. i ca n't tell

**Question:**what color is the building ?

**MRR candidate set**

1. white
2. grey
3. it is white
4. greyish
5. beige white

**Top 10 from the remaining NDCG candidates**

1. white and black
2. blue
3. beige
4. blue with a glass door
5. bright blue
6. off white with some UNK of gray and red brown throughout
7. there is 1 building that is a cream color
8. it is glass
9. creme
10. it 's a brownish white color



**Question:is the photo in color ? MRR candidate set**

1. yes
2. yes it is
3. yes, it is
4. yes , it is in color
5. yes , the photo is in color
6. yes it is in color

**Top 10 from the remaining NDCG candidates**

1. yes , the image is in color
2. yes in color
3. yes , the picture is in color
4. the photo is in color
5. yes the picture is in color
6. it is
7. sure is
8. yes ,
9. yup
10. ye

**Question:is the day sunny ? MRR candidate set**

1. yes
2. no
3. not really
4. hope

**Top 10 from the remaining NDCG candidates**

1. yes it is
2. looks like it
3. no , it is not
4. yes , it is
5. i think so
6. UNK
7. sort of
8. i do n't think so
9. i suppose
10. ca n't really see the sun think it 's pretty overcast

1. it 's in color
2. yes , a brown 1
3. i think so
4. UNK
5. no it is in a paper boat
6. it 's okay
7. white
8. no , just on the table
9. yes , i can see some cars
10. it is in a box

**Question:is the sink made out of metal ? MRR candidate set**

yes
2. i can not tell
3. i ca n't tell
4. ca n't really tell
5. ca n't tell

**Top 10 from the remaining NDCG candidates**

1. not sure
2. no
3. yes it looks like it is
4. yes it is
5. i think so
6. looks like it
7. yes , it is
8. it look like it
9. hope
10. too far away to see that

**Question:is there people there apart from the man ?**

MRR candidate set**

no
2. no just 1 person
3. not that i can see
4. none
5. i do n't see any

**Top 10 from the remaining NDCG candidates**

no other people

i don ' t see anyone at all
3. 0
4. no people are in the picture
5. i do n't think so
6. no people
7. no , there are n't any people visible
8. not really
9. no , there are n't any people
10. no people in the room 



**Question:are the elephants eating ? MRR candidate set**

1. no
2. yes
3. nope
4. no , do n't think so

**Top 10 from the remaining NDCG candidates**

1. no does n't look like it
2. i do n't think so
3. not that i can see
4. no standing
5. it looks like it
6. only the 1 with its head down is
7. i think so
8. yes they appear to be
9. yes , they appear to be
10. 1 of the 2 of them are

1. no
2. yes
3. no it isn
4. nope

**Top 10 from the remaining NDCG candidates**

1. looks that way
2. i do n't think so
3. no not really
4. i can not tell
5. looks like it
6. looks like it is
7. no sure
8. yes it is
9. looks like
10. it appears to be

6. hope

**Top 10 from the remaining NDCG candidates**

1. not sure
2. i do n't think so
3. not that i can see
4. i can not tell
5. hard to say
6. maybe
7. i ca n't tell
8. ca n't tell
9. i ca n't see any
10. they look serious

**Question:are there any other animals in the picture ? MRR candidate set**

1. no
2. 0
3. npope
4. 0 at all
5. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no , there are 0
2. i ca n't see any
3. no , just the 1
4. i do n't think so
5. nope , just 1
6. 0 sen
7. no animals
8. no animals either
9. no other people
10. no people 

3. no not really
4. UNK
5. no it does not
6. no
7. i think so
8. no really
9. nope
10. no it 's not
2. i do n't think so
3. i can not tell
4. not really
5. yes i think so
6. nothing visible
7. yes , they are
8. i ca n't tell that
9. yes , a bunch of umbrellas
10. yes they are

**Question:are the kids playing with anything ? MRR candidate set**

1. holding umbrellas
2. no
3. yes
4. looks like a boy has something in his hand
5. nope
6. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no , all are kids
2. i do n't think so
3. i can not tell
4. not really
5. yes i think so
6. nothing visible
7. yes , they are
8. i ca n't tell that
9. yes , a bunch of umbrellas
10. yes they are 

6. 1 looks old while the other looks mid 30s
7. very old
8. they look fairly big
9. no
10. i ca n't tell , but judging by the UNK , probably

**Question:is it a metal fence ? MRR candidate set**

1. yes
2. wire
3. yes it is
4. it appears to be
5. yes , it is

**Top 10 from the remaining NDCG candidates**

1. it is
2. yes , it looks like it is
3. it looks like it
4. looks like it is
5. i think so it
6. it has some lines , but yes
7. i think so
8. yes they are
9. yes , i can see a fence
10. yes i think so , hard to tell

**Question:are there pictures on the walls ? MRR candidate set**

1. yes
2. no
3. yes there are
4. yes for sure

**Top 10 from the remaining NDCG candidates**

1. i think so
2. yes , 1 that i can see
3. not that i can see
4. hope
5. no pictures on the wall
6. yeah , 3
7. 1
8. might be but it is hard to see
9. no , id o n't see any
10. yes , but not too many 



**Question:**can you see clouds ?

**MRR candidate set**

1. i see no clouds the sky is fairly grey though
2. no clouds are visible
3. yes i can see the sky cloudy
4. no
5. no , it looks cloudy

**Top 10 from the remaining NDCG candidates**

1. no ,
2. there seems to be a few clouds
3. nope
4. yes they are clouds
5. very few
6. not that i can see
7. not really
8. i do n't think so
9. yes i can
10. 0

**Question:**where are the onions ?

**MRR candidate set**

1. on the right of the plate
2. you can not really tell
3. i can not tell
4. not sure
5. i ca n't tell

**Top 10 from the remaining NDCG candidates**

1. yes
2. ca n't tell
3. UNK
4. sitting at the table
5. it looks like a box of them and 2 sitting out
6. there is squash , shredded carrots , and shredded cabbage
7. not that i can see
8. the UNK of heaven
9. yes , it is
10. i think os



**Top 10 from the remaining NDCG candidates**

1. black
2. yes and black
3. no
4. no , her eyes are brown
5. nope
6. it looks to be maybe of shoulder length
7. ca n't tell
8. i can n't tell
9. yes i do
10. i can not tell

**Question:are there other people ? MRR candidate set**

1. there is a gentleman beside her
2. no
3. yes , there are 2 other people in the background
4. no , there is not
5. not that i can see
6. no there are not
7. nope

**Top 10 from the remaining NDCG candidates**

1. no other people in the photo
2. 0 that i can see in the picture
3. no , there are no other people com
4. 0 i can see
5. no there 's not
6. not from what i can see
7. no , i ca n't see anyone
8. not within view
9. yes 2 people can be seen in the background
10. i do n't think so



**Question:is the yard large ? MRR candidate set**

1. i can not tell
2. yes
3. i ca n't tell
4. ca n't tell
5. can not tell too close up
6. not sure

**Top 10 from the remaining NDCG candidates**

1. from what i can see
2. seems like it very close up photo
3. i think so
4. maybe
5. no
6. no , but hard to tell from the close up
7. yes it is
8. yes , it is
9. not really
10. yes it 's large
11. yes
2. no
3. yes , it does
4. yes it does match the caption
5. yes it does
6. no it is not

**Top 10 from the remaining NDCG candidates**

1. hope
2. no ,
3. not really
4. it does
5. yes , the picture is in color
6. yes , it 's in color
7. it is
8. yes , it is
9. yes it is



**Question:are there any lights ? MRR candidate set**

1. no
2. i think it 's light coming from a camera he is holding
3. yes
4. not that i can see
5. 0 that i can see
6. nope

**Top 10 from the remaining NDCG candidates**

1. not seen
2. i can not see them
3. i do n't think so
4. yes , well lit
5. i ca n't tell
6. 0 at all
7. i think so
8. i can not tell
9. yes there are

**Question:how many kids are there ? MRR candidate set**

1. 2
2. there are 2 people but they are n't kids
3. i think 2
4. only 2
5. there is only 1

**Top 10 from the remaining NDCG candidates**

1. 2 , possibly 3
2. 1
3. 3
4. looks like 3
5. i can see 2 clearly and there may be part of a third but i 'm not sure
6. there is 1
7. only see 1 , too close up to see much
8. 3 total 1 looks like the guide
9. there are tow
10. i see 1 arm besides them



**Question:are there any bath mats ? MRR candidate set**

1. no
2. yes there is a bath mat
3. i can not tell
4. nope
5. no , it does n't appear so

**Top 10 from the remaining NDCG candidates**

1. yes
2. yes there are
3. no ,
4. yes there is 1
5. there are
6. not that i can see
7. not that you can see
8. just 1
9. i do n't think so
10. 0 that i can see

**Question:can you see the sky ? MRR candidate set**

1. no , i can only see the close up of what ' s on the desk
2. no
3. not that i can see
4. i do n't see 1
5. no there is n't

**Top 10 from the remaining NDCG candidates**

1. not that can be seen

2. not visible
3. nope
4. do not see 1
5. do n't see any
6. 0 that i can see
7. ca n't tell
8. i do n't think so
9. i can not tell
10. not sure

**Question:are there any people in the image ? MRR candidate set**

1. no
2. no people
3. nope
4. no i do not see people
5. not that i can see

**Top 10 from the remaining NDCG candidates**

1. noo
2. 0 are visible
3. 0 i can see
4. 0
5. nobody around
6. 0 sen
7. do n't really see any
8. mp
9. i do n't think so
10. not really 

3. can not tell
4. i can not tell
5. i do n't know
6. i ca n't tell for sure
7. not sure
8. i don ' t know
9. hard to tell
10. not that i can see
10. the die is a grey and white i believe
10. the die is a grey and white i believe

**Question:**are they tall ?

**MRR candidate set**

1. yes , they are tall
2. yes
3. yes they are
4. yes they definitely are

**Top 10 from the remaining NDCG candidates**

1. yes a few stories at least
2. looks like
3. yes , some are
4. i think so
5. not really
6. yes , it is
7. no
8. yes it is
9. hope
10. possibly 



**Question:**does it look rusty ?

**MRR candidate set**

1. no
2. yes
3. not at all
4. nope

**Top 10 from the remaining NDCG candidates**

1. not really
2. no it is not
3. kind of
4. no but looks like it did previously
5. yes , it looks UNK
6. not that i can see
7. yes , it does
8. i do n't think so
9. yeah
10. sort of

**Question:**is this inside ?

**MRR candidate set**

1. yes
2. yes it is
3. yes , it is
4. yes it appears to be
5. i think so

**Top 10 from the remaining NDCG candidates**

1. yes it 's indoors
2. i believe so
3. yes* and yes
4. appear to be
5. most likely indoors
6. it might be
7. i believe so but it is hard to tell for sure
8. most
9. yes it 's clean
10. maybe

**Question:**how many other people are there ?

**MRR candidate set**

1. 2 people
2. a crowd
3. lots and lots
4. very many
5. approximately 20

**Top 10 from the remaining NDCG candidates**

1. it looks like over 20
2. 15 maybe
3. 10
4. about a 100
5. about 10
6. ca n't tell
7. 8 or so
8. 12 or so

9. around 10
* i can not tell

**Question:is she wearing shorts ? MRR candidate set**

1. no a white skirt
2. yes
3. no , she is wearing a skirt
4. no
5. yes she is
6. it think so

**Top 10 from the remaining NDCG candidates**

1. white
2. hard to tell , but i think so
3. nope
4. yes , it is
5. can
6. yes , they are
7. yes he is
8. yes it is
9. he
10. maybe

**Question:what type of flooring ? MRR candidate set**

1. carpet
2. tile
3. i can not tell
4. i ca't tell
5. can't tell

**Top 10 from the remaining NDCG candidates**

1. not sure
2. i don ' t know for sure
3. maybe tan
4. ca n't tell from the picture
5. no idea
6. brown
7. grey
8. seem grayish from this angle
9. green
10. color of what 



**Question:cab you tell the women 's race ? MRR candidate set**

1. caucasian
2. white
3. no i can not
4. no i can n't

**Top 10 from the remaining NDCG candidates**

1. no
2. no i do not
3. npe
4. yes
5. i can not tell
6. not sure
7. not really
8. yes , i can
9. yes i can
10. i can n't tell

**Question:cab you see the scoreboard ? MRR candidate set**

1. no
2. yes
3. i can not
4. hope
5. no i can not

**Top 10 from the remaining NDCG candidates**

1. no
3. n is cut off
2. not really
3. n
4. i do n't think so
5. not that i can see
6. i can not tell
7. not sure
8. barely
9. i ca n't tell

**Question:is she wearing sneakers ? MRR candidate set**

1. no
2. not visible
3. i can n't see her feet
4. ca n't tell
5. i can not tell
6. i can n't tell

1. ca n't know
2. npe
3. i ca n't tell from the picture
4. can
5. not sure
6. i do n't think so
7. not that i can see
8. ca n't see the bottom part
9. maybe i am not sure
10. yes

**Question:is it sunny outside ? MRR candidate set**

1. yes
2. no
3. not really
4. no it is not

**Top 10 from the remaining NDCG candidates**

1. no , it is cloudy
2. it appears sunny
3. no , it is not sunny outside
4. overcast
5. hope
6. it looks cloudy
7. no cloudy
8. it 's somewhat overcast
9. yes it is sunny
10. yes , looks sunny 

7. i think so
8. no , they are not
9. on the ground
10. yes it is a side view only

**Question:is he holding it to his ear or using UNK ?****MRR candidate set**

1. it's up to his ear
2. ca n't tell
3. i can n't tell
4. i can not tell

**Top 10 from the remaining NDCG candidates**

1. ca n't see
2. not sure
3. not that i can see
4. can you see
5. no
6. you ca n't tell
7. hope
8. not really it 's kind of far away
9. both
10. yes 



**Question:**could this be a zoo ?

**MRR candidate set**

1. yes
2. maybe
3. i think so
4. i would think so
5. ca n't tell if it is

**Top 10 from the remaining NDCG candidates**

1. looks like it
2. it looks like it
3. ca n't tell
4. possibly , not sure
5. i ca n't tell
6. possibly , ca n't really tell
7. it looks like it is
8. not sure
9. they appear to be
10. it 's hard to tell

1. yes
2. no
3. can't tell
4. ic an't tell
5. id on't know

**Top 10 from the remaining NDCG candidates**

1. i can not tell
2. not sure
3. hard to tell
4. hope
5. it does n't appear to be
6. i do not think so
7. no it is n't
8. i do n't think so
9. ca n't tell , maybe
10. no is not

**Question:**are there words on the plane ?

**MRR candidate set**

1. yes , the plane says UNK
2. yes
3. yes , there are words
4. yes , there are
5. yes there are
6. yes there is

**Top 10 from the remaining NDCG candidates**

1. yes , it says bp

2. yes , quite a few
3. some yes
4. yes , it is
5. i think so
6. it appears to be a website i see words and an image but ca n't make them out
7. yes , i see some from the window
8. yes , many windows
9. yes , water
10. yes it is

**Question:**can you see any store signs ?

**MRR candidate set**

1. no , it is blurry
2. no
3. yes
4. no i do n't think so
5. hope
6. not that i can see

**Top 10 from the remaining NDCG candidates**

1. there are 0 of those
2. in the distance
3. yes in the background
4. not really
5. yes , several in the background
6. i do n't think so
7. yes some
8. barely
9. yes quite a few in the background
10. 1 partially off in the distance 



**Question:is the man wearing a mask ? MRR candidate set**

1. no
2. yes
3. no he does not
4. no , he is not
5. nope

**Top 10 from the remaining NDCG candidates**

1. i do n't think so
2. does n't look like it
3. not that i can see
4. not really
5. no , he 's wearing a helmet
6. yes indeed
7. he is
8. yes he is
9. i do n't think he is , hard to tell with most of his head down
10. it looks like a helmet

**Question:does the room look big ? MRR candidate set**

1. it is decent size
2. yes , it does
3. yes
4. i 'd say so
5. yes , very

**Top 10 from the remaining NDCG candidates**

1. looks like it
2. it looks like
3. not too big
4. no , it is pretty small
5. yes , it is
6. average
7. it could be
8. i think it
9. not really
10. very



**Top 10 from the remaining NDCG candidates**

1. looks like it
2. i think so
3. on a portable desk thing , like a lap desk
4. yes it is on
5. yes , just 1
6. no , it looks like it is on a table
7. no on a stand
8. no a stand
9. yes , he she is
10. yes , there is something below the tv

**Question:**what color is her hair ?

**MRR candidate set**

1. brown
2. blonde
3. dark blonde
4. it's brown
5. her hair is brown
6. she has brown hair

**Top 10 from the remaining NDCG candidates**

1. dark brown
2. dirty blonde
3. dark , brown
4. i think it brown
5. brown , i think
6. it 's dirty blonde
7. black
8. red
9. UNK rose UNK
10. looks dirty



**Question:any walls ? MRR candidate set**

1. yes
2. yes, it is
3. yes a large 1
4. yes

**Top 10 from the remaining NDCG candidates**

1. yes 1
2. yes it is
3. hello it 's mostly brown
4. UNK
5. yes , to the left
6. i think so
7. no
8. barely part of 1
9. is pink
10. not that i can see



**Question:**can you see any trees ?

**MRR candidate set**

1. yes
2. yes in the background
3. yes , a few in the background
4. yes , in the background
5. yep

**Top 10 from the remaining NDCG candidates**

1. yes some
2. in the distance , a few
3. in the background
4. there are several in the background
5. there are a few large trees near by and several in background
6. yes , there are a lot of trees
7. uses
8. some
9. yes , far in the background
10. yes 1

**Question:**what is the gender of the people ?

**MRR candidate set**

1. both boys
2. female
3. hard to say maybe male and female
4. not sure

**Top 10 from the remaining NDCG candidates**

1. they look caucasian
2. looks male
3. yes
4. it appears so
5. ca n't tell
6. they are both posing for the picture 1 posing as if he 's going to hit the tennis ball the
7. i can not tell
8. no 1 is on 1st , 2nd and 3rd are not visible
9. i ca n't tell
10. yes , it is

8. can not tell not visible
9. no it does n't
10. 2

**Question:**what kind of toy is the boy playing with ?

**MRR candidate set**

1. can not tell which toy it is
2. look to be a wii video game
3. i can not tell
4. i ca n't tell
5. ca n't tell
6. i m not sure

**Top 10 from the remaining NDCG candidates**

1. you can not see who he is playing with
2. i do n't know
3. a small remote controlled 1
4. not sure
5. i see 1 , but its far
6. a tiny bird
7. kind of a derby i think it 's called
8. i ca n't see the ball
9. a bowl
10. maybe a UNK

**Question:**is the sky cloudy ?

**MRR candidate set**

1. yes
2. yes , blue skies some clouds
3. yes it is
4. yes , it is

**Top 10 from the remaining NDCG candidates**

1. yes '
2. yes but it is blue sky
3. yeah pretty much
4. yes , it is very clear
5. yes very clear
6. yes partially
7. i think so
8. it appears overcast
9. a little bit
10. relatively 

1. yes
2. metal and wood - looks like a zoo enclosure
3. yes it is
4. it appears to be
5. yes , it is

**Top 10 from the remaining NDCG candidates**

1. yes it is ,
2. i think so
3. yep
4. wood
5. hmm i think so
6. no
7. i ca n't really tell
8. they are and have metal parts too
9. i ca n't tell
10. i can not tell

**Question:**do any of them have their hair up ?

**MRR candidate set**

1. no
2. not that i can see
3. no they do n't
4. i can't tell
5. hope

**Top 10 from the remaining NDCG candidates**

1. ca n't tell
2. i do n't think so
3. i can not tell
4. i do n't know
5. not sure
6. too far away in photo to tell
7. hard to tell since they 're far away
8. the picture does not show 1
9. 0
10. maybe , not sure

**Question:is** there other kites in the sky or just the 1 ?

**MRR candidate set**

1. there is 1 other kite
2. 1
3. i can only see 1 in this picture
4. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no there are n't
2. no
3. nope
4. yes a few
5. yes
6. just beside it
7. yes , both
8. no , no 1 else in the photo
9. in the air 

10. no just her

**Question:is it a close up of the bear ? MRR candidate set**

1. yes
2. yes , it is
3. yes , pretty close
4. yes it is
5. it is , yes

**Top 10 from the remaining NDCG candidates**

1. sort of
2. i think so
3. it look like it
4. a little bit , yes
5. no
6. not really
7. yes there is
8. nope
9. yes , it is very clear
10. a few inches or so

**Question:is the shelf wood ? MRR candidate set**

1. yes
2. i can not tell
3. yes , it is
4. i think so
5. yes , i think so
6. yes it is

**Top 10 from the remaining NDCG candidates**

1. no
2. looks like it
3. no it 's not
4. ca n't tell
5. i ca n't tell
6. i think it is , but hard to say for sure
7. not sure
8. it is hard to tell
9. yes it is brown
10. i do n't think so 



**Question:**can you see any roads ?

**MRR candidate set**

1. no
2. nope
3. no i ca n't
4. id o n't
5. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no you can not
2. 0
3. np
4. i do n't think so
5. not really
6. nobody around
7. no , there are no signs
8. no buildings
9. no , it 's sunny
10. no it 's too far away

**Question:**is it sunny ?

**MRR candidate set**

1. yes
2. yes , it is
3. yes it is sunny
4. it looks overcast , but i ca n't see the sky
5. it appears to be sunny
6. yes , its daytime and it looks sunny
7. no , it 's looks overcast

**Top 10 from the remaining NDCG candidates**

1. no it looks cloudy
2. does n't really seem to be
3. yes it is
4. no
5. it seems like it
6. looks like it is yes
7. appears to be
8. it 's overcast
9. not really
10. i believe it is



**Top 10 from the remaining NDCG candidates**

1. hope
2. there is 0
3. 0 that i can see
4. not that i can see
5. 0 visible
6. 0
7. no , ca n't see the street
8. not really
9. i do n't think so
10. no , she is not

**Question:are they outside ? MRR candidate set**

1. yes
2. yes , they 're outside
3. yes , they are
4. yes they are
5. it looks like it

**Top 10 from the remaining NDCG candidates**

1. it seems like it
2. ithink so
3. i believe so
4. yes , it is
5. yes it is outside
6. probably
7. looks like
8. yes , on a street corner
9. it looks outside
10. yes it is

8. yes
9. blue yellow and white
10. blue and black **Question:is** it sunny outdoors ?

**MRR candidate set**

1. yes
2. yes it is
3. yes , it is
4. yes , it is a sunny day
5. yes sunny

**Top 10 from the remaining NDCG candidates**

1. it is
2. very
3. yes , the weather looks nice
4. yes the sun is shining
5. yes appears so
6. it appears to be sunny
7. it seems to be sunny
8. seems like it
9. yes , but i can not see the sun
10. yrd

\begin{tabular}{l l}
**Question:do you see any lights ?** \\
**MRR candidate set** & \\  1. no & 2. yes \\
3. yes i see 2 lights & \\  4. i think so & \\  5. yes many & \\
**Top 10 from the remaining NDCG candidates** & \\  1. 1 in the background & \\  2. no i do n't & \\  3. no , i don n't & \\  4. hope & \\  5. yes , 1 big 1 & \\  6. yes , some on the floor & \\  7. i do n't think so & \\  8. ’no i do n't & \\  9. not that i can see & \\  10. yes above the sink & \\ 



**Top 10 from the remaining NDCG candidates**

1. i am not sure
2. ca n't tell
3. hard to tell
4. no , it 's indoors
5. i can not tell
6. i do n't think they are based on their clothing
7. not that i can see
8. not sure
9. ca n't tell - maybe
10. i think so

**Question:is it a passenger train ? MRR candidate set**

1. yes
2. no
3. yes it is
4. yes , it is
5. yes , it think so

**Top 10 from the remaining NDCG candidates**

1. yes , it looks like it
2. it appears to be
3. it think so
4. it looks like it , hard to tell
5. yes , it 's big
6. could be
7. i ca n't tell for sure , but i think so
8. maybe
9. UNK
10. 2



**Question:**describe the person holding the treat ?

**MRR candidate set**

1. ca n't see the person
2. i can only see their arm they are wearing blue
3. all i can see is a hand but probably a male
4. i can not tell
5. ca n't tell from the picture
6. i am unable to tell

**Top 10 from the remaining NDCG candidates**

1. i can not see
2. i ca n't tell
3. not visible not seen
4. i ca n't tell his back is towards me
5. ca n't tell
6. he looks confused
7. not sure
8. woman
9. UNK smiling
10. the shirt is striped in black and white

1. 2
2. 3
3. there are 2
4. i think i see only 2
5. 2 in the picture

**Top 10 from the remaining NDCG candidates**

1. there are 3
2. at least 4
3. 4
4. i can see 4 or 5
5. there are tow
6. 1
7. more than 3
8. i can not tell
9. not sure
10. i ca n't tell

2. yes , it is day
3. yes , daytime
4. yes it is daytime
5. it is day time
6. yes it is , it 's sunny
7. yes , i believe so
8. it is daytime
9. i think so
10. it 's daytime

**Question:are there any utensils on the table ? MRR candidate set**

1. no
2. there are no utensils
3. nope
4. no , there are not
5. no , not that i can see
6. no there is not

**Top 10 from the remaining NDCG candidates**

1. no , there are 0
2. not that i can see
3. 0 that i can see
4. 0 that can be seen
5. 0
6. 0 are visible
7. not seen
8. i do n't think so
9. not really that i can see
10. no other thing

**Question:can you see the floor ?MRR candidate set**

1. no
2. yes
3. nope
4. ic an't
5. no ic a't

**Top 10 from the remaining NDCG candidates**

1. no i can not
2. no , the floor is n't visible
3. no , not at all
4. no you can not
5. not really
6. barely
7. i do n't think so
8. not that i can see
9. a little bit
10. yes , i can 



**Question:**what color is the dog ?

**MRR candidate set**

1. brown
2. it is black and tan
3. it is mostly brown with some tan
4. brown and white
5. light brown
6. brown with white

**Top 10 from the remaining NDCG candidates**

1. tan color
2. it is reddish-brown
3. brown and black
4. brown , black , tan , and white
5. black and brown
6. i think its a golden retriever , so light yellow
7. browns and blacks
8. white and brown
9. no people
10. UNK

**Question:**are they wearing ski goggles ?

**MRR candidate set**

1. i can not tell
2. no
3. i just see there back s i ca n't see their faces
4. yes
5. i ca n't tell
6. no , they are n't wearing goggles
7. can't tell

**Top 10 from the remaining NDCG candidates**

1. ca n't see faces see backs
2. i am not sure
3. no they are not
4. i believe so
5. yes they are
6. hope
7. not that i can see
8. i do n't think so
9. again , i can not see their face
10. they look like it



**Top 10 from the remaining NDCG candidates**

1. yes very fresh looking
2. the 1 i can see does , yes
3. long
4. ithink so
5. at least 2 of them do
6. i 'm sure it can
7. yes , it is
8. yes there 's clearance
9. mostly brown color
10. maybe

**Question:**how big is the table ?

**MRR candidate set**

1. i can not tell
2. i ca n't tell
3. hard to say , it 's too close up
4. ca n't tell
5. close up ca n't tell

**Top 10 from the remaining NDCG candidates**

1. ca n't tell from the picture
2. looks pretty big to me
3. pretty small
4. not sure
5. pretty large
6. quite large
7. medium size
8. medium sized
9. big
10. it 's more narrow than most



**Question:any signs ? MRR candidate set**

1. no
2. yes
3. 0
4. not that i can see
5. nope
6. not that i see

**Top 10 from the remaining NDCG candidates**

1. 0 visible
2. not visible
3. nno
4. yes a couple
5. id o n't think so
6. yes a few
7. yes , 1
8. 1
9. not really
10. ca n't see

**Question:is it american ? Question:is the bed outside ? MRR candidate set**

1. do n't know
2. no
3. yes
4. ca n't tell
5. ic an't tell
6. can not tell
7. i can not tell

**Top 10 from the remaining NDCG candidates**

1. not sure
2. i do n't think so
3. nope
4. i think so , i ca n't tell
5. maybe
6. no it is n't
7. i believe so , but ca n't say for sure
8. i think so
9. perhaps
10. it appears to be but it is hard to confirm

1. no
2. i can not tell
3. no , it does n't appear to be
4. no it is not
5. no it 's not
6. hope

**Top 10 from the remaining NDCG candidates**

1. do n't think so
2. io n't think so
3. does n't look like it
4. don ' t see a bed
5. no , it is inside
6. not that i can see
7. inside
8. i ca n't tell
9. i do n't think so , it 's hard to tell
10. it is not from what i see

**Question:is there windows ? MRR candidate set**

1. yes there is
2. yes
3. a few
4. there is a window which has bright sunlight streaming in
5. yes , it looks like

**Top 10 from the remaining NDCG candidates**

1. there are about 3
2. yes , it is
3. i think so
4. yes there are tree 's
5. yes it is
6. only 2 are
7. yes , there are some storefronts
8. in the distance
9. 1
10. none 

1. no there is n't
2. no
3. not that i see
4. not seen in the image
5. yes , there appears to be one , but i can not actually see it
6. not visible
7. ca n't see it in the picture
8. yes there is a faucet no actual sink
9. ca n't see 1
10. yes there is
3. yes it is
4. yes , it is

**Top 10 from the remaining NDCG candidates**

1. yes it 's pretty big
2. not really
3. i think so
4. kinda
5. medium size
6. no
7. normal size
8. medium sized
9. medium
10. it 's wide but not tall 

7. looks to be
8. barely
9. a bit
10. yes , it is

**Question:are the man and woman the only people in the photo ? MRR candidate set**

1. yes
2. no
3. no , they are not
4. yes both are
5. nope

**Top 10 from the remaining NDCG candidates**

1. i think so
2. no other people that i can see
3. yes that is all
4. i do n't think so
5. just the woman and the little girl
6. yes , it is
7. yes it is
8. not that i can see
9. yes it 's look like it was take late at night
10. not really

**Question:is anyone watching him ?MRR candidate set**

1. no
2. i do not see anyone else in the picture
3. not that i can see
4. i ca n't tell
5. nope

**Top 10 from the remaining NDCG candidates**

1. i do n't think so
2. i can not tell
3. ca n't tell
4. no he 's not
5. hard to see
6. no he is not
7. not sure 

8. can
9. possibly
10. i believe so , but ca n't say for sure

**Question:are they big boats or small boats ? MRR candidate set**

1. small
2. they look small
3. they are all different sizes
4. to scale they are large , medium and small UNK
5. medium

**Top 10 from the remaining NDCG candidates**

1. mix
2. both
3. on the small side
4. no , more like small yachts
5. they look older
6. no , there are 2 larger ones and 1 smaller 1
7. no the boats are all on the sand
8. no they are small
9. no about medium like the kind that cut hair
10. yes

**Question:are there other people there ?MRR candidate set**

1. no
2. yes
3. yes there are

**Top 10 from the remaining NDCG candidates**

1. yes a few others
2. yes , there are
3. not that i can see
4. yeah , 3 total
5. not in the photo
6. hope
7. yes , there are 2 other people in the background
8. a couple in the background , and 1 woman barely seen in the foreground
9. yes a lot
10. 3 people 

1. the carrots and what looks like maybe corn are about equal
2. i can not tell
3. i ca n't tell
4. i ca n't see the image
5. not sure

**Top 10 from the remaining NDCG candidates**

1. meat
2. ca n't tell
3. it looks quite fresh
4. it looks like it might be apple
5. it appears to be people but i do n't have the best view
6. looks brown or black spots on cream
7. light colored
8. a little bit
9. hard to tell , tomato sandwich
10. just the ocean

**Question:**does the field have a lot of grass ?

**MRR candidate set**

1. no
2. it is in a parking lot
3. no , it does n't
4. hope
5. not that i can see

**Top 10 from the remaining NDCG candidates**

1. that part is not in the photo
2. no not in view
3. not really
4. not especially
5. 0
6. ca n't tell
7. i do n't think so
8. i can not tell
9. i ca n't tell
10. grass 

1. white
2. there is a blue sign and a white with red writing sign
3. blue
4. black and white
5. white and black

**Top 10 from the remaining NDCG candidates**

1. white with writing
2. it is white
3. white , black and red
4. the sign is black
5. yes , it is
6. red and white
7. they are white with black around the edges
8. 1 yellow and 1 white
9. yes it is
10. UNK

**Question:**how old does she look ?

**MRR candidate set**

1. 2
2. 3
3. about 9 , i am not sure
4. maybe 2 at oldest
5. maybe under 1 year old
6. maybe 5

**Top 10 from the remaining NDCG candidates**

1. maybe 7
2. 4
3. 1
4. maybe 7-10
5. looks like a preteen girl
6. 7-9
7. 5
8. not sure
9. 7 or 8
10. around 8 

3. yes but it is not on
4. yes , 1
5. yeds
6. nope
7. not seen
8. no , there is n't
9. the top of the desk is not visible
10. no there is n't
11. no
2. no people
3. nope
4. no there are no people
5. no there are n't

**Top 10 from the remaining NDCG candidates**

1. no people are visible
2. not that i can see
3. no people at all
4. i do n't see any
5. 0 that i can see
6. 0
7. not that can be seen
8. nobody in the photo
9. no , this is a looking up picture
10. i do n't think so

**Question:**how many stories is it ?

**MRR candidate set**

1. ca't tell
2. i can not tell
3. ca n't tell because the photo is cut but at least 3
4. i ca n't tell
5. maybe 3
6. not sure

**Top 10 from the remaining NDCG candidates**

1. 3
2. i think 4 

3. at least 4
4. ca n't see the bottom part
5. 2
6. at least 5
7. 4
8. 5 that i can see
9. i see 4
10. there are only 2

**Question:is the grass green ? MRR candidate set**

1. yes
2. yes , some is
3. some of it yes
4. yes it is
5. yes , it is

**Top 10 from the remaining NDCG candidates**

1. there is , yes
2. some
3. yes , i think so
4. some parts of it
5. light greenish yellow
6. lots of grass but yellow
7. i think so
8. green
9. yes , they are
10. yes light brown

**MRR candidate set**

1. the colors are very vibrant
2. no
3. not really
4. hope

**Top 10 from the remaining NDCG candidates**

1. yes
2. not that i can see
3. no , there are n't
4. i do n't think so
5. looks great
6. just the sea doo
7. 0
8. no there are n't
9. not visibly , but i 'd say probably so
10. yes , it is 



**Question:are there any farmers near them ? MRR candidate set**

1. there are no people visible
2. no , there are n't any people in the image
3. no , there is not
4. no there are not
5. no there are n't
6. none

**Top 10 from the remaining NDCG candidates**

1. no
2. not that i can see
3. there are no people in the image
4. no there isn ' t
5. no , there are no people
6. id do n't see any
7. no there 's not
8. there are no people
9. no people
10. 0 i can see

**Question:what does the sign say ?MRR candidate set**

1. welcome to UNK
2. UNK no UNK
3. it just says UNK dame with an arrow pointing left
4. words
5. UNK UNK
6. s gay st

**Top 10 from the remaining NDCG candidates**

1. join our csa
2. beautiful code : leading UNK explain how they think
3. the UNK of heaven
4. the original tour
5. twin peaks with 200 sign on top
6. number 9
7. not sure
8. UNK
9. no
10. alaska



**Question:**what is made out of wood ?

MRR candidate set

1. wood
2. wood maybe
3. i can not tell
4. cabinets , stove
5. ca n't tell
6. hard to tell
7. i ca n't tell
8. not sure

Top 10 from the remaining NDCG candidates

1. looks like metal with wood slats
2. looks like a table
3. table
4. the walls
5. ca n't really tell but i would assume so
6. yes
7. yes , it is
8. i think so
9. i ca n't tell but i am guessing a large bin
10. it looks to be scissors and a beer can box
1. no
2. yes
3. hope
4. no there are n't

Top 10 from the remaining NDCG candidates

1. 1 is
2. no there is not
3. i do n't think so
4. not that i see
5. you can 't see the floor
6. 1 is the other is sitting on top of a container
7. not that i can see
8. ys
9. i think so
10. not really

3. yes there is
4. yes , there is
5. yep

**Top 10 from the remaining NDCG candidates**

1. UNK
2. i think so
3. yes , it is
4. yes it is
5. yes behind
6. yes there are 2
7. yes , several of them
8. yes it 's black
9. yes there is a rug
10. yes on a hamper

**Question:**do the trees have leaves ?

**MRR candidate set**

1. yes
2. yes a lot
3. yes , very
4. yes they do
5. 'yes

**Top 10 from the remaining NDCG candidates**

1. yes , they are
2. some have leaves
3. yes a few of them
4. i think so
5. kinda
6. yes , they look fine
7. yes , it is
8. yes , a few
9. green
10. yes it is 



**Question:if you were to guess , what building is the image set in ? MRR candidate set**

1. i can not tell
2. i can not see it from here
3. i ca n't tell
4. ca n't tell
5. not sure

**Top 10 from the remaining NDCG candidates**

1. it is in a home
2. maybe an office or a school
3. can not tell since it is outside
4. it looks like a restaurant
5. this was not taken by a professional
6. in a parking lot somewhere
7. i can not tell by the picture how nice the restaurant is
8. it is hard to tell , i can only see the table cloth
9. no , you can not
10. he is n't in a kitchen

**Question:what is on the sign ? MRR candidate set**

1. letters
2. 1 says stop and the other says danger
3. pictures
4. i do n't understand the question
5. a flickr UNK

**Top 10 from the remaining NDCG candidates**

1. look like a game
2. land in the background
3. some website not sure
4. UNK
5. not sure
6. a track and a little gate building
7. a few penguins out in penguin element
8. yes , i see like 6 billboard signs
9. it 's a dessert
10. large rocks on the side of the road



**Question:is there any cream or sugar near the mug ? MRR candidate set**

1. no , there is a spoon next to the mug on the plate however
2. yes , a small scoop of yellowish ice cream on top of the pastry
3. no there is not
4. no that i can see
5. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no
2. nope
3. not visible in the picture , no
4. i do n't think so
5. yes
6. yes there is
7. i can not tell
8. 0
9. mug
10. not really & Question:can you see his opponent ? MRR candidate set**

1. no
2. no i can not
3. no , i can not
4. nope

**Top 10 from the remaining NDCG candidates**

1. no , not at all
2. not in my view
3. no just 1 person
4. i do n't think so
5. not that i can see
6. i 'm not sure
7. no , only people
8. not really
9. there is no scoreboard
10. sure why not

1. no
2. not that i can see
3. nope
4. not that i can tell
5. 0 that are visible

**Top 10 from the remaining NDCG candidates**

1. id o n't think so
2. 0
3. no , thankfully
4. yes
5. i do n't know
6. i think so
7. it is a close up of the plate
8. not really
9. i can not tell
10. yes 1
1. yes
2. yes , it is
3. yes it is
4. yes in color
5. yes it appears to be

**Top 10 from the remaining NDCG candidates**

1. it is
2. yes , i think so
3. yea
4. i think so
5. yes* and yes
6. appears to be
7. i believe so
8. most
9. hell yeah
10. most likely 

1. a house
2. a UNK and the sky
3. tall grass
4. yes , there are a few buildings
5. rocks and small bushes
6. the brick building
7. a tree
8. yes
9. not sure
10. out in the open

**Question:is there anyone on the motorcycle ? MRR candidate set**

1. no
2. i can not tell
3. i can not see
4. not that i can see
5. nope

**Top 10 from the remaining NDCG candidates**

1. i ca n't tell
2. ca n't tell
3. id o n't think so
4. no 1
5. i ca n't see any but probably
6. not sure
7. it 's too blurry to tell
8. 0
9. only his feet are visible
10. no people 

3. no , it is not
4. yes it is
5. not in the picture no
6. yes , it is
7. no not open
8. no it 's not
9. i think so
10. it looks like it

**Question:is he eating over the sink ? MRR candidate set**

1. no he is eating over the island counter
2. no he is n't
3. no he is not
4. no he isn ' t
5. no , he is not
6. npope

**Top 10 from the remaining NDCG candidates**

1. no
2. no not eating
3. i do n't think so
4. yes
5. yes he is
6. i can not tell
7. i ca n't tell
8. yes , he is
9. not that i can see
10. i ca n't see that part 

8. not sure
9. ca n't tell
10. yes 1

**Question:**are there any cacti ?

**MRR candidate set**

1. no
2. 0
3. hope
4. not that i can see
5. no there are not

**Top 10 from the remaining NDCG candidates**

1. no people are visible
2. no i don ' t
3. there are no people
4. no people at all
5. no people
6. not that i can see
7. no , just the food
8. 0 at all
9. 0
10. there are 0 of those

**Question:**are there any cacti ?

**MRR candidate set**

1. no
2. 0
3. hope
4. not that i can see
5. no there are not

**Top 10 from the remaining NDCG candidates**

1. 0 at all
2. no i do n't see any
3. i don ' t see any
4. not any in the picture
5. i do n't think so
6. sadly , no
7. non
8. do n't have a clue
9. yes
10. no , just the grasses 

4. yes , but ca n't make it out
5. yes there is
**Top 10 from the remaining NDCG candidates**

1. just numbers
2. there may be but it is too difficult to see
3. amtrak
4. yes , it is
5. not that i can see
6. i think so
7. no
8. hope
9. no there is not
10. yes , a little
3. yes it does
4. no
5. no , it is n't
6. not really

**Top 10 from the remaining NDCG candidates**

1. i can not tell
2. it looks mostly sunny
3. npope
4. i ca n't tell
5. i guess so , it is hard to tell
6. yes , i 'd say so
7. not that i can see
8. not sure
9. ca n't tell
10. i do n't think so 

3. yes it is
4. looks like it
5. yes , it is
6. i think so
7. i only see a portion of it , so ca n't tell
8. yes i can see it
9. not sure
10. yes absolutely

**Question:**can you see the floor ?

**MRR candidate set**

1. no
2. no , you can not
3. no , i ca n't
4. hope
5. no i ca n't

**Top 10 from the remaining NDCG candidates**

1. no you can not
2. no it is not visible
3. not really
4. no , i ca n't see the ground
5. yes the floor is visible
6. yes , the floors are visible
7. i do n't think so
8. no this is inside on a table
9. yes

**Question:are there any other people in the picture ? MRR candidate set**

1. yes there are several people walking in the background
2. yes
3. there are people in the background
4. there are a few
5. yes lots of them

**Top 10 from the remaining NDCG candidates**

1. there are people
2. ye
3. there are trees in the background , yes
4. yes , a couple ball boys
5. yes , audience
6. yes , it is
7. i think so
8. no other people can be seen in the photo 

9. there is 1 man in the background
10. there are buildings far off in the distance , it is the only close 1

**Question:is the noodles beside the chicken ? MRR candidate set**

1. no it is all mixed together
2. i can not tell
3. yes
4. no
5. yes it is
6. yes , it is
7. yes !

**Top 10 from the remaining NDCG candidates**

1. no it is n't
2. across from it
3. hope
4. i think so
5. possibly yes
6. i do n't believe so
7. not that i can see
8. i do n't think so
9. i think it is , but hard to say for sure
10. not really

**Question:can you read the writing on the ball ? MRR candidate set**

1. no , i ca n't make it out
2. no
3. i ca n't , it 's blurry
4. it 's in another language
5. yes i can but it is in a foreign language

**Top 10 from the remaining NDCG candidates**

1. yes
2. yes i can
3. nope
4. there are a lot of small words it is hard to read
5. yes i do
6. not really
7. i can not tell
8. there are several , but i ca n't see the names 

9. ca n't see
10. i can not tell
2. it appears clean
3. i ca n't tell
4. very clear
5. can not tell
6. ca n't tell

**Top 10 from the remaining NDCG candidates**

1. it is clean
2. clean
3. neither
4. can barely see the water
5. not sure
6. it 's a fairly deep bluish-green , does n't look like you could see through it
7. it does not look brown
8. grey
9. choppy
10. looks pretty gross

**Question:is it a living room ? MRR candidate set**

1. yes
2. i think so
3. it appears to be
4. yes it is
5. yes i think so

**Top 10 from the remaining NDCG candidates**

1. looks like it
2. yes , it is
3. seems to be
4. it looks like it could be
5. it could be
6. maybe
7. it is
8. might be , not sure
9. no
10. not sure 

1. brown
2. black
3. i ca n't see from the angle
4. i can not tell
5. i ca n't tell

**Top 10 from the remaining NDCG candidates**

1. gold
2. beige and brown
3. brown and green
4. ca n't tell
5. not sure
6. the look like they are black and white
7. beige
8. looks like they are red
9. gray
10. light grey

**Question:**can you see the clouds ?

**MRR candidate set**

1. no
2. no i can not see the sky
3. no ca n't see sky
4. no , i ca n't see the sky

**Top 10 from the remaining NDCG candidates**

1. no i ca n't
2. no i can not
3. npe
4. i ca n't
5. i can not
6. i can not tell
7. no the sky is clear
8. i ca n't tell
9. i do n't think so
10. not that i can see 

7. a round beige 1
8. just about every color UNK
9. it is white
10. white

**Question:**how old is the kid ?

**MRR candidate set**

1. not sure maybe 3-4
2. around eleven
3. can not tell
4. ca n't tell
5. i can not say
6. i can not tell

**Top 10 from the remaining NDCG candidates**

1. ca n't tell from the picture
2. i ca n't tell
3. probably around 2
4. not sure
5. about 2-3
6. maybe 1
7. he looks young maybe 6 or 7
8. i ca n't make out their age
9. maybe 8
10. the boy looks about 7 or 8
11. yes
2. i think so
3. yes , he is
4. yep

**Top 10 from the remaining NDCG candidates**

1. yes , i think so
2. he 's inside
3. seems to be
4. it is indoors
5. it looks like it is
6. yes it looks like it was taken indoors
7. yes it is
8. yes , it is
9. looks like
10. inside i think 



**Question:is the woman old ? MRR candidate set**

1. no
2. it is hard to tell , but i would say she is in her twenties
3. probably in her 30s
4. not very young
5. i do n't know
6. npe

**Top 10 from the remaining NDCG candidates**

1. i do n't think so
2. no , she seems like she could be in early college
3. not sure
4. doubtful
5. middle aged i 'd say
6. no not really
7. ca n't tell
8. not really
9. not that i can see
10. i think so , ca n't see her face well

**Question:are there buildings ?MRR candidate set**

1. no
2. not that i can see
3. i ca n't see any
4. nope

**Top 10 from the remaining NDCG candidates**

1. no there are n't
2. it does n't seem like there are any
3. 0 seen
4. 0
5. ca n't see
6. i do n't think so
7. no , on a sidewalk of a street
8. no it is a close up of them
9. not really

* **Question:**what color are the countertops ?
* **MRR candidate set**

1. white
2. beige
3. they look beige
4. they are white
5. stainless steel

**Top 10 from the remaining NDCG candidates**

1. i think off white
2. black
3. like a beige
4. dark wood
5. brown
6. tan
7. eggshell
8. egg white
9. variety of colors
10. black i think

1. no
2. yes
3. nope

**Top 10 from the remaining NDCG candidates**

1. no i do n't
2. i do n't
3. no i do not
4. no i don
5. no
6. i do n't think so
7. not in the picture
8. not that i can see
9. no it doesn

' t look like it
10. not really

**Question:**can you tell what they are stuffed with ?

**MRR candidate set**

1. no , they are in an self serving orange bottle , it almost looks like a fruit drink
2. i can not tell
3. no
4. not really
5. nope

**Top 10 from the remaining NDCG candidates**

1. i ca n't tell
2. not sure 

3. not that i can see
4. only the top , which looks to be crumbs for some reason beyond me
5. ca n't tell
6. id o n't see any
7. no the big ones
8. yes
9. i do n't think so
10. it looks like a seasoned french bread

**Question:**what color are the sweatpants ?

**MRR candidate set**

1. grey and white
2. grey
3. gray
4. brown
5. black
6. they are khaki

**Top 10 from the remaining NDCG candidates**

1. gray mostly
2. not clear
3. 1 is black and the other is grey
4. 1 is tan and the other is brown
5. they 're gray metal
6. look orange or tan
7. grey , black , white
8. 1 is dark , the other a lighter brown
9. black and gold
10. not sure 

1. i ca n't tell if there are doors
2. it is open
3. yes
4. i can t tell from this angle
5. yes ,
6. no they are closed
7. no they are all closed
8. no
9. yes it is open
10. halfway it looks like

**Question:is there only 1 baseball player ? MRR candidate set**

1. 1 batting and a catcher
2. yes
3. yes there is
4. 1 that is visible yes

**Top 10 from the remaining NDCG candidates**

1. only 3 players can be seen from the angle
2. yes , he plays for the mets
3. no i see 2
4. there are 2
5. it appears to be
6. es
7. no
8. yes about 5
9. yes , it is

**Question:how old do they seem to be ? MRR candidate set**

1. they are pretty far away but seem to be an older couple
2. i can not tell
3. i can't tell
4. can't tell

**Top 10 from the remaining NDCG candidates**

1. middle aged
2. i ca n't tell his back is facing the camera
3. i would say 30s
4. maybe mid thirties
5. not sure
6. 20-30
7. pretty old , in their 60s
8. 20 's or 30 's
9. mid thirties 

10. maybe late 20's to early 30's

**Question:**are there any utensils shown ?

**MRR candidate set**

1. no
2. yes
3. nope
4. no there are n't
5. yes , there are

**Top 10 from the remaining NDCG candidates**

1. not that i can see
2. yes there are some
3. not really
4. i do n't think so
5. 0
6. i think so
7. 0 at all
8. yes a glass bowl
9. lots
10. 1

**Question:**is the photo in color ?

**MRR candidate set**

1. yes
2. yes it is
3. yes , it is
4. yes , it 's a color image
5. yes , the image is in color
6. yes it 's in color

**Top 10 from the remaining NDCG candidates**

1. yes , the picture is in color
2. yes the picture is in color
3. yes in color
4. it is
5. it is in color
6. yes ,
7. yep
8. definitely
9. i believe it is , yes
10. yes , it look professional 

5. 5 people
6. i see 4 people

**Top 10 from the remaining NDCG candidates**

1. i see 7
2. 3
3. there are 4 total
4. 8
5. 3 that i can see
6. about 10
7. around 9
8. 4 or 5 , they are very tiny and hard to count
9. about 3
10. looks like 3

**Question:is there any people ? MRR candidate set**

1. no
2. just a photo of 1
3. nope
4. not that i can see

**Top 10 from the remaining NDCG candidates**

1. no there is n't
2. no people
3. not in the image
4. 0 that i can see
5. i do n't see any
6. no 1
7. no people in the shot
8. no people present
9. there are no people
10. 0

**Question:is it clean ? MRR candidate set**

1. yes
2. kind of , it 's under construction
3. no
4. yes , it is

**Top 10 from the remaining NDCG candidates**

1. appears to be
2. yes it is
3. it looks like it
4. somewhat
5. hard to tell
6. kind of
7. i think so
8. sort of 

9. not really
10. hope **Question:is** it a male ? **MRR candidate set**

1. yes
2. hard to tell
3. i can not tell
4. is like that
5. ca n't tell
6. i ca n't tell

**Top 10 from the remaining NDCG candidates**

1. not sure
2. i think so
3. possibly , i ca n't tell
4. no sure
5. looks like
6. it appears to be
7. could be
8. seems to be
9. yes , it is
10. yes it is 



**Question:**would you say this is a mother and child ?

**MRR candidate set**

1. yes i would
2. yes
3. possibly
4. it think so
5. yes it is
6. not sure

**Top 10 from the remaining NDCG candidates**

1. maybe
2. i can not tell
3. yes , it is
4. i ca n't tell
5. ca n't really tell , but i would believe so
6. no
7. no can not tell
8. i do n't think so
9. ca n't tell
10. hope

**Question:**is it day or night ?

**MRR candidate set**

1. day
2. daytime
3. it is daytime
4. day time
5. it 's daytime

**Top 10 from the remaining NDCG candidates**

1. it looks to be day time
2. it is mid day
3. day time i think
4. it looks like it may be late afternoon
5. in the even or morning
6. probably afternoon
7. yes
8. yes it is
9. morning
10. yes , it is

6. not necessarily
7. silver
8. hard to tell , some kind of pine i think
9. all gray
10. grey

**Question:**can you see any other cows ?

**MRR candidate set**

1. no
2. nope
3. no i can not

**Top 10 from the remaining NDCG candidates**

1. no , i ca n't
2. just the 1
3. only 1
4. i do not
5. no i do n't
6. not that i can see
7. 0 at all
8. no3
9. 0
10. i do n't think so