SlowGuess/NAACL_2021_OCR huggingface dataset:
Idan Schwartz
idansc@cs.technion.ac.il
Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., 'yeah' and 'yes'). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as 'I don't know.' Crafting a model that excels on both MRR and NDCG metrics is challenging (Murahari et al., 2020). Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art performance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https://github.com/idansc/mrr-ndcg.
Das et al. (2017) introduced the task of Visual Dialog, which requires an agent to converse about visual input. Evaluating visually aware conversation should examine both linguistic properties and visual reasoning. Analysis of generative metrics for dialog often shows no correlation with human judgments (Liu et al., 2016). Hence, to evaluate the correctness of the candidate answers, a retrieval approach is preferred. Two metrics are standard,
Question: what is the nightstand made of?
1. can't tell it's covered in cloth
2. it appears to be a large red
pillow that may be leather
3. I can't tell
4. I can not tell
5. not sure
6. can't tell
7. some kind of metal, it's out of focus
8. Wood
…
99.0
100. I can't see a baggage cart
Figure 1: A visual dialog interaction. The question asks, "what is the nightstand made of ?". We show our final ranking, created by the ensemble of an MRR/NDCG models' rankings. The MRR/NDCG models are trained to optimize the MRR/NDCG metric. The MRR metric measures the number of retrievals to retrieve the human-derived answer. Hence, the MRR model favors human-like and detailed answers. On the other hand, the NDCG metric measures the rank of all the correct candidates based on dense annotation, which are often general and uncertain. Our ensemble approach seeks a minimal candidate set that is likely to contain the human-derived answer. The remaining candidates are ranked according to the NDCG model.
MRR and NDCG. The MRR metric focuses on a single human-derived ground-truth answer. Despite preferring the more human-like answer, the metric ignores many correct candidate answers. Differently, the NDCG considers the rank of all the correct answers. The metric relies on dense annotation, where three annotators were asked to mark all the correct candidate answers. However, the candidate answers are generated plausible answers. The analysis shows that the NDCG metric favors uncertain, generally correct answers, such as "not sure" (Murahari et al., 2020; Qi et al., 2020).
Prior work in visual dialog focused on a single metric. Ideally, an AI agent should answer humanlike and detailed reply (the MRR metric) and be able to validate the correctness of any answer (the NDCG metric). However, crafting a model that excels in both metrics is challenging (Murahari et al., 2020). To this end, we propose principals to ensemble the rankings of strong MRR and NDCG models. Our approach is to find a minimal set that is likely
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3272-3363 June 6-11, 2021. ©2021 Association for Computational Linguistics
to hold the human-derived answer. This permits ranking the rest of the candidates according to the NDCG model. Our approach won the recent Visual Dialog 2020 challenge and achieved strong performance on both the MRR and the NDCG metrics simultaneously.
Visual conversation evaluation: Early attempts to marry conversation with vision used street scene images, and binary questions (Geman et al., 2015). While binary answers are easy to verify, such an approach is limiting for an AI agent. On the other hand, analysis of generative metrics for dialog often show no correlation with human judgements (Liu et al., 2016). Intuitively, metrics like BLEU-scores rely on corresponding words with the ground-truth answer and often miss synonyms or the subjective nature. More importantly, generative metrics are geared toward textual assessment rather than visual reasoning, which results in models mainly relying on textual cues (Schwartz et al., 2019a). Malinowski and Fritz (2014) suggest Wu-Palmer similarity metric that calculates similarity based on the depth of two words based on the WordNet taxonomy (Miller, 1995). A different approach suggested in the VQA dataset focus only on brief, mostly 1-word answers (Antol et al., 2015). In this setup, the task turns into popular answers classification, alleviating many text-generation challenges. Notably, VQA requires 3 out of 10 annotators to agree on the answer, which is robust to inter-person variation. Still, accuracy ignores the reasoning process. Hudson and Manning (2019) propose GQA, which extends the accuracy metric and uses a scene graph for both question generation and evaluation. Following, Das et al. (2017) propose the VisDial dataset for the visual dialog task, which formulates multiple image-language interactions via a dialog. Concurrently, de Vries et al. (2017) propose Guess-What, a goal-driven dialog dataset for object identification. Different from VQA and goal-driven dialogs, the VisDial answers are detailed and more human-like. For instance, in Fig. 1, the answer is "Can't tell...cloth", while a VQA answer would be "cloth". Therefore, metrics that require exact matching are no longer suitable. Instead, each question is accompanied with 100 candidate answers. Consequently, the metric has been shifted from accuracy to retrieval-based metrics, e.g., MRR and NDCG. Prior works focus on optimizing a single metric (Guo et al., 2019; Jiang et al., 2020; Hu
et al., 2017; Gan et al., 2019). Differently, Murahari et al. (2020) attempt to optimize both metrics with a joint loss. Still, a dedicated single metric model is superior. Instead, we propose principals to ensemble two dedicated models, one for NDCG and one for MRR. Our approach allows most of the MRR and NDCG to be preserved simultaneously.
Visual dialog models: Various approaches were proposed to solve the Visual Dialog task. Most of them focus on dialog history reasoning per interaction. Serban et al. (2017) propose history hierarchical encoding. Seo et al. (2017) introduce a memory network based on attention, which also addressed co-referential issues. Kottur et al. (2018) focus on visual co-reference. Jain et al. (2019) concatenate representations of all the cues (e.g., image, question, history, and caption) per candidate answer. Zheng et al. (2019) employ a graph structure learning. Schwartz et al. (2019b) propose a model, namely Factor Graph Attention (FGA), that lets all entities (e.g., question-words, image-regions, answer-candidate, and caption-words) interact to infer an attention map for each modality. An ensemble of five FGA models achieves the state-of-the-art MRR performance. However, FGA optimizes using the sparse annotations, i.e., the human-derived answer. Murahari et al. (2020) recently propose Large-Scale(LS) model, which pre-trains on related vision-language datasets, e.g., Conceptual Captions and Visual Question Answering(Sharma et al., 2018; Antol et al., 2015). Concurrently, Wang et al. (2020) leverage the pretrained BERT language models, and Nguyen et al. (2020) propose a lightweight Transformer that handles the interplay between many modalities. The three methods mentioned above finetune using the dense annotation (i.e., human assessment of all the candidates), resulting in a substantial improvement on the NDCG metric. Importantly, Murahari et al. find that finetuning a model for NDCG hurts MRR performance. This work demonstrates that re-ranking MRR model (e.g., FGA) and NDCG model (e.g., LS) with simple principles keeps most MRR and NDCG performance.
3 Two-step Rank Ensemble
The MRR metric depends on a single human-derived answer. Hence, given that this answer is ranked highly, the remaining candidates can be ranked according to the NDCG model. In the following, we describe two steps: (i) the MRR step responsible for preserving the human-derived rank
high, and (ii) the NDCG step responsible for ranking the remaining candidates based on the NDCG model.
We are given a set of dialog questions  $\{(q,\mathcal{C}_q)_i\}_{i = 1}^d$  where  $d$  is the dataset size,  $q$  is a dialog question, and  $\mathcal{C}_q = \{c_{q,j}\}_{j = 1}^{100}$  are the corresponding candidates. The MRR metric, i.e., the inverse harmonic mean of rank, is defined as:
$$
\mathrm {M R R} = \frac {1}{d} \sum_ {i = 1} ^ {d} \frac {1}{r _ {\mathrm {i}}}, \tag {1}
$$
where  $r_i$  is the rank of the human response for the  $i$ -th dialog question. The DCG, i.e., discounted cumulative gain over the  $K$  correct answers, is defined as:
$$
\mathrm {D C G} _ {\mathrm {K}} = \sum_ {i = 1} ^ {K} \frac {s _ {i}}{\log_ {2} (i + 1)}, \tag {2}
$$
where  $s_i$  is a binary score, representing the fraction of annotators that marked the candidate at position as correct. We normalize by the ideal DCGK score (IDCGK), i.e., NDCGK = DCGK/IDCGK. We denote the set of MRR models as  $\mathcal{M} = \{M_1, \dots, M_{n_m}\}$  where  $n_m$  is the number of MRR models. Each MRR model is built by altering the initial conditions. We denote the NDCG model as  $N$ . We define an operator  $T(M, n, q)$  that returns the model  $M$ 's top  $n$  responses given a question  $q$ . Next, we describe the MRR step that aims to keep the MRR score.
3.2 MRR step
The purpose of the MRR step is to find a minimal candidate set  $\mathcal{C}_{\mathrm{MRR},q}$  that is likely to contain the human-derived answer given a question  $q$ . We build this set as a union of three sets, as follows:
$$
\mathcal {C} _ {\mathrm {M R R}, q} = \mathcal {T} _ {q} \cup \mathcal {N} _ {q} \cup \mathcal {H} _ {q}, \tag {3}
$$
where  $\mathcal{T}_q$  is a set of first ranked candidates according to MRR models,  $\mathcal{N}_q$  is a set of high ranked candidates by both MRR and NDCG models,  $\mathcal{H}_q$  is a set of high-certainty candidates agreed by all the MRR models. All sets are conditioned by the question  $q$ . In the following, we formally define those sets.
3.2.1 High-certainty answers
One of the most significant signals to be the human-derived answer is being a top MRR-model's answer. However, in many subjective questions, the
MRR model is not certain. We found that in those cases, the top answers often varies between different MRR models. Thus, to verify the top candidate's certainty, we require an agreement of MRR-models. Let  $q$  be a dialog question, we define the high-certainty set as follows:
$$
\mathcal {H} _ {q} = \left\{c \mid (\forall M \in \mathcal {M}; c \in \mathrm {T} (M, \rho_ {h}, q)) \right\}, \tag {4}
$$
where  $\rho_h\in \mathbb{R}$  is an hyperparameter. Intuitively, a low  $\rho_{h}$  results in higher certainty. We Next, we add the MRR-models' answer at first retrieval.
3.2.2 Top answers
The MRR metric prioritizes the first-ranked answer (see Eq.(1)). This property suits the nature of dialog models that reply with a single response. Consequently, we keep the first responses of the MRR models. Let  $q$  be a dialog question, the top-answers set is defined as:
$$
\mathcal {T} _ {q} = \{c | (\exists M \in \mathcal {M}; c \in \mathrm {T} (M, \rho_ {t}, q)) \}, \tag {5}
$$
where  $\rho_t \in \mathbb{R}$  is an hyperparameter. We note that  $\rho_t$  should be low to maintain candidates' certainty. In the next step, we consider top NDCG candidates.
3.2.3 NDCG-agreement answers
When the NDCG model and the MRR model agree that a candidate is likely to be correct, it implies that both the NDCG and MRR metrics gain by ranking this candidate high. Thus, we want to rank it high. We note that the MRR set is ranked first, so we include these candidates in the MRR set. Let  $q$  be a dialog question, the ndcg-agreement set is defined as:
$$
\mathcal {N} _ {q} = \{c | \exists M \in \mathcal {M}; c \in \mathrm {T} (N, \rho_ {n} ^ {n}, q) \cap \mathrm {T} (M, \rho_ {m} ^ {n}, q) \}, \tag {6}
$$
where  $\rho_n^n,\rho_m^n\in \mathbb{R}$  are hyperparameters that indicate relevancy to NDCG and MRR, respectively. I.e., as  $\rho_{nn}$  increases, we may include more relevant candidates according to the NDCG model.
Up until this stage we have built a minimal set  $C_{\mathrm{MRR},q}$  that is likely to hold the human-derived answer. In the following we describe how we rank this set.
3.2.4 MRR ranking
Let  $r_{M_i,c,q}$  denote the rank according to  $M_{i}\in \mathcal{M}$  of candidate  $c$  for a question  $q$ . We compute the MRR rank of candidate  $c\in \mathcal{C}_{\mathrm{MRR},q}$  via geometric mean:  $r_{\mathrm{MRR},c,q} = \prod_{i = 1}^{n_m}r_{M_i,c,q}$ .
Model	MRR↑	R@1↑	R@5↑	R@10↑	Mean↓	NDCG↑
NMN (Kottur et al., 2018)	58.80	44.15	76.88	86.88	4.81	58.10
NN (Zheng et al., 2019)	61.37	47.33	77.98	87.83	4.57	52.82
CorefNMN (Kottur et al., 2018)	61.50	47.55	78.10	88.80	4.40	54.70
RvA (Niu et al., 2019)	63.03	49.03	80.40	89.83	4.18	55.59
HACAN (Yang et al., 2019)	64.22	50.88	80.63	89.45	4.20	57.17
MReal - BDAI† (Qi et al., 2020)	52.62	40.03	68.85	79.15	6.76	74.02
ReDAN (Gan et al., 2019)	53.13	41.38	66.07	74.50	8.91	61.86
ReDAN+† (Gan et al., 2019)	53.74	42.45	64.68	75.68	6.64	64.47
DualVD (Jiang et al., 2020)	63.23	49.25	80.23	89.70	4.11	56.32
DL-61 (Guo et al., 2019)	62.20	47.90	80.43	89.95	4.17	57.32
DL-61† (Guo et al., 2019)	63.42	49.30	80.77	90.68	3.97	57.88
DAN (Kang et al., 2019)	63.20	49.63	79.75	89.35	4.30	57.59
DAN† (Kang et al., 2019)	64.92	51.28	81.60	90.88	3.92	59.36
LTMI (Nguyen et al., 2020)	64.20	50.20	80.68	90.35	4.05	59.03
LTMI‡ (Nguyen et al., 2020)	52.14	38.93	66.60	80.65	6.53	74.88
FGA (Schwartz et al., 2019b)	63.75	49.58	80.97	88.55	4.51	52.12
5×FGA† (Schwartz et al., 2019b)	69.37	55.65	86.73	94.05	3.14	57.29
LS(CE)*‡ (Murahari et al., 2020)	50.74	37.95	64.13	80.00	6.28	74.47
LS(CE+NSP)*‡ (Murahari et al., 2020)	63.92	50.78	79.53	89.60	4.28	68.08
LS* (Murahari et al., 2020)	67.50	53.85	84.68	93.25	3.32	63.87
VD-BERT*†‡ (Wang et al., 2020)	51.17	38.90	62.82	77.98	6.69	75.35
5xFGA + LS*†	71.24	58.28	87.55	94.45	2.96	64.04
5xFGA + LS + LS(CE)*†‡	68.78	55.72	85.02	93.55	3.26	69.22
Ours*†‡	70.41	58.18	83.85	90.83	3.66	72.16
Visual Dialog Challenge 2020 Leaderboard
LS	68.79	55.20	86.15	93.88	3.12	63.34
VD-BERT	51.84	39.91	63.45	78.56	6.57	75.92
MReaL Lab (3rdPlace)	64.12	50.81	80.03	90.92	3.83	75.70
SES-100M (2ndPlace)	63.84	55.62	72.20	83.70	5.84	75.86
Ours (1stPlace)	70.42	58.59	82.85	88.84	3.96	73.35
Table 1: Performance on VisDial v1.0 test-standard. (*) denotes the use of external knowledge. (†) indicates ensemble model, and (‡) signifies fine-tuning on the dense annotations. Shown are the MRR, NDCG, the mean rank of the human-derived answer, and the recall at a certain number of retrievals.
\( \mathcal{H} \)	\( \mathcal{T} \)	\( \mathcal{N} \)	\( \mathrm{{MRR}} \dagger \)	R@1↑	R@5↑	R@10↑	Mean↓	\( \mathrm{{NDCG}} \uparrow \)	\( \left| {\mathrm{C}}_{\mathrm{{MRR}}}\right| \)
✓	✘	✘	70.83	58.87	84.32	90.67	3.67	74.32	2.34
✘	✓	✘	68.63	59.12	79.08	88.53	4.15	74.31	1.12
✘	✘	✓	61.75	51.74	70.35	85.88	4.94	74.69	3.97
✘	✓	✓	69.21	59.17	78.68	88.53	4.11	74.33	4.27
✓	✘	✓	71.15	59.11	84.38	90.67	3.65	73.29	4.87
✓	✓	✘	71.06	59.15	84.49	90.78	3.64	72.98	2.39
✓	✓	✓	71.26	59.18	84.62	90.78	3.62	73.23	4.91
LS(CE)	52.21	39.92	65.04	80.62	6.16	75.24	-
LS	69.00	55.80	85.36	93.13	3.35	64.89	-
5xFGA	69.38	56.17	86.15	92.95	3.32	58.68	-
5xFGA + LS	72.25	59.20	88.55	94.52	2.84	65.34	-
5xFGA + LS + LS(CE)	69.14	56.79	84.24	92.37	3.43	73.78	-
Table 2: MRR candidates set ablation analysis. Performance reported on the VisDialv1.0 val set.
3.3 NDCG step
In this step, we rank the remaining candidates  $\mathcal{C}_{\mathrm{NDCG},q} = \mathcal{C}_q\setminus \mathcal{C}_{\mathrm{MRR},q}$ . We assume the correct MRR answer is in  $\mathcal{C}_{\mathrm{MRR}}$ . Thus, we rank the remaining candidates, according to the NDCG model via geometric mean:  $r_{\mathrm{NDCG},c,q} = (r_{N,c,q})^p\cdot r_{M,c,q}$  where  $M\in \mathcal{M}$  is the most accurate MRR model, and  $p\in \mathbb{R}$  is a calibration hyperparameter which controls the trade-off between MRR and NDCG.
To conclude, let  $q$  be a dialog question and  $\mathcal{C}_q$  the corresponding candidates. We first find  $\mathcal{C}_{\mathrm{MRR},q}$ , and rank the set according to  $r_{\mathrm{MRR},c,q}$ . We then rank the remaining candidates, according to  $r_{\mathrm{NDCG},c,q}$ .
We show our results on the VisDial v1.0 dataset, where 123,287 images are used for training, 2,000 images for validation, and 8,000 images for testing (Das et al., 2017). Each image is associated with ten questions, and each question has 100 cor
Figure 2: Performance of a naive score ensemble of the MRR model and the NDCG model on the VisDialv1.0 val set. We calibrate the importance of each model with a scalar  $\alpha$ .
responding answer candidates. We use two MRR models (i.e.,  $n_m = 2$ ), FGA (Schwartz et al., 2019b) and an ensemble of LS (Murahari et al., 2020) with FGA. We use LS(CE) as the NDCG model. We set  $\rho_h = 3$ ,  $\rho_t = 1$ ,  $\rho_n^n = 5$ ,  $\rho_m^n = 10$ , and  $p = 3$ . We tune these parameters using the validation set.
Comparison to state-of-the-art: In Tab. 1 we compare our method to naive ensembles and previous baselines. We first ensemble the LS's output with the FGA's output. By combining them, we achieve the new MRR state-of-the-art (71.24% vs. 69.37%). Next, we build a naive ensemble of the MRR model and the NDCG model. We do so by adding the MRR ensemble scores (denoted by  $S_{M}$ ) and LS(CE) scores (denoted by  $S_{N}$ ), as follows:  $\alpha \cdot S_{M} + (1 - \alpha)S_{N}$ , where  $\alpha \in \mathbb{R}$  calibrates the trade-off between MRR and NDCG performance. We show in Fig. 2 an analysis of different  $\alpha$  values on the validation set. In Tab. 1, we report results for  $\alpha = 0.8$ . Our two-step method outperforms the MRR (70.41% vs. 68.78%) and NDCG (72.16% vs. 69.22%) metrics, despite lacking the output scores and only requiring rankings.
We also compare our approach to previous baselines. Most methods use the sparse annotations, i.e., the human-derived answer, while MReal-BDAI, VD-BERT, and LS(CE) finetune using the dense annotations. Finetuning with the dense annotations tremendously boosts the NDCG performance but loses MRR performance. The MRR performance decline can be attributed to NDCG being biased toward uncertain answers. We also note that LS leverages large-scale image-text corpora. LS(CE+NSP) optimizes both the dense and sparse annotations but still suffers from a performance drop compared to metric-dedicated LS models, i.e., MRR (63.92% vs. 67.50%) and NDCG (68.08% vs. 74.47%). Unlike the method mentioned above, our method re-rank the candidates based on two distinct models, with
Figure 3: MRR and NDCG scores for different hyperparameter values.
two distinct steps, to keep the human-derived answer high. In doing so, we achieve a good MRR performance (70.41% vs. 71.24%), yet notably with limited NDCG drop (72.25% vs. 75.35%). This property comes in handy in the recent Visual Dialog challenge, where the winners were picked based on both the NDCG and MRR evaluation metrics. Our method performs well on both metrics simultaneously and won the challenge.
Ablation analysis: The MRR candidate set consists of different subsets. In Tab. 2 we show the influence of each of subset independently on the retrieval metrics. Further, omitting a subset harms the performance, i.e. each component is essential to preserve both the MRR and NDCG metrics. We also report the average size of the MRR-candidate set, and the validation performance of the MRR model (i.e., 5xFGA) and the NDCG model (i.e., LS(CE)). In addition we provide the results of the MRR ensemble, and the naïve NDCG and MRR ensemble for  $\alpha = 0.8$ .
In Fig. 3, we examine how the NDCG and MRR metrics are affected by modifying one hyperparameter while maintaining the others. On the first figure from the left, we alter  $\rho_{c}$ . The higher  $\rho_{c}$ , we require higher agreement between the MRR models, resulting in higher certainty for elements in the MRR set. Because the MRR models are responsible for the MRR set ranking, an MRR set that is too large hurts the NDCG metric. For the same reason, in the second image from the left, increasing  $\rho_{t}$ , significantly harms the NDCG performance. In the third figure from the left, we show that considering more candidates that both NDCG and MRR models agree upon (i.e., increasing  $\rho_{n}^{n}$ ) helps both metrics' performance. However, adding too many candidates harms the NDCG metric. In the fourth image from the left, we show that the performance remains stable when  $\rho_{m}^{n}$  is larger than three. Last, on the fifth image from the left, we show the effect of changing  $p$ , which calibrates the trade-off between MRR and NDCG during the NDCG ranking step.
Qualitative analysis: In Fig. 4, we show two sample visual dialogs from test-standard. For each sample,
Figure 4: An illustration of two visual dialog samples. Each sample includes the MRR candidate set and four answers from the remaining NDCG candidates. We find that the MRR candidate set has more certain answers. We colorize the high-certainty candidates  $(\mathcal{H})$  with orange, the NDCG-agreement candidates  $(\mathcal{N})$  with purple, and the top-answers subset  $(\mathcal{T})$  with red. Note, if a candidate belongs to more than one set, we sketch the colors in the following order: orange→red→purple.
we provide the ranked MRR candidate set and the next 4 NDCG candidates. The analysis reveals the answers' ambiguity and that the MRR candidate set mostly consists of certain responses. In addition, we highlight the candidates within each MRR candidate subset with different colors. Additional samples can be found in the appendix.
We describe a non-parametric method to ensemble the candidate ranks of two strong MRR and NDCG models into a single ranking that excels on both NDCG and MRR. Intuitively, we use the MRR-model for non-ambiguous questions with certain answers. The dense-annotations cue is more applicable in ambiguous questions than the sparse annotations. Thus, in the case of low certainty, our method relies almost entirely on the NDCG model. We hope the proposed principles can guide the community towards a parametric model that can employ answers' semantics to measure certainty.
We thank Yftah Ziser, Itai Gat, Alexander Schwing and Tamir Hazan for useful discussions.