TASK DEFINITION: In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.
PROBLEM: The graphs in figure 1 show the distances between duplicate and non-duplicate questions using different embedding systems. Large achieved a F1 score of 0.71 for this task without any specific training, simply by choosing a threshold below which all sentence pairs are considered duplicates. In order to test whether these results generalised to our domain, we devised a test that would make use of what little data we had to evaluate.

SOLUTION: How is the accuracy of the system measured?

PROBLEM: Finally, the decoder task, which predicts the target sequence probability at time INLINEFORM3 based on previous output and context information INLINEFORM4 , can be formulated as: DISPLAYFORM0

SOLUTION: How does their decoder generate text?

PROBLEM: For evaluation, the Visual Dialog task employs four metrics. NDCG is the primary metric of the Visual Dialog Challenge which considers multiple similar answers as correct ones. The other three are MRR, recall@k, and mean rank where they only consider the rank of a single answer.

SOLUTION:
What metrics are used in challenge?