In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.

For the comparison we chose the best text model for each representation. As expected we obtain the largest improvement ($22-26\%$ E.R) when text-based unsupervised models are combined with image representations.
How much better is inference that has addition of image representation compared to text-only representations? 

We are in line with recent work BIBREF16 , proposing to shift evaluation from absolute values to more exploratory evaluations focusing on weaknesses and strengths of the embeddings and not so much in generic scores. For example, one metric could consist in checking whether for any given word, all words that are known to belong to the same class are closer than any words belonging to different classes, independently of the actual cosine.
What new metrics are suggested to track progress?

To verify our assumption that target encoding and orthogonal regularization help to boost the diversity of generated sequences, we use two metrics, one quantitative and one qualitative, to measure diversity of generation. First, we simply calculate the average unique predictions produced by both INLINEFORM0 and INLINEFORM1 in experiments shown in Section SECREF36 .  In this example there are 29 ground truth phrases. Neither of the models is able to generate all of the keyphrases, but it is obvious that the predictions from INLINEFORM0 all start with “test”, while predictions from INLINEFORM1 are diverse.

How is keyphrase diversity measured?