TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
PROBLEM: I first measured the effectiveness of the embeddings at the skip-gram's training task, predicting context words INLINEFORM0 given input words INLINEFORM1 . This task measures the methods' performance for predictive language modeling. I used four datasets of sociopolitical, scientific, and literary interest: the corpus of NIPS articles from 1987 – 1999 ( INLINEFORM2 million), the U.S. presidential state of the Union addresses from 1790 – 2015 ( INLINEFORM3 ), the complete works of Shakespeare ( INLINEFORM4 ; this version did not contain the Sonnets), and the writings of black scholar and activist W.E.B. Du Bois, as digitized by Project Gutenberg ( INLINEFORM5 ). For each dataset, I held out 10,000 INLINEFORM6 pairs uniformly at random, where INLINEFORM7 , and aimed to predict INLINEFORM8 given INLINEFORM9 (and optionally, INLINEFORM10 ). Since there are a large number of classes, I treat this as a ranking problem, and report the mean reciprocal rank. The experiments were repeated and averaged over 5 train/test splits. 
 Question: What is MRR?

SOLUTION: mean reciprocal rank

PROBLEM: We evaluate the generations with the ROUGE BIBREF29 and METEOR BIBREF30 metrics using the true sentences as targets. 
 Question: What metrics are used for evaluation?

SOLUTION: ROUGE BIBREF29 and METEOR BIBREF30

PROBLEM: We measure how contextual a word representation is using three different metrics: self-similarity, intra-sentence similarity, and maximum explainable variance. 
 Question: What experiments are proposed to test that upper layers produce context-specific embeddings?

SOLUTION:
They measure self-similarity, intra-sentence similarity and maximum explainable variance of the embeddings in the upper layers.