TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
PROBLEM: Detection: In hypernymy detection, the task is to classify whether pairs of words are in a hypernymy relation. For this task, we evaluate all models on five benchmark datasets: First, we employ the noun-noun subset of bless, which contains hypernymy annotations for 200 concrete, mostly unambiguous nouns. Second, we evaluate on leds BIBREF13 , which consists of 2,770 noun pairs balanced between positive hypernymy examples, and randomly shuffled negative pairs. Direction: In direction prediction, the task is to identify which term is broader in a given pair of words. For this task, we evaluate all models on three datasets described by BIBREF16 : On bless, the task is to predict the direction for all 1337 positive pairs in the dataset. Pairs are only counted correct if the hypernymy direction scores higher than the reverse direction, i.e. INLINEFORM0 . We reserve 10% of the data for validation, and test on the remaining 90%. On wbless, we follow prior work BIBREF17 , BIBREF18 and perform 1000 random iterations in which 2% of the data is used as a validation set to learn a classification threshold, and test on the remainder of the data. We report average accuracy across all iterations. Finally, we evaluate on bibless BIBREF16 , a variant of wbless with hypernymy and hyponymy pairs explicitly annotated for their direction. Graded Entailment: In graded entailment, the task is to quantify the degree to which a hypernymy relation holds. For this task, we follow prior work BIBREF19 , BIBREF18 and use the noun part of hyperlex BIBREF20 , consisting of 2,163 noun pairs which are annotated to what degree INLINEFORM0 is-a INLINEFORM1 holds on a scale of INLINEFORM2 . 
 Question: Which benchmark datasets are used?

SOLUTION: noun-noun subset of bless leds BIBREF13 bless wbless bibless hyperlex BIBREF20

PROBLEM: Validated transcripts were sent to professional translators. In order to control the quality of the professional translations, we applied various sanity checks to the translations BIBREF11. We also sanity check the overlaps of train, development and test sets in terms of transcripts and voice clips (via MD5 file hashing), and confirm they are totally disjoint. 
 Question: How is the quality of the data empirically evaluated? 

SOLUTION: Validated transcripts were sent to professional translators. various sanity checks to the translations  sanity check the overlaps of train, development and test sets

PROBLEM: Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus. 
 Question: What type of evaluation is proposed for this task?

SOLUTION:
Answer with content missing: (Evaluation Metrics section) Precision, Recall, F1-scores, Strict match, METEOR, ROUGE-2