In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.

[EX Q]: Span detector. We adopt a multi-turn answer module for the span detector BIBREF1 . Formally, at time step INLINEFORM0 in the range of INLINEFORM1 , the state is defined by INLINEFORM2 . The initial state INLINEFORM3 is the summary of the INLINEFORM4 : INLINEFORM5 , where INLINEFORM6 . Here, INLINEFORM7 is computed from the previous state INLINEFORM8 and memory INLINEFORM9 : INLINEFORM10 and INLINEFORM11 . Finally, a bilinear function is used to find the begin and end point of answer spans at each reasoning step INLINEFORM12 : DISPLAYFORM0 DISPLAYFORM1

The final prediction is the average of each time step: INLINEFORM0 . We randomly apply dropout on the step level in each time step during training, as done in BIBREF1 .
[EX A]: What is the architecture of the span detector?

[EX Q]: GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. We report the 12-category accuracy results for all models and human evaluation in Table TABREF14.
[EX A]: What is the performance of the models on the tasks?

[EX Q]: Detection: In hypernymy detection, the task is to classify whether pairs of words are in a hypernymy relation. For this task, we evaluate all models on five benchmark datasets: First, we employ the noun-noun subset of bless, which contains hypernymy annotations for 200 concrete, mostly unambiguous nouns. Second, we evaluate on leds BIBREF13 , which consists of 2,770 noun pairs balanced between positive hypernymy examples, and randomly shuffled negative pairs. Direction: In direction prediction, the task is to identify which term is broader in a given pair of words. For this task, we evaluate all models on three datasets described by BIBREF16 : On bless, the task is to predict the direction for all 1337 positive pairs in the dataset. Pairs are only counted correct if the hypernymy direction scores higher than the reverse direction, i.e. INLINEFORM0 . We reserve 10% of the data for validation, and test on the remaining 90%. On wbless, we follow prior work BIBREF17 , BIBREF18 and perform 1000 random iterations in which 2% of the data is used as a validation set to learn a classification threshold, and test on the remainder of the data. We report average accuracy across all iterations. Finally, we evaluate on bibless BIBREF16 , a variant of wbless with hypernymy and hyponymy pairs explicitly annotated for their direction. Graded Entailment: In graded entailment, the task is to quantify the degree to which a hypernymy relation holds. For this task, we follow prior work BIBREF19 , BIBREF18 and use the noun part of hyperlex BIBREF20 , consisting of 2,163 noun pairs which are annotated to what degree INLINEFORM0 is-a INLINEFORM1 holds on a scale of INLINEFORM2 .
[EX A]:
Which benchmark datasets are used?