TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question based on the context. You have to classify the questions into "Extractive", "Abstractive", or "Yes-no" questions. Extractive questions can be answered by concatenating extracts taken from a context into a summary while answering abstractive questions involves paraphrasing the context using novel sentences. Yes-no question is a question whose expected answer is one of two choices, one that affirms the question and one that denies the question. Typically, the choices are either yes or no.
PROBLEM: To counter that, we use a left-to-right attention mask, similar to the one employed in the original Transformer decoder BIBREF1. For the input tokens in $X$, we apply such mask to all the target tokens $Y$ that were concatenated to $X$, so that input tokens can only attend to the other input tokens. Conversely, for target tokens $y_t$, we put an attention mask on all tokens $y_{>t}$, allowing target tokens $y_t$ to attend only to the input tokens and the already generated target tokens. 
 Question: What is different in BERT-gen from standard BERT?

SOLUTION: Abstractive

PROBLEM:   In a second step, we will train a neural monolingual translation system, that translates from the output of the PBMT system INLINEFORM0 to a better target sentence INLINEFORM1 . 
 Question: Do they train the NMT model on PBMT outputs?

SOLUTION: Yes-no

PROBLEM: In HotpotQA, on average we can find 6 candidate chains (2-hop) in a instance, and the human labeled true reasoning chain is unique. A predicted chain is correct if the chain only contains all supporting passages (exact match of passages).

In MedHop, on average we can find 30 candidate chains (3-hop). For each candidate chain our human annotators labeled whether it is correct or not, and the correct reasoning chain is not unique. A predicted chain is correct if it is one of the chains that human labeled as correct.

The accuracy is defined as the ratio: The accuracy is defined as the ratio: 
 Question: What benchmarks are created?

SOLUTION:
Abstractive