Definition: In this task, you will be presented with a context from an academic paper and a question based on the context. You have to classify the questions into "Extractive", "Abstractive", or "Yes-no" questions. Extractive questions can be answered by concatenating extracts taken from a context into a summary while answering abstractive questions involves paraphrasing the context using novel sentences. Yes-no question is a question whose expected answer is one of two choices, one that affirms the question and one that denies the question. Typically, the choices are either yes or no.

Positive Example 1 -
Input: We build a dataset of Twitter accounts based on two lists annotated in previous works. For the non-factual accounts, we rely on a list of 180 Twitter accounts from BIBREF1. On the other hand, for the factual accounts, we use a list with another 32 Twitter accounts from BIBREF19 that are considered trustworthy by independent third parties. 
 Question: How did they obtain the dataset?
Output: Extractive

Positive Example 2 -
Input: We evaluate our pointer-generator performance using BLEU score. The baseline language model is trained using RNNLM BIBREF23 . Perplexity measure is used in the evaluation.

 
 Question: Did they use other evaluation metrics?
Output: Yes-no

Negative Example 1 -
Input: For the English version, we performed both a thorough manual analysis and automatic evaluation across three commonly used TS datasets from two different domains  For the English version, we performed both a thorough manual analysis and automatic evaluation across three commonly used TS datasets from two different domains  The evaluation of the German version is in progress. 
 Question: What are the corpora used for the task?
Output: Yes-no

Negative Example 2 -
Input: For Turkish, as the first dataset, we utilised the movie reviews which are collected from a popular website. The number of reviews in this movie corpus is 20,244 and the average number of words in reviews is 39. Each of these reviews has a star-rating score which is indicative of sentiment. These polarity scores are between the values 0.5 and 5, at intervals of 0.5. We consider a review to be negative it the score is equal to or lower than 2.5. On the other hand, if it is equal to or higher than 4, it is assumed to be positive. We have randomly selected 7,020 negative and 7,020 positive reviews and processed only them. 
 Question: What details are given about the movie domain dataset?
Output: Extractive

Now complete the following example -
Input: The input of our model are the words in the input text $x[1], ... , x[n]$ and query $q[1], ... , q[n]$ . We concatenate pre-trained word embeddings from GloVe BIBREF40 and character embeddings trained by CharCNN BIBREF41 to represent input words. The $2d$ -dimension embedding vectors of input text $x_1, ... , x_n$ and query $q_1, ... , q_n$ are then fed into a Highway Layer BIBREF42 to improve the capability of word embeddings and character embeddings as

$$\begin{split} g_t &= {\rm sigmoid}(W_gx_t+b_g) \\ s_t &= {\rm relu } (W_xx_t+b_x) \\ u_t &= g_t \odot s_t + (1 - g_t) \odot x_t~. \end{split}$$ (Eq. 18) The same Highway Layer is applied to $q_t$ and produces $v_t$ . Next, $u_t$ and $v_t$ are fed into a Bi-Directional Long Short-Term Memory Network (BiLSTM) BIBREF44 respectively in order to model the temporal interactions between sequence words: Then we feed $\mathbf {U}$ and $\mathbf {V}$ into the attention flow layer BIBREF27 to model the interactions between the input text and query. Therefore, we introduce Self-Matching Layer BIBREF29 in our model as

$$\begin{split} o_t &= {\rm BiLSTM}(o_{t-1}, [h_t, c_t]) \\ s_j^t &= w^T {\rm tanh}(W_hh_j+\tilde{W_h}h_t)\\ \alpha _i^t &= {\rm exp}(s_i^t)/\Sigma _{j=1}^n{\rm exp}(s_j^t)\\ c_t &= \Sigma _{i=1}^n\alpha _i^th_i ~. \end{split}$$ (Eq. 20) Finally we feed the embeddings $\mathbf {O} = [o_1, ... , o_n]$ into a Pointer Network BIBREF39 to decode the answer sequence as

$$\begin{split} p_t &= {\rm LSTM}(p_{t-1}, c_t) \\ s_j^t &= w^T {\rm tanh}(W_oo_j+W_pp_{t-1})\\ \beta _i^t &= {\rm exp}(s_i^t)/\Sigma _{j=1}^n{\rm exp}(s_j^t)\\ c_t &= \Sigma _{i=1}^n\beta _i^to_i~. \end{split}$$ (Eq. 21) Therefore, the probability of generating the answer sequence $\textbf {a}$ is as follows

$${\rm P}(\textbf {a}|\mathbf {O}) = \prod _t {\rm P}(a^t | a^1, ... , a^{t-1}, \mathbf {O})~.$$ (Eq. 23) 
 Question: What QA models were used?
Output: 


Answer: Abstractive