In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.

Input: Consider Input: Our video question answering task is novel and to our knowledge, no model has been designed specifically for this task. As a first step towards solving this problem, we evaluated the performance of state-of-the-art models developed for other QA tasks, including a sentence-level prediction task and two segment retrieval tasks. Baselines ::: Baseline1: Sentence-level prediction
Given a transcript (a sequence of sentences) and a question, Baseline1 predicts (starting sentence index, ending sentence index). The model is based on RaSor BIBREF13, which has been developed for the SQuAD QA task BIBREF6. RaSor concatenates the embedding vectors of the starting and the ending words to represent a span. Following this idea, Baseline1 represents a span of sentences by concatenating the vectors of the starting and ending sentences. The left diagram in Fig. FIGREF15 illustrates the Baseline1 model.

Model. The model takes two inputs, a transcript, $\lbrace s_1, s_2, ... s_n\rbrace $ where $s_i$ are individual sentences and a question, $q$. The output is the span scores, $y$, the scores over all possible spans. GLoVe BIBREF14 is used for the word representations in the transcript and the questions. We use two bi-LSTMs BIBREF15 to encode the transcript.

where n is the number of sentences . The output of Passage-level Encoding, $p$, is a sequence of vector, $p_i$, which represents the latent meaning of each sentence. Then, the model combines each pair of sentence embeddings ($p_i$, $p_j$) to generate a span embedding.

where [$\cdot $,$\cdot $] indicates the concatenation. Finally, we use a one-layer feed forward network to compute a score between each span and a question.

In training, we use cross-entropy as an objective function. In testing, the span with the highest score is picked as an answer. Baselines ::: Baseline2: Segment retrieval
We also considered a simpler task by casting our problem as a retrieval task. Specifically, in addition to a plain transcript, we also provided the model with the segmentation information which was created during the data collection phrase (See Section. SECREF3). Note that each segments corresponds to a candidate answer. Then, the task is to pick the best segment for given a query. This task is easier than Baseline1's task in that the segmentation information is provided to the model. Unlike Baseline1, however, it is unable to return an answer span at various granularities. Baseline2 is based on the attentive LSTM BIBREF17, which has been developed for the InsuranceQA task. The right diagram in Fig. FIGREF15 illustrates the Baseline2 model.

Model. The two inputs, $s$ and $q$ represent the segment text and a question. The model first encodes the two inputs.

$h^s$ is then re-weighted using attention weights.

where $\odot $ denotes the element-wise multiplication operation. The final score is computed using a one-layer feed-forward network.

During training, the model requires negative samples. For each positive example, (question, ground-truth segment), all the other segments in the same transcript are used as negative samples. Cross entropy is used as an objective function.

 Baselines ::: Baseline3: Pipeline Segment retrieval
We construct a pipelined approach through another segment retrieval task, calculating the cosine similarities between the segment and question embeddings. In this task however, we want to test the accuracy of retrieving the segments given that we first retrieve the correct video from our 76 videos. First, we generate the TF-IDF embeddings for the whole video transcripts and questions. The next step involves retrieving the videos which have the lowest cosine distance between the video transcripts and question. We then filter and store the top ten videos, reducing the number of computations required in the next step. Finally, we calculate the cosine distances between the question and the segments which belong to the filtered top 10 videos, marking it as correct if found in these videos. While the task is less computationally expensive than the previous baseline, we do not learn the segment representations, as this task is a simple retrieval task based on TF-IDF embeddings.

Model. The first two inputs are are the question, q, and video transcript, v, encoded by their TF-IDF vectors: BIBREF18:

We then filter the top 10 video transcripts(out of 76) with the minimum cosine distance, and further compute the TF-IDF vectors for their segments, Stop10n, where n = 10. We repeat the process for the corresponding segments:

selecting the segment with the minimal cosine distance distance to the query. 
 Question: What baseline algorithms were presented?

Output: a sentence-level prediction algorithm, a segment retrieval algorithm and a pipeline segment retrieval algorithm


Input: Consider Input: To create minimal pairs exemplifying a wide array of linguistic contrasts, it is necessary to artificially generate all datasets. This ensures both that we have sufficient unacceptable examples, and that the data is fully controlled, allowing for repeated isolation of a single linguistic phenomenon in each paradigm BIBREF30. The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences. Examples SECREF6 and SECREF6 show one such template for the `acceptable' and `unacceptable' sentences within a pair: the sole difference between them is the underlined word, which differs only in whether the anaphor agrees in number with its antecedent. Our generation codebase and scripts are freely available. This generation procedure is not without limitations, and despite the very detailed vocabulary we use, implausible sentences are occasionally generated (e.g., `Sam ran around some glaciers'). In these cases, though, both the acceptable and unacceptable sentences will be equally implausible given world knowledge, so any difference in the probability assigned to them is still due to the intended grammatical contrast. 
 Question: How is the data automatically generated?

Output:  The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences.


Input: Consider Input: Results reveal that, contrary to the common belief, correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization. Furthermore, we show a large variance of correlations between different Rouge variants and the manual evaluations which further makes the reliability of Rouge for evaluating scientific summaries less clear. 
 Question: What is the common belief that this paper refutes? (c.f. 'contrary to the common belief, ROUGE is not much [sic] reliable'


Output: correlations between Rouge and the Pyramid scores are weak, which challenges its effectiveness for scientific summarization