In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.

Ex Input:
Experiments ::: Baselines
For comparison, we select several public models as baselines including semantic parsing models: BiDAF BIBREF3, an MRC model which utilizes a bi-directional attention flow network to encode the question and passage;

QANet BIBREF12, which utilizes convolutions and self-attentions as the building blocks of encoders to represent the question and passage;

BERT BIBREF23, a pre-trained bidirectional Transformer-based language model which achieves state-of-the-art performance on lots of public MRC datasets recently;

and numerical MRC models: NAQANet BIBREF6, a numerical version of QANet model.

NAQANet+, an enhanced version of NAQANet implemented by ourselves, which further considers real number (e.g. “2.5”), richer arithmetic expression, data augmentation, etc. Syn Dep BIBREF6, the neural semantic parsing model (KDG) BIBREF22 with Stanford dependencies based sentence representations;

OpenIE BIBREF6, KDG with open information extraction based sentence representations;

SRL BIBREF6, KDG with semantic role labeling based sentence representations;

and traditional MRC models: 
 Question: what are the existing models they compared with?

Ex Output:
Syn Dep OpenIE SRL BiDAF QANet BERT NAQANet NAQANet+


Ex Input:
Tests and Gold-Standard Data for Intrinsic Evaluation
Using the gold standard data (described below), we performed three types of tests:

Class Membership Tests: embeddings corresponding two member of the same semantic class (e.g. “Months of the Year", “Portuguese Cities", “Smileys") should be close, since they are supposed to be found in mostly the same contexts.

Class Distinction Test: this is the reciprocal of the previous Class Membership test. Embeddings of elements of different classes should be different, since words of different classes ere expected to be found in significantly different contexts.

Word Equivalence Test: embeddings corresponding to synonyms, antonyms, abbreviations (e.g. “porque" abbreviated by “pq") and partial references (e.g. “slb and benfica") should be almost equal, since both alternatives are supposed to be used be interchangeable in all contexts (either maintaining or inverting the meaning).

Therefore, in our tests, two words are considered:

distinct if the cosine of the corresponding embeddings is lower than 0.70 (or 0.80).

to belong to the same class if the cosine of their embeddings is higher than 0.70 (or 0.80).

equivalent if the cosine of the embeddings is higher that 0.85 (or 0.95). 
 Question: What intrinsic evaluation metrics are used?

Ex Output:
Class Membership Tests Class Distinction Test Word Equivalence Test


Ex Input:
All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively. SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times. On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts. Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes.

All corpora show similar average answer candidate lengths ( INLINEFORM0 ), except for InfoboxQA where each line in the infobox is considered a candidate. SelQA and SQuAD show similar average question lengths ( INLINEFORM1 ) because of the similarity between their annotation schemes. It is not surprising that WikiQA's average question length is the smallest, considering their questions are taken from search queries. InfoboxQA's average question length is relatively small, due to the restricted information that can be asked from the infoboxes. InfoboxQA and WikiQA show the least question-answer word overlaps over questions and answers ( INLINEFORM2 and INLINEFORM3 in Table TABREF2 ), respectively. In terms of the F1-score for overlapping words ( INLINEFORM4 ), SQuAD gives the least portion of overlaps between question-answer pairs although WikiQA comes very close. 
 Question: How do they analyze contextual similaries across datasets?

Ex Output:

They compare the tasks that the datasets are suitable for, average number of answer candidates per question, number of token types, average answer candidate lengths, average question lengths, question-answer word overlap.