TASK DEFINITION: In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.
PROBLEM: The graphs in figure 1 show the distances between duplicate and non-duplicate questions using different embedding systems. Large achieved a F1 score of 0.71 for this task without any specific training, simply by choosing a threshold below which all sentence pairs are considered duplicates. In order to test whether these results generalised to our domain, we devised a test that would make use of what little data we had to evaluate.

SOLUTION: How is the accuracy of the system measured?

PROBLEM: The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set.

SOLUTION: What is the difference between the full test set and the hard test set?

PROBLEM:  We draw our data from news publications, wine reviews, and Reddit, which in addition to large volume, also let us characterize binomials in new ways, and analyze differences in binomial orderings across communities and over time. 

SOLUTION:
What online text resources are used to test binomial lists?