TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
PROBLEM: We perform a direct comparison of expert and crowd contributors, for 1000 term groups based on the number of total annotations(200 term groups with 2 total annotations, 200 term groups with 3 total annotations, and so on up to term groups with 6 total annotations).  Evaluators were given a summary of the annotations received for the term group in the form of:The term group "inequality inequity" received annotations as 50.0% sadness, 33.33% disgust, 16.67% anger. Then, they were asked to evaluate on a scale from 1 to 5, how valid these annotations were considered. 
 Question: How do they compare lexicons?

SOLUTION: Human evaluators were asked to evaluate on a scale from 1 to 5 the validity of the lexicon annotations made by the experts and crowd contributors.

PROBLEM: Our novelties include:

Using self-play learning for the neural response ranker (described in detail below).

Optimizing neural models for specific metrics (e.g. diversity, coherence) in our ensemble setup.

Training a separate dialog model for each user, personalizing our socialbot and making it more consistent.

Using a response classification predictor and a response classifier to predict and control aspects of responses such as sentiment, topic, offensiveness, diversity etc.

Using a model predictor to predict the best responding model, before the response candidates are generated, reducing computational expenses.

Using our entropy-based filtering technique to filter all dialog datasets, obtaining higher quality training data BIBREF3.

Building big, pre-trained, hierarchical BERT and GPT dialog models BIBREF6, BIBREF7, BIBREF8.

Constantly monitoring the user input through our automatic metrics, ensuring that the user stays engaged. 
 Question: What is novel in author's approach?

SOLUTION: They use self-play learning , optimize the model for specific metrics, train separate models per user, use model  and response classification predictors, and filter the dataset to obtain higher quality training data.

PROBLEM: The observed performance of both the BasicDCGAN and MultitaskDCGAN using 3-classes is comparable to the state-of-the-art, with 49.80% compared to 49.99% reported in BIBREF16  
 Question: What model achieves state of the art performance on this task?

SOLUTION:
BIBREF16