In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.

Our novelties include:

Using self-play learning for the neural response ranker (described in detail below).

Optimizing neural models for specific metrics (e.g. diversity, coherence) in our ensemble setup.

Training a separate dialog model for each user, personalizing our socialbot and making it more consistent.

Using a response classification predictor and a response classifier to predict and control aspects of responses such as sentiment, topic, offensiveness, diversity etc.

Using a model predictor to predict the best responding model, before the response candidates are generated, reducing computational expenses.

Using our entropy-based filtering technique to filter all dialog datasets, obtaining higher quality training data BIBREF3.

Building big, pre-trained, hierarchical BERT and GPT dialog models BIBREF6, BIBREF7, BIBREF8.

Constantly monitoring the user input through our automatic metrics, ensuring that the user stays engaged.
What is novel in author's approach?

Methods ::: Baselines ::: BiGRU s with attention:
This is very similar to Sum-QE but now $\mathcal {E}$ is a stack of BiGRU s with self-attention BIBREF21, instead of a BERT instance. The final summary representation ($h$) is the sum of the resulting context-aware token embeddings ($h = \sum _i a_i h_i$) weighted by their self-attention scores ($a_i$). We again have three flavors: one single-task (BiGRU-ATT-S-1) and two multi-task (BiGRU-ATT-M-1 and BiGRU-ATT-M-5).

Methods ::: Baselines ::: ROUGE:
This baseline is the ROUGE version that performs best on each dataset, among the versions considered by BIBREF13. Although ROUGE focuses on surface similarities between peer and reference summaries, we would expect properties like grammaticality, referential clarity and coherence to be captured to some extent by ROUGE versions based on long $n$-grams or longest common subsequences.

Methods ::: Baselines ::: Language model (LM):
For a peer summary, a reasonable estimate of $\mathcal {Q}1$ (Grammaticality) is the perplexity returned by a pre-trained language model. We experiment with the pre-trained GPT-2 model BIBREF22, and with the probability estimates that BERT can produce for each token when the token is treated as masked (BERT-FR-LM). Given that the grammaticality of a summary can be corrupted by just a few bad tokens, we compute the perplexity by considering only the $k$ worst (lowest LM probability) tokens of the peer summary, where $k$ is a tuned hyper-parameter.

Methods ::: Baselines ::: Next sentence prediction:
BERT training relies on two tasks: predicting masked tokens and next sentence prediction. The latter seems to be aligned with the definitions of $\mathcal {Q}3$ (Referential Clarity), $\mathcal {Q}4$ (Focus) and $\mathcal {Q}5$ (Structure & Coherence). Intuitively, when a sentence follows another with high probability, it should involve clear referential expressions and preserve the focus and local coherence of the text. We, therefore, use a pre-trained BERT model (BERT-FR-NS) to calculate the sentence-level perplexity of each summary:

where $p(s_i|s_{i-1})$ is the probability that BERT assigns to the sequence of sentences $\left< s_{i-1}, s \right>$, and $n$ is the number of sentences in the peer summary.
What simpler models do they look at?

We evaluate our method on two widely-studied datasets provided by Waseem and Hovey BIBREF5 and Davidson et al. BIBREF9.

Which publicly available datasets are used?