Task: Question Generation
Input (English): One of the most popular metrics for QG, BLEU BIBREF21 provides a set of measures to compare automatically generated texts against one or more references. In particular, BLEU-N is based on the count of overlapping n-grams between the candidate and its corresponding reference(s). Therefore, we adopt Self-BLEU, originally proposed by BIBREF33, as a measure of diversity for the generated text sequences.  Therefore, given a question-context pair as input to a QA model, two type of metrics can be computed:

n-gram based score: measuring the average overlap between the retrieved answer and the ground truth.

probability score: the confidence of the QA model for its retrieved answer; this corresponds to the probability of being the correct answer assigned by the QA model to the retrieved answer.
Output (English): What automated metrics authors investigate?