Task: Summarization
Input (English): We review the limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE:  a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.


Output (English): This paper proposes a new JAUNE metric for the evaluation of machine translation and text summarization systems, showing that their model corresponds better to ground truth similarity labels than BLEU.