Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Observing shortcomings of BLEU and ROUGE, the paper proposes, JAUNE, a set of criteria for a good evaluation metric. These criteria include: high correlation with human judgement; being able to distinguish similar but contradicting statements; penalizing grammatical errors, and hard to game.

The paper, as its current form, is not ready for publishing. Some suggestions and comments:

- Please carefully check the paper and fix typos and confusing sentences. I was collecting these errors but eventually stopped. Some examples. Sec. 2.3: punctuation missing between "RUSE" and "this method", comma missing after "a discrete space"; Sec. 4.1.1: "made to ,for example"....

- The motivation of the paper is unclear. Is your criticism only about BLEU and ROUGE, or the state of the arts in NLP evaluation in general? To make JAUNE appealing, one has to argue that the state of the arts in NLP evaluation is ineffective. For this, the paper needs to review a boarder range of metrics beyond just BLUE and ROUGE. 

- While the authors suggest a data-driven metric, it reads to me like a model-driven metric (RoBERTAa specifically). Doesn't it systematically bias towards a certain family of metrics? 

- Better and more comprehensive experimental results are highly desired. 
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: 
=== Summary ===

The authors motivate the development of new (automatic) metrics to evaluate language generation by using similarity with a given reference: standard metrics like BLEU, ROUGE or METEOR have been shown to have poor correlation with human judgment on a number of tasks and are vulnerable to changes in word re-ordering, semantics-changing word replacement, and syntactic transformations.
  
They then propose a multi-dimensional evaluation criteria to evaluate sentence similarity based on semantic similarity (something that correlates with human judgments of the same), logical equivalence and fluency.
  
The paper then goes on to describe possible directions to tackle several key problems in evaluation: evaluating semantic similarity by using models trained on the GLUE benchmark, evaluating logical equivalence using models trained on the MNLI corpus and fluency based on the CoLA corpus.

=== Decision ===

The problem this paper seeks to tackle is clearly one of great
importance in the field, but I find it hard to argue that this paper
significantly contributes to the existing body of work (more on this
below) and as a result I vote to reject this paper.

There are two possible contributions for this paper: a set of criteria for what makes a good evaluation metric and the concrete proposals to implement these criteria.
  
For the first, I find the proposed criteria to be overly generic and not helpful at providing additional clarity on what makes for a good evaluation: for example, how is semantic similarity different from logical consistency? Does it make sense to compare the semantic similarity of two sentences if one of them isn't even near grammatical? A lot of prior work already argue the shortcomings of the existing metrics this paper is making, e.g. Conroy and Dang (2008), Liu et al. (2016), Novikova et al. (2017). I think it would be valuable to present new axes to decompose the evaluation problem, but more work is needed to clarify and develop the axes presented in this paper.
  
For the second possible contribution, the idea of evaluating language generation along dimensions is not novel and in fact quite standard in the NLP community. The challenge has been showing that there are subset of tasks that can be used a reliable metrics across different domains and systems. Unfortunately, this paper does not actually evaluate its own proposals, making it hard to evaluate how effective its proposals are.  

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper argues that BLEU and ROUGE, two metrics that are used for the evaluation of machine translation and text summarization systems, are flawed, and proposes a new JAUNE metric to replace it.

The authors train a regressor on the STS-B dataset, and show that their model (which is using sentence embeddings from RoBERTa) corresponds better to the ground truth similarity labels than then scaled (but otherwise unchanged) BLEU scores. This is probably not surprising, given the small size and specific nature of the STS-B task and dataset. 

I could agree with many of the problems that the authors describe, but the proposed solution seems to be a very specific solution that works on a given dataset (for which supervised training data is available), but I do not think it will generalize well to unseen test data in different domains. I also do not understand how the BLEU score can simply be rescaled from 0-5 - how do you determine the maximum BLEU score before rescaling?

The paper should be proofread by a native speaker, many sentences are unclear, and spacing as well as punctuation are used in weird ways.

review_title: Official Blind Review #2

Meta-review: The authors tackle the questions of automatic metrics for assessing document similarity and propose the use of Transformed-based language models as a critic providing scores to samples. As a note, ideas like these have been also adopted in Computer Vision with the use of the Inception score as a proxy the quality of generated images. The authors ask great questions in the paper and they clearly tackle a very important problem, that of automatic measures for assessing text quality. While their first indications are not negative, this paper lacks the rigor and depth of experiments of a conference paper that would convince the research community to abandon BLEU and ROUGE in lieu of some other metric. It's perhaps a good workshop paper or a short paper at a *CL conference. Specifically, we would need more tasks where BLEU/ROUGE is the standard measure and so how the proposed measure correlates better with humans,  so cases where word overlap is in theory a good proxy of similarity assuming reference sentence (e.g., logical entailment is not such a prototypical task). MT is a first step towards that, but summarization is also a necessary I would say. Other questions of interest relate to the type of LM (does it only need to be Roberta?) and the quality of LM (what if i badly tune my LM?)  On a more personal note: We all know that BLEU is not a good metric (especially for document-level judgements) and every now and then there have been proposals to replace BLEU that do correlate better (e.g., http://ccc.inaoep.mx/~villasen/bib/Regression%20for%20machine%20translation%20evaluation.pdf) . However, BLEU is still here due to each simplicity. Please keep pushing this research and I’m looking forward to seeing more experimental evidence.
Decision: Reject