JAUNE: Justified And Unified Neural language Evaluation
Authors: Hassan Kané
Yusuf Kocyigit
Ali Abdalla
Pelkins Ajanoh
Mohamed Coulibali
Venue:  (2020)
Keywords: NLP
Evaluation Metrics
Summarization
Translation
BLEU
ROUGE
Transformers

Abstract: We review the limitations of BLEU and ROUGE -- the most popular metrics used to assess reference summaries against hypothesis summaries, and introduce JAUNE:  a set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-based Language Models to assess reference summaries against hypothesis summaries.



Under review as a conference paper at ICLR 2020
JAUNE: JUSTIFIED AND UNIFIED NEURAL LAN- GUAGE EVALUATION
Anonymous authors Paper under double-blind review
ABSTRACT
We review the limitations of BLEU and ROUGE the most popular metries used to assess reference summaries against hypothesis summaries, and introduce JAUNE: set of criteria for what a good metric should behave like and propose concrete ways to use recent Transformers-aatt Language Models to assess ref- erence summaries/transltioon. against hypothesis summaries/translation..
INTRODUCTION
Evaluation metries play central role in the machine learning community. They direct research efforts and define the state of the art models. In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU (Papineni et al.] 2002) and ROUGE (Lin) 2004). Both approaches rely counting the matching n-grams in the candidate text to n-grams in the reference text. BLEU is precision focused while ROUGE is recall focused.
These metrics have posed serious limitations and have already been criticized by the academic com- munity <Reiter 2018) (Callison- Ballcs et al.) 2006) (Sulem et al. 2018) (Novikova al. 2017). In this work, we formulate an empirical criticism of BLEU and ROUGE, establish JAUNE: a set of criteria that a sound evaluation metric should pass. Furthermore recent advances in NLP to design data-driven metrics addressing the weaknesses found in BLEU and ROUGE while scoring high on the criteria for sound evaluation metric.
propose concrete ways to
2 RELATED WORK
2. BLEU, ROUGE AND N-GRAM MATCHING APPROACIES
BLEU (Bilingual Evaluation Understudy) Papineni et al., 2002) and ROUGE (Recall-Oriented Un- derstudy for Gisting Evaluation) (Lin 2004) have been used to evaluate many NLP tasks for almost two decades. The general acceptance of these methods depend on many factors including their sim- plicity and intuitive interpretability. Moreover, the main factor is the claim that they highly correlate with human judgement (Papineni et al.) 2002).
The shortcomings of these methods have been widely criticised and studied. Reiter (Reiter 2018). in his structured review of BLEU, finds low correlation between BLEU and human judgment. Callison et al (Callison- Burion et al.] 2006) examine BLEU in the context of machine translation and find that BLEU neither correlates with human judgment adequacy (whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor on fluency(the quality of language in sentence). Sulem et al (Sulem al. 2018) examine BLEU in the context of text

---

Under review as a conference paper at ICLR 2020
simplification grammaticality, meaning preservation and simplicity. They report very low, and, in some cases, negative correlation with human judgment.
Considering these results, it is natural step pursue avenues for natural language evaluation and, with the advent of deep learning, using neural networks for this task is promising step forward.
2.2 TRANSFORMERS, BERT AND GPT
Language modeling has become an important NLP technique, thanks to the ability to apply it to various NLP tasks as explained in Radford et al (Radford et al. 2019). There are two leading architectures for language modeling: Recurrent Neural Networks (MNNss et al. 2010) and Transformers /Vaswani et al.) (017). RNNS handle the input tokens, words or characters, one by one through time to learn the relationship between them, whereas, transformers receive a segment of tokens and learn the dependencies between them using attention mechanism.
2.3 MODEL-BASED METRICS
While BLEU and ROUGE are defined in discrete space, new evaluation metric can be defined in this continuous space. BERTscore (Zhang al. (019) uses word embeddings and cosine similarity to create score array and uses greedy matching to maximize the similarity score. Sentence Mover's Similarity (Clark et al. 2019) uses the mover similarity, Wasserstein distance, between sentence embedding generated from averaging the word embeddings in sentence.
Both of these methods report stronger correlations with human judgment and better results when compared to BLEU and ROUGE. While they are using word embeddings (Mikolov et al. 2013) to transfer their sentence a continuous space, they are still using distance metrics to evaluate that sentence. BLEND (Ma et al. 2017) uses an SVM combine different existing evaluation metries.
Another proposed evaluation method is RUSE (Shimanaka et al. 2018). This method embeds both sentences separately and pool them to given size. After, the method uses pre-trained MLP to predict on different tasks. This quality estimator metric then proposed to be used in language evaluation.
Our proposed methodology to take neural language evaluation beyond architecture specifications. We proposing framework in which evaluator's success can be determined.
2.4 GLUE BENCHMARK
The GLUE Benchmark is tool for evaluating and analyzing the performance of models across diverse range of existing NLU tasks (Wang et al. 2018). The recent introduction this benchmark has catalyzed the development of architectures scoring well wide variety of tasks and encour- aged the NLP community to move away from specialized models doing well on a single task to models performing well across benchmarks. The variety of tasks introduced in the GLUE Bench- mark are linguistic acceptability, sentiment analysis, semantic similarity, question answering, logical inference and reading comprehension. To be assessed according to that benchmark, models such as Transformers are usually pre-trained on a large corpus in an unsupervised manner and fine-tuned on dataset used for the specific task of the benchmark.
3 CHALLENGES WITH BLEU AND ROUGE
In this part, we will discuss the limitations of BLEU and ROUGE. There are simple ways to attack these n-gram based metrics like adding single word negation changing all possible words with synonyms. Although these are theoretically plausible scenarios we also wanted to analyze which cases forced these metrics to fail in real life.
We took 100 examples from the STS-B dataset CCer et al. 2017) where the absolute difference be- tween the BLEU/ROUGE score and normalized label was the biggest. This does not necessarily capture all failure cases of BLEU/ROUGE, but a variety failure cases can be observed. Through this analysis, we that there systematically recurring real life examples, just like in our theo-
2

---

Under review as a conference paper at ICLR 2020
retical examples, where BLEU and ROUGE are failing to assess the level of similarity between two sentences.
We also observe that though of the shortcomings of unigram metrics are mitigated through higher order n-grams, they open the door for different problems. Some of the most common failure cases that have encountered listed in table [
3.1 IDIOMS AND ADDING DETAILS
commonly encountered failure mode involves the use of idioms and the addition of extra exam- ples/details in one of the sentences. These types of errors are especially common in more natural conversations. These types of errors also made 25 of our analysis. Here, we characteristically see humans giving high scores to these sentences because they are aware of which part holds the core meaning of the sentence while BLEU/ROUGE lack this ability.
An example from the dataset is "You should take this animal to vet right away." and "As covered in the other answers, your only option is to see a in order to have surgery done." while the true score for this sentence pair is 3.6 out of 5, BLEU gives this sentence a 0.46 out 5f
3.2 CHANGING WORDS
Another common failure mode of BLEU and ROUGE occurs when few important words of a sentence is changed while the rest the structure kept the same.
An example from STS-B B is "a man is speaking." and "a man is spitting.". While human judges give these two sentences a similarity score of 0.64 out of 5, BLEU-1 gives this sentence a 3.75 out of 5.
3.3 GENERAL PARAPHRASE
We frequently see BLEU and ROUGE failing in the context of:
General paraphrases where words are replaced by synonyms. This accounts for 12 % of cases.
Sentences are reordered. This accounts for around 10% of observed errors.
Different verb tenses used. This accounts for 15% of observed errors.
While the higher order n-grams are supposed preserve the intelligibility of the sentence and not reward a model that outputs words in random order, they also punish valid re-orderings of sub sentences or words. In these smoothed methods, changing a word with a synonym will also result in much higher penalty.
BLEU and ROUGE are methods that are much more frequently under scoring sentence pairs than over scoring them. Similarly to Reiter (Reiter 2018) conclude that BLEU/ROUGE can be fruitful in deciding whether model is bad but not whether it is good.
We also provide detailed examples of these failure cases in the appendix.
3.4 EXPERIMENTS
3.4.1
SOME EXAMPLES FROM STS-B SENTENCES PAIRS
To illustrate our argument, will give some examples from the dataset with their BLEU/ROUGE scores as well as a score generated from a RoBERTa model fine tuned on the STS-B dataset. We refer to this model RoBERTa-STS. Note that in this paper the BLEU* and ROUGE* scores not between 0 and but are scaled with 5 to be more understandable with the seale of the scoring metric used in the dataset. That is why refer them BI BLEU* and ROUGE*
3

---

Under review as a conference paper at ICLR 2020
Table 1: BLEU*/ROUGE* and RoBERTa-STS scores on sentence pair examples from STS-B
Sentence pair
BLEU
ROUGE# 1.42
RoBERTa-STS
Label
The last time the survey was conducted, in 1995,
0.99
4.65/5
5.00/5
those numbers matched.
In 1995, the last survey, those numbers were equal.
band performing stage.
1.14
2.29
3.85/5
5.00/5
band is playing onstage.
Two white dogs are swimming in the water.
3.00
3.23
1.19/5
0.80/5
The birds are swimming in the water.
man plays the piano.
0.92
2.17
5.00/5
5.00/5
A man is playing a piano.
Pardon the brevity of this answer, but I would say "named" is preferred within the context of your
0.25
0.72
3.73/5
4.40/5
example.
Named preferred in your example, since you are formally giving name to your method.
Cald Lab01
Gold Laboi
RoBERTa Scores, Scaled to
BLEU Scores, Scaled to
(a)
(b)
Figure 1: Comparison of RoBERTa-STS scores(a) and BLEU scores(b) with labels from the STS-B dev set.
3.4.2 SEMANTIC SIMILARITY EXPERIMENTS
that development scores of RoBERTa-STS in figure la and BLEU* in figure lb compared to the gold labels. While we will look at the correlation scores, we can also see that the average errors in BLEU and RoBERTa-STS are remarkably different.
In figure
Table Correlation with human judgement of similarity on STS-B Benchmark development set
ROUGE
BLEU
RoBERTa-STS
Pearson correlation with human judgement
0.55
0.50
0.92
4

---

Under review as a conference paper at ICLR 2020
ROUGE
BLEU
RoBERTa-STS
Spearman's RC
0.255
0.216
0.744
Kendall's 0.215
0.186
0.69
Table 3: Results of logical entailment experiments
ASSESSING EVALUATION METRICS
4.1 METRIC SCORECARD
To overcome the previously highlighted challenges and provide framework in which metrics com- paring reference summaries/transltionn can be assessed and improved, we establish prst-principles criteria on what a good evaluator should do:
The first one that it should highly correlated with human judgement of semantic similarity.
The second one is that it should be able distinguish sentences which are in logical con- tradiction, logically unrelated or in logical agreement.
The third is that given s1, s2 which are semantically similar, eval(s1,s2) > eval(s1 (((oorrppte)) 2 eval(s (sm corrupted)) where corruption here includes re- moving words, adding noise the word order including grammatical mistakes.
4.2 IMPLEMENTING THE METRIC SCORECARD
We will now show how the scorecard can be implemented. For every dimension of the scorecard, the experiments are done with three metries. BLEU with equal weights between to 4 grams. ROUGE with averaging ROUGE-1 and ROUGE-2 and neural evaluator. The evaluator is the RoBERTa- STS.
4.2.1 SEMANTIC SIMILARITY
The first expectation from good similarity metric is to correlate highly with human judgment in terms of assessing semantic similarity. Here we assessed BLEU and ROUGE on the STS-B benchmark and compared their performance RoBERTa model fine tuned for semantic similarity (Table 2).
4.2.2 LOGICAL ENTAILMENT
Another characteristic a good metric is to differentiate the argument, core meaning in a sentence and take it into account when assessing hypothesis text with references. Here, used the MNLI dataset where ffor each text, we have three hypothesis texts representing contradiction, neutral and entailment. We expect good metrie to rank entailment higher than neutral, and .both of them, higher than contradiction. To assess the quality of a metric, we propose to use the Spearman's 's ranked correlation and jin (Table 3), also experiment with Kendall's Here, observe that the RoBERTa-STS model remarkably outperforms BLEU and ROUGE and both of these metrics show very little correlation with human judgment.
4.2.3 ROBUSTNESS TO GRAMMATICAL ERRORS
For assessing the third criteria. We start with 3479 sentence pairs from the MNLI dataset that are labelled as entailment. We introduce random corruptions such as random insertion, deletion and grammatical errors in (Zhao et al. 2019). We use two different set of parameters for different corruption levels, and expect that a good metric would rank the original similar sentence higher than the less corrupted and both higher than the more corrupted sentence. Here, also propose to the Spearman's ranked correlation and also experiment with Kendall's т. We report results on Table Where we that the RoBERTa-ST model once outperforms BLEU and ROUGE.
5

---

Under review as a conference paper at ICLR 2020
ROUGE
BLEU
RoBERTa-STS
Spearman's RC
0.528
0.472
0.718
Kendall's r
0.478
0.419
0.667
Table 4: Results of grammatical error experiments
ROUGE-1
ROUGE-2
ROUGE-L.
BLEU
RoBERTa-STS
Pearson Corr.
0.498
0.491
0.526
0.253
0.63
Table 5: Results of WMT experiments
MACHINE TRANSLATION EXPERIMENTS
In the previous section, outlined series of criteria assess evaluation metrics and shown how, for each dimension, RoBERTa-STS significantly outperformed ROUGE and BLEU. In the following section, we report results showing how BLEU, ROUGE and RoBERTa-STS correlate with human judgement of quality in the case of machine translation.
Using the WMT2015 and WMT2016 datasets, collected reference and hypothesis translations along with human judgment scores, where the translation was from another language to English. This gave us 5360 sentence pairs. For each of these sentence pairs, we computed the BLEU, ROUGE-1, ROUGB-2, scores. We also computed the RoBERTa-STS predicted semantic similarity.
We computed the correlation of the various scores with human judgements and reported the scores Table 5. Here again, we can see Transformers based model outperforming BLEU and ROUGE.
CONCLUSION AND FUTURE WORK
In this work, have established framework to metrics comparing the quality of refer- ence and hypothesis summary/translation. Based on these criteria, we compare evaluators using recent Transformers (in this case RoBERTa-STS) good performance on our scorecard translates on previously unseen machine translation datasets. Such results highlight the potential to replace BLEU and ROUGE with data-driven models such as RoBERTa-STS.
BLEU and ROUGE. We also show how this
REFERENCES
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluation the role of bleu in ma- chine translation research. In 1/th Conference of the European Chapter of the Association for Computational Linguistics, 2006.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity mulliiingul and cross-lingual focused evaluation. arXiv preprint arXiv: 1700.0008..55 2017.
Elizabeth Clark, Asli Celikyilmaz, and Noah Smith. Sentence mover's similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2748-21111 2760, 2019.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Qingsong Ma. Yvette Graham, Shugen Wang, and Qun Liu. Blend: a novel combined mt metric based on direct assessment-caase casict--eu submission to wmt17 metrics task. In Proceedings of the second conference on machine translation, pp. 598-603, 2017.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. Eleventh annual conference of the international speech communication association, 2010.

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp- 3111-3119, 2013.
Jekaterina Novikova, Ondiej Dušek, Amanda Cereas Curry, and Verena Rieser. Why need new evaluation metrics for nlg, arXiv preprint arXiv: (eti::/700.007555 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 311-318. Association for Computational Linguisties, 2002.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ily Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Ehud Reiter. A structured review of the validity of bleu. Computational Linguistics, 44(3):393-401, 2018.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. Ruse: Regressor using sentence
embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Tosk Papers, pp. 751-758, 2018.
Elior Sulem, Omri Abend, and Ari Rappoport. Bleu is not suitable for the evaluation text simpli- fication. arXiv preprint arXiv: 1xIv:18100005995,200 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-59980 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.077 07461, 2018
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint
2019.
Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 156-165, 2019.
A APPENDIX
In the appendix, will discuss the failure cases of BLEU, ROUGE and RoBERTa-STS in detail to provide better understanding of how these models can fall short in language evaluation. This important because good metric scorecard has to represent the quality of an evaluator. These experiments are to show that our metrics cover many of the failure cases and can assess them without the burden of manually evaluating the outputs of every evaluator.
We will start by taking examples from the BLEU and ROUGE dataset. As in the paper the BLEU scores used always uniform average up to 4-grams and the ROUGE score the average of ROUGE-1 and ROUGE-2. Both scores are scaled up to 5 to increase the interpretebility of the scores given that the labels in the similarty dataset are between 0 and 5.
Table 6: BLEU*/ROUGE* and RoBERTa-STS scores on sentence pair examples from STS-B
ld Sentence pair
BLEU
ROUGE*
RoBERTa-STS
Label 5.00/5
1
The company claims it's the largest single Apple VAR Xserve sale date. The company claimed it is the largest sale of Xserves by an Apple retailer.
0.44
1.62
4.36/5

---

woman queen Under review as a conference paper at ICLR 2020
2 woman puts flour on piece of meat.
0.63
1.78
5.07/5
5.00/5
A woman is putting flour onto meat.
3 He later learned that the incident was caused by the Concorde's 's sonic boom.
0.74
2.58
4.96/5
5.00/5
He later found out the alarming incident had been caused by Concorde's powerful sonic boom.
4 It indeed appears the Andromeda galaxy (M31) and The Milky Way (MW) are en route to a colli- sion.
0.20
1.09
3.37/5
4.40/5
In a few billion years, the Milky Way and An- dromeda will collide.
5 You definitely do NOT want to be supporting your weight with your arms on the bike for normal rid- ing.
0.28
1.13
2.73/5
4.20/5
No, don't support your weight on your arms Your hands simply aren't it really made for supporting all that weight.
6
detained for 'house sister' scandal
0.33
1.52
4.25/5
4.20/5
China detains for "house sister" scandal
man plays violin.
1.14
2.41
5.12/5
5.00/5
A man is playing violin.
8
is simply the number of balls bowled divided by the number of wickets taken.
0.80
1.84
3.83/5
4.40/5
Bowling strike rate is defined for bowler as the average number of balls bowled per wicket taken.
9 Police helicopter crashes into pub in Glasgow several casualties Helicopter crashes into roof of Glasgow club
0.47
1.36
3.58/5
4.00/5
10
Oil falls in Asian trade Oil prices down in Asian trade
1.62
3.14
4.89/5
5.00/5
11 A skateboarder jumps off the stairs.
3.21
3.77
1.09/5
0.80/5
dog jumps off the stairs.
12 Wigan 3-2 Wolves: Match report, pictures & video highlights Arsenal 0-0 Chelsea: Match report, pictures & video highlights
3.39
3.26
0.58/5
1.20/5
In t we see examples of many different error cases and jin most sentences, we also have more than one cause for the drastic difference between BLEU/ROUGE and the label. For instance, in rows and we see that the cause for the error the reordering of sub-sentences, spelling/punctuation and newly introduced words that don't 't change the meaning but merely extend it. While BLBU and ROUGE are failing in these examples, we see that the RoBERTa-STT model scores similarly to the label. In line 7, we can see that the RoBERTa-
model score is above 5.
In rows 2 and 7, we see that the main difference is the form or tense of the verb in a sentence. This makes BLEU severely under score simple changes with synonyms or valid re-orderings as seen in the examples below. This characteristic of BLEU reinforces the point that BLEU and ROUGE are not useful tracking the state of the art and comparing best methods but are tools to weed out bad models fairly simply.
In rows 3 and we see sentences that differ due to using descriptive phrases instead word or extending the sentence with more information. These types of errors changes are also caught with language models since we know they have the ability to hold the meaning of multiple words and incorporate them reach related word as in the famous example of king Mikolov et al. (2013).
In 4,8 and 5 different sentence. all cases we see a drastic difference between BLEU/ROUGE and the label but these cases also unearth specific characteristic of the neural evaluator. In 4 and we that the
general paraphrases with the meaning represented in generally
8

---

Under review as a conference paper at ICLR 2020
error of the RoBERTa-STS model comparatively lower than row 5. While it is hard to determine the exact cause through only looking at these examples table for the RoBERTa-STS failure cases will make this case more compelling.
While language models have general sense the context a given sentence, they still lack a general knowledge of the world. Hence in the second sentence of row 5, because the words riding, bike, bicycle are missing the model has a hard time recognising that the second sentence is also about the same topic. To test this added "while riding" "on bike" the end of sentence and the score immediately went up to 3.6/5 while barely changing the BLEU and the ROUGE score. In row 4 and however, the context of the sentence defined explicitly with the key phrases. We see this bias affecting RoBERTa-STT scoring in the examples below.
Table 7: BLEU*/ROUGE* and RoBERTa-STS scores on sentence pair examples from STS-B
Id
Sentence pair
BLEU
ROUGE$
RoBERTa-STS
Label
would be unusual for snake to attack station-
0.34
0.00
1.4/5
4.20/5
ary person.
I'm no herpetologist, but my experience, snakes in the "you don't bug me, I won't bug you" category.
2
New UN peacekceping chief named for Central African Republic
1.16
2.39
3.69/5
2.00/5
UN takes over peacekeeping in Central African Republic
3 From Broadway comedies like "The Seven Year Itch" (1952), "Will Success Spoil Rock Hunter?" Playwright George Axelrod, who anticipated the sexual revolution with The Seven Year Itch and Will Success Spoil Rock Hunter?
2.03
1.31
3.16/5
2.00/5
4
a group of navy seals are singing group military personnel are playing in brass quintet.
0.40
1.45
0.75/5
2.40/5
In the above examples, we will find two points that will helps us better understand the RoBERTa- STS as a neural evaluator. Firstly, we see that the neural network sometimes lacks sense of context that is not given in the sentence explicitly. While these language models are trained large corpus and capture a sense the words and language, we still see that their performance not perfect. We these examples in row 4, where the model cannot relate navy seal as military personnel. Or as in row 1, where the model cannot model an idiom.
The second and more critical place where need further development is especially detecting whether the core argument/message about the same things. As in rows 2 and 3. We see the same landmark words and can clearly say that the sentences are talking about the same things, but what a human can distinguish is that they saying unrelated things. This is of the key motivations in including the language inference task in the scorecard. Since detecting whether a pair sentences are related on what level a key part of detecting sentence similarity.
a sentence is the same beyond whether if they are talking
One last thing we will mention is that while RoBERTa-STS and BLEU/ROUGE have different error cases, their performance on these error
is also remarkably different in favor of the former. Table & shows the mean eтrоr of BLEU and the RoBERTa-STS model on each others top 500, which is one third of the development set, error cases.
Table Average error of BLEU and RoBERTa-STS in the their low scoring sets. With rows corresponding to which models failure cases and the columns to which model is used to score
BLEU
RoBERTa-STS
BLEU
2.93/5
0.47/5
RoBERTa-STS
1.68/5
0.89/5

---

Under review as a conference paper at ICLR 2020
We see in table 8 that BLEU* has remarkable error in both its failure cases and also the failure cases of RoBERTa-S -TEETT--STTTT while RoBERTa-STS outperforms BLEU in each category.
While neural evaluators have also room for improvement, outperforming classical methods and with methodical way improving them can bolster progress of NLP research.
with confidence say that they
10


Reviews:
[{"review_id": "HyeKi-nlcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n=== Summary ===\n\nThe authors motivate the development of new (automatic) metrics to evaluate language generation by using similarity with a given reference: standard metrics like BLEU, ROUGE or METEOR have been shown to have poor correlation with human judgment on a number of tasks and are vulnerable to changes in word re-ordering, semantics-changing word replacement, and syntactic transformations.\n  \nThey then propose a multi-dimensional evaluation criteria to evaluate sentence similarity based on semantic similarity (something that correlates with human judgments of the same), logical equivalence and fluency.\n  \nThe paper then goes on to describe possible directions to tackle several key problems in evaluation: evaluating semantic similarity by using models trained on the GLUE benchmark, evaluating logical equivalence using models trained on the MNLI corpus and fluency based on the CoLA corpus.\n\n=== Decision ===\n\nThe problem this paper seeks to tackle is clearly one of great\nimportance in the field, but I find it hard to argue that this paper\nsignificantly contributes to the existing body of work (more on this\nbelow) and as a result I vote to reject this paper.\n\nThere are two possible contributions for this paper: a set of criteria for what makes a good evaluation metric and the concrete proposals to implement these criteria.\n  \nFor the first, I find the proposed criteria to be overly generic and not helpful at providing additional clarity on what makes for a good evaluation: for example, how is semantic similarity different from logical consistency? Does it make sense to compare the semantic similarity of two sentences if one of them isn't even near grammatical? A lot of prior work already argue the shortcomings of the existing metrics this paper is making, e.g. Conroy and Dang (2008), Liu et al. (2016), Novikova et al. (2017). I think it would be valuable to present new axes to decompose the evaluation problem, but more work is needed to clarify and develop the axes presented in this paper.\n  \nFor the second possible contribution, the idea of evaluating language generation along dimensions is not novel and in fact quite standard in the NLP community. The challenge has been showing that there are subset of tasks that can be used a reliable metrics across different domains and systems. Unfortunately, this paper does not actually evaluate its own proposals, making it hard to evaluate how effective its proposals are.  \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xKiqomqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Observing shortcomings of BLEU and ROUGE, the paper proposes, JAUNE, a set of criteria for a good evaluation metric. These criteria include: high correlation with human judgement; being able to distinguish similar but contradicting statements; penalizing grammatical errors, and hard to game.\n\nThe paper, as its current form, is not ready for publishing. Some suggestions and comments:\n\n- Please carefully check the paper and fix typos and confusing sentences. I was collecting these errors but eventually stopped. Some examples. Sec. 2.3: punctuation missing between \"RUSE\" and \"this method\", comma missing after \"a discrete space\"; Sec. 4.1.1: \"made to ,for example\"....\n\n- The motivation of the paper is unclear. Is your criticism only about BLEU and ROUGE, or the state of the arts in NLP evaluation in general? To make JAUNE appealing, one has to argue that the state of the arts in NLP evaluation is ineffective. For this, the paper needs to review a boarder range of metrics beyond just BLUE and ROUGE. \n\n- While the authors suggest a data-driven metric, it reads to me like a model-driven metric (RoBERTAa specifically). Doesn't it systematically bias towards a certain family of metrics? \n\n- Better and more comprehensive experimental results are highly desired. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BylXkCeatH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that BLEU and ROUGE, two metrics that are used for the evaluation of machine translation and text summarization systems, are flawed, and proposes a new JAUNE metric to replace it.\n\nThe authors train a regressor on the STS-B dataset, and show that their model (which is using sentence embeddings from RoBERTa) corresponds better to the ground truth similarity labels than then scaled (but otherwise unchanged) BLEU scores. This is probably not surprising, given the small size and specific nature of the STS-B task and dataset. \n\nI could agree with many of the problems that the authors describe, but the proposed solution seems to be a very specific solution that works on a given dataset (for which supervised training data is available), but I do not think it will generalize well to unseen test data in different domains. I also do not understand how the BLEU score can simply be rescaled from 0-5 - how do you determine the maximum BLEU score before rescaling?\n\nThe paper should be proofread by a native speaker, many sentences are unclear, and spacing as well as punctuation are used in weird ways.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors tackle the questions of automatic metrics for assessing document similarity and propose the use of Transformed-based language models as a critic providing scores to samples. As a note, ideas like these have been also adopted in Computer Vision with the use of the Inception score as a proxy the quality of generated images. The authors ask great questions in the paper and they clearly tackle a very important problem, that of automatic measures for assessing text quality. While their first indications are not negative, this paper lacks the rigor and depth of experiments of a conference paper that would convince the research community to abandon BLEU and ROUGE in lieu of some other metric. It's perhaps a good workshop paper or a short paper at a *CL conference. Specifically, we would need more tasks where BLEU/ROUGE is the standard measure and so how the proposed measure correlates better with humans,  so cases where word overlap is in theory a good proxy of similarity assuming reference sentence (e.g., logical entailment is not such a prototypical task). MT is a first step towards that, but summarization is also a necessary I would say. Other questions of interest relate to the type of LM (does it only need to be Roberta?) and the quality of LM (what if i badly tune my LM?)  On a more personal note: We all know that BLEU is not a good metric (especially for document-level judgements) and every now and then there have been proposals to replace BLEU that do correlate better (e.g., http://ccc.inaoep.mx/~villasen/bib/Regression%20for%20machine%20translation%20evaluation.pdf) . However, BLEU is still here due to each simplicity. Please keep pushing this research and I’m looking forward to seeing more experimental evidence.
Decision: Reject