Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yuan Wang
authors: Wuying Liu
reviews: 
Comments: This paper analyzes the correlation between evaluation metrics for machine translation, specifically focusing on seven commonly used morphological evaluation metrics and four semantic evaluation metrics based on pre-trained models. The study claimed that there is a strong correlation between morphological evaluation metrics, as well as between semantic evaluation metrics. The main contribution of this paper is providing insights into the relationship between different evaluation metrics and suggesting personalized selection methods for evaluation metrics.
Strengths: This paper provides an insightful analysis of the correlation between evaluation metrics for machine translation. The study systematically analyzed the correlation between seven morphological evaluation metrics and four semantic evaluation metrics, and  reveals a strong correlation among different morphological evaluation metrics. This research contributes to the understanding of how different evaluation metrics relate to each other and suggests personalized selection methods based on language characteristics.

In terms of quality, the authors conduct a thorough analysis of the correlation coefficients obtained through Pearson, Kendall, and Spearman tests, providing comprehensive results and supporting their conclusions. 

Clarity is another strength of this paper. The authors present their findings in an organized manner and provide clear explanations of their methodology and results. They also use appropriate visual aids like figures and tables to help readers understand their findings better.
Weaknesses: The paper inadequately presents the results. It only shows thumbnails of ZhoX without enlarged images, making specific values difficult to discern.
The paper only presents results for VieZho. It is recommended to include additional language results  in the Appendix.
The paper has a strong start and weak finish. The value of the work will be much improved if the authors conduct a more detailed analysis of why these metrics perform differently across various languages and which types of languages are better suited to specific metrics.
In linguistics, semantics and morphology have their own definitions and functions, and there is naturally a connection between them. The current work is insufficient to support the conclusion that “the semantics of language do not exist.”
Recommendation: 5
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The paper investigates correlation coefficients between different automatic scores, some of them used for machine translation. A machine translation system was developed for translation from Chinese into 20 different languages, automatic scores were calculated, and three correlation coefficients were investigated: Pearson's, Spearman's and Kendall's.
Strengths: A thorough investigation of different automatic metrics is important.
Weaknesses: The authors should familiarise themselves with machine translation and its evaluation.

The most important shared task involving many machine translation systems, language pairs and evaluation metrics: 
https://www2.statmt.org/wmt24/

To name some of the problems: 

WMT shared task is not at all mentioned/cited

the division of the metrics in groups is incorrect: there is no "morphological" and "semantic" metrics, only overlap/match based metrics and neural metrics based on embeddings/representations

while some widely used automatic metrics are included, some of the automatic scores are not at all used as machine translation evaluation metric

the analysis was carried out using one single MT system, so the behaviour of the metrics for comparing different systems is not tested at all

the definition of machine translation is incorrect "uses a target natural language form to paraphrase the semantic of a source natural language"
Recommendation: 1
Reviewer confidence: 5
Soundness/correctness: 1
Clarity: 1
reviews: 
Comments: This work looks at understanding the relation between machine translation metrics that operate at the surface level and metrics that try to operate at the sentence meaning level. The setup consists of 40 language pairs (XXtoZh and ZhtoXX directions) where MT models are trained from scratch and outputs from these models are evaluated with different MT metrics. This is followed by correlating the scores between different metrics per language pair. Analysis of how different metrics operate is carried out based on these correlation scores.
Strengths: 1. The choice of languages within language pairs is extensive. Further, by keeping Zh as the central language, new insights about MT evaluation can be uncovered. 
2. The use of semi-supervised data augmentation to build MT models is quite interesting. As all of the data and models are in-house, the setup can be useful for carefully analysis
Weaknesses: 1. While the setup contains 11 different evaluation strategies, the proposed experiment setup may need further validation to ensure their reliability (please see details in the next section).
2. Contemporary metrics like COMET (https://arxiv.org/abs/2009.09025), UniTE (https://aclanthology.org/2022.acl-long.558/), or even BERTScore (https://arxiv.org/abs/1904.09675), that is closest to the mentioned setup for semantic similarity have not been included in the analysis
3. Previous literature has not been appropriately cited.
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper conducts a correlational analysis of the machine translation evaluation metrics. The metrics of interest includes "morphological" metrics which covers metrics that checks for surface-form similarity, such as BLEU and chrF, and "semantic" metrics, which basically computes the similarity of sentence embeddings between the translation and the reference. The study finds very strong correlations with morphological metrics, while also showing that semantic metrics also have strong correlations when different semantic embeddings are used, and also with the morphological metrics. The study on 20 Chinese-centric language pairs also show that correlations between semantic vs. morphological metrics also vary highly between different language pairs.
Strengths: 1. This is a thorough study that covers many different correlation measures, metrics variations, as well as language pairs.
2. The data setup is a Chinese-centric one, which offers a novel perspective on top of many existing English-centric studies.
Weaknesses: 1. At its current shape, the paper unfortunately doesn't add much to the existing literature, mainly because the paper seems detached in many ways from the state-of-the-art of the machine translation metrics research, elaborated in the following aspects:
1.1. The authors made no mentions at all about the findings of the past WMT metrics shared tasks (most recent: http://www2.statmt.org/wmt24/metrics-task.html), which overlaps heavily with the topic of study here.
1.2. While the choice of morphological metrics seems reasonable, the choice of semantic metrics seem very unorthodox. I would suggest the authors consider some more frequently used options: (1) COMET (https://arxiv.org/abs/2009.09025) (2) BERTScore (https://arxiv.org/abs/1904.09675) (3) BLEURT (https://arxiv.org/abs/2004.04696) (4) MetricX (https://aclanthology.org/2023.wmt-1.63.pdf). (5) AutoMQM (https://arxiv.org/abs/2308.07286). The aforementioned shared task is also a good starting point to find more frequently used metrics.
2. A lot of the key questions on how the evaluation is conducted is unclear, to name a few: specific implementation of metrics such as BLEU/ChrF, details of the MT systems used, source of the Chinese-centric evaluation data (e.g. what domain/website were those crawled from?). I listed some of them in the question section.
3. The presentation could be more focused on the contribution of the study, rather than re-introducing concepts such as correlation coefficients (note that you SHOULD mention important implementation details like how the ties are treated in Kendall's tau correlation).
Recommendation: 1
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3