A Holistic Approach to Reference-Free Evaluation of Machine Translation
Year: 2023

Abstract:
Traditional machine translation evaluation relies on references written by humans. While reference-free evaluation gets rid of the constraints of labor-intensive annotations, it can pivot easily to new domains and is more scalable. In this paper, we propose a referencefree evaluation approach that characterizes evaluation as two aspects: (1) fluency: how well the candidate translation conforms to normal human language usage; (2) faithfulness: how well the candidate translation reflects the source data. We further split the faithfulness into word-level and sentence-level. Extensive experiments spanning WMT18/19/21 Metrics segment-level daRR and MQM datasets demonstrate that our proposed reference-free approach, ReFreeEval, outperforms SOTA reference-free metrics like YiSi-2, SentSim and BERTScore-MKD in most language directions. The code can be found at ReFreeEval Repo1.

heading: 1 Introduction
text: Machine translation evaluation has conventionally relied on reference, where outputs are compared against translations written by humans. This is in contrast to the reference-free manner in which translation quality is directly assessed with the source text. Reference-free evaluation (Napoles et al., 2016; Thompson and Post, 2020; Agrawal et al., 2021) has the potential to free the evaluation model from the constraints of labor-intensive annotations, allowing it to pivot easily to new domains. In this way, reference-free evaluation metrics are substantially more scalable and have lately been in the spotlight.
The history of reference-free evaluation for MT can trace back to “QE as a Metric” track of
˚ Equal contribution. : Corresponding author.
1https://github.com/cocacola-lab/ Reference-Free-Evaluation-of-Machine-Translation. git
WMT2019 Metrics Task (Ma et al., 2019). YiSi2 (Lo, 2019) and XBERTScore (Zhang* et al., 2020; Leiter, 2021) are embedding-based methods that adopt contextual word embeddings to calculate the lexical similarity between the source and candidate translation words. Quality estimation (Fonseca et al., 2019) system metrics such as UNI+ (Yankovskaya et al., 2019) and COMETQE (Rei et al., 2020a, 2021) also leverage contextual word embeddings and feed them into a feedforward network. However, they are trained to regress on human scores that are expensive to collect, and gross discrepancies exist when different humans are asked to label the scores.
More challenging but worthwhile, we focus on dispensing with references as well as human scores. Nevertheless, embedding-based methods are limited to token-level semantic similarity while neglecting sentence-level faithfulness (Song et al., 2021). Besides, it’s difficult for word embeddings to discriminate matched word pairs from random ones (Zhao et al., 2020a).
In addition, current reference-free evaluation methods rarely take fluency into account. For the unfluent candidates whose content is roughly consistent with the source, the embedding-based metrics can hardly discriminate and provide accurate evaluation scores2. Moreover, the general goal of evaluation metrics is to estimate not only the semantic equivalence between source and candidate but also the general quality (i.e., fluency and naturalness) (Banchs et al., 2015; Feng et al., 2020; Yuan et al., 2021).
In this work, we propose a holistic approach (i.e., ReFreeEval) to enhance the evaluation model in aspects of fluency and faithfulness, meanwhile on both word and sentence levels. With regard to fluency, we pose a data augmentation method and train a fluency discrimination module. For word-level faithfulness, we adopt a self-guided
2We provide more details and case studies in Appendix B.
623
contrastive word-alignment method. For sentencelevel faithfulness, we execute knowledge distillation with SBERT (Reimers and Gurevych, 2019) to capture more fine-grained semantics. Our method builds on the framework of XBERTScore. Extensive experiments spanning WMT18/19/21 Metrics (Ma et al., 2018, 2019; Freitag et al., 2021) segment-level daRR and MQM datasets demonstrate that our proposed reference-free approach, ReFreeEval, outperforms SOTA reference-free metrics like YiSi-2, SentSim and BERTScoreMKD in most language directions.

heading: 2 Approach
text: Reference-free evaluation of MT can be characterized as two aspects: (1) fluency: how well it conforms to normal human language usage; and (2) faithfulness: how well the translated text reflects the source data. We assess faithfulness at different granularity: word level and sentence level. Figure 1 is the illustration of our ReFreeEval method.

heading: 2.1 Sentence-Level Fluency
text: We explore a data augmentation method to perturb the fluency of target sentences with noise which is difficult to be identified. Then we train a fluency discrimination module with contrastive learning (Gao et al., 2021; Zhang et al., 2021; Wu et al., 2022; Wang et al., 2022) to distinguish fluent samples from perturbed samples (namely, challenging negative samples).

heading: Data Augmentation Using Clause Permutation
text: A complex or compound sentence3 has two or more clauses and relative clauses that are joined together with conjunctions or punctuation. As logical relations exist between these clauses, we manipulate and permute the clauses separated by punctuation, instead of words. In this way, the meaning is preserved inside the clauses, meanwhile, the sentence is often unfluent and unnatural. Similar to complex and compound sentences, for a simple sentence with only one clause4, we randomly split it into two fragments and permute the two fragments. Compared to permutation on the token level, clauselevel permutation has less influence on sentence fluency and semantic change. The clause-based
3https://simple.wikipedia.org/wiki/Sentence# Types_of_sentence
4https://simple.wikipedia.org/wiki/Simple_ sentence
permutation method brings perturbed samples that are more challenging and hard to be recognized.
Fluency Discrimination We denote a source and target sentence in parallel data as x and y. Perturbed samples augmented from y are ŷ1, ŷ2, ..., ŷk. A reliable metric has the ability to give the original fluent target y a higher evaluation score than those k perturbed unfluent samples.
As for the score, we adopt the same calculation measure as BERTScore but replace the pre-trained monolingual model (Devlin et al., 2019; Liu et al., 2019) with a cross-lingual model (Devlin et al., 2019; Conneau et al., 2019) to do reference-free evaluation (Zhou et al., 2020; Song et al., 2021) denominated as XBERTScore (Leiter, 2021). We use 9th layer of XLM-Roberta-Base to extract contextual word embeddings. Here we only use FBERT as evaluation score between source x and targetside y or ŷi, which is represented as swpx, yq or swpx, ŷiq. Then we can obtain word-level faithfulness scores swpx, yq, swpx, ŷ1q, ..., swpx, ŷkq of pk ` 1q pairs.
In order to discriminate fluent sentences from perturbed ones according to these scores, we treat the original target and its corresponding perturbed samples as opposite and assign them 1/0 hard labels. The cross-lingual model which produces XBERTScore is trained to classify target-side sentences with a cross-entropy loss function. The objective function on N training samples is as follows:
Lfl “ ´ 1 N
ÿ
x,y
log eswpx,yq
eswpx,yq ` řki“1 eswpx,ŷiq (1)

heading: 2.2 Word-Level Faithfulness
text: As for word-level faithfulness, each word in the source sentence should have a corresponding crosslingual representation in the target sentence and each word in the target sentence should be an accurate translation of its source word. This motivates us to do word-alignment training to enhance wordlevel evaluation.
This module shares similar architecture with sentence-level fluency where word embeddings are derived from 9th layer of XLM-Roberta-Base.
We take the same steps as (Dou and Neubig, 2021) to extract alignments. First, we compute the dot product between source and target word embeddings to obtain the similarity matrix S. Then S is normalized in source and target dimensions. And
we get source-to-target alignment matrix Sxy and target-to-source alignment matrix Syx. A source/target token and a target/source token whose similarity value in alignment matrix Sxy/Syx exceed threshold c1 are regarded as aligned. The bidirectional alignment matrix A is deduced:
A “ pSxy ą c1q ˚ pSTyx ą c1q (2)
Aij “ 1 means xi and yj are aligned. Dou and Neubig (2021) also propose the self-training objective to align words with this bidirectional alignment, which improves alignment performance most.
Based on this objective, we adopt a self-guided contrastive cross-lingual word-alignment method. By contrast, we not only pull semantic aligned words to have closer contextual representations but also push unrelated words away (Luo et al., 2021; Su et al., 2022; Meng et al., 2022), which encourages the model to discriminate matched word embeddings from semantically unrelated ones.
The source token and target token are deemed to be unrelated if their similarity value is low. In our method, these unmatched pairs constitute negative samples and are pushed away. Moreover, we set threshold c2 to further restrict the negative samples. The unmatched pairs whose similarity value is lower than c2 are discarded from negatives as this unmatched relation can be easily distinguished by the model. In this way, we can control the difficulty of negative samples and only preserve those indistinguishable ones (hard negatives) to train the model.
B “ pSxy ą c2q ˚ pSTyx ą c2q (3)
Bij “ 1 means xi and yj are aligned or a part of hard negatives, which are preserved to train.
In Figure 1, the dark blue positions mean bidirectional alignment while the light blue positions are hard negative examples.
Finally, based on two dimensions of source and target, the positive and negative samples mentioned above, we construct a self-guided contrastive learning objective function on the word level as follows:
Lx “ ´ 1 m
mÿ
i“1 řn j“1 1pAij “ 1qeSxyijřn j“1 1pBij “ 1qeSxyij (4)
Ly “ ´ 1 n
nÿ
i“1
řm j“1 1pATij “ 1qeS T yxij řm j“1 1pBTij “ 1qeS T yxij
(5)
Lword “ Lx ` Ly (6)

heading: 2.3 Sentence-Level Faithfulness
text: The main idea is to improve sentence-level faithfulness evaluation. Concretely, we distill sentencelevel semantic meaning from SBERT into the wordlevel shared model.
We use SBERT to extract semantically meaningful sentence embeddings. Sentence semantic similarity between x and y is calculated with cosinesimilarity between sentence embeddings x and y:
sspx, yq “ x ¨ y}x}}y} (7)
The semantic similarity reflects the sentencelevel faithfulness from target to source. Then we can obtain sentence-level faithfulness scores sspx, yq, sspx, ŷ1q, ..., sspx, ŷkq. We use KLdivergence as the objective function to reduce the
discrepancy between sentence-level and word-level similarity:
Lfa “ ÿ x,y1PYx sspx, y1q log sspx, y 1q swpx, y1q (8)
In this distillation module, SBERT plays a role of a teacher. Sentence-level semantic knowledge is distilled into the word-level shared model through these sentence-level faithfulness scores. In this way, evaluation is no longer limited to word level but incorporated sentence semantics.
On the other hand, SBERT plays a role as a corrector. It is unreasonable that a disturbed sample with slightly changed semantics is considered to be completely contrary to the original sentence. We correct the binary classification and convert the 0/1 discrete value in the fluency discrimination module to continuous variables.
For sentence-level training, we combine fluency with faithfulness. This joint architecture is motivated by (Ren et al., 2021). The objective is:
Lsent “ Lfl ` αLfa (9)
α is a hyper-parameter to control the weight that the sentence-level faithfulness module accounts for.

heading: 3 Experiment

heading: 3.1 Setup
text: Datasets We train and evaluate on four language pairs: EnglishØChinese and EnglishØGerman. For training, we use the datasets following Awesome-Align (Dou and Neubig, 2021). The En-Zh training dataset is collected from the TsinghuaAligner5 website and En-De training data is Europarl v7 corpus. For evaluation, we use the segment-level daRR dataset of WMT18/19 and MQM dataset of WMT21 Metrics Task. Details about datasets are introduced in Appendix C.1.
Embeddings We use the 9th layer of XLMRoberta-Base to extract contextual word embeddings. This follows the default setting of BERTScore6. For sentence embeddings, we adopt xlm-r-bert-base-nli-stsb-mean-tokens model7 the same as SentSim.
5http://nlp.csai.tsinghua.edu.cn/~ly/systems/ TsinghuaAligner/TsinghuaAligner.html
6https://github.com/Tiiiger/bert_score 7https://huggingface.co/sentence-transformers/
xlm-r-bert-base-nli-stsb-mean-tokens
Baselines For reference-based metrics, we choose sentBLEU (Papineni et al., 2002) and YiSi1 (Lo, 2019). For reference-free metrics, we choose XBERTScore (Leiter, 2021) , YiSi-2 (Lo, 2019), SentSim (Song et al., 2021) and BERTScoreMKD (Zhang et al., 2022). Most results of baseline models are reported in the original paper (Ma et al., 2018, 2019; Freitag et al., 2021; Zhang et al., 2022). We also implement experiments that have not been reported, such as XBERTScore, SentSim and BERTScore-MKD.
Training Process For ReFreeEval, sentencelevel module is first trained. Then word-level faithfulness module is trained based on the best checkpoint of sentence-level training. Training details are in Appendix C.3.
Evaluation Measures For WMT18/19 segmentlevel evaluation, Kendall’s Tau-like formulation is used to measure the scores against daRR.
τ “ |Concordant| ´ |Discordant||Concordant| ` |Discordant| (10)
For WMT21 segment-level evaluation, conventional Kendall-tau statistic is used to measure the correlation between our scores and MQM scores.

heading: 3.2 Results
text: The main results are displayed in Table 1, 2, 3. First, we observe that fluency, word-level faithfulness, and sentence-level faithfulness module improve the evaluation performance respectively. We also find that the main improvement comes from sentencelevel fluency indicating that XBERTScore as a token-level evaluation metric lacks sentence-level knowledge. Then, the ensemble model combining the advantages of the three modules achieves even better results. And compared with some referencebased baselines it achieves comparable results or even outperforms them. More details of experimental results are in Appendix C.4.

heading: 4 Conclusion
text: We propose a reference-free evaluation approach ReFreeEval that comprehensively considers three aspects: fluency, word-level faithfulness, and sentence-level faithfulness. Extensive experiments spanning datasets from WMT18/19/21 demonstrate the superiority of each module designed for each
8Reported by Zhang et al. (2022). Our implementations here are 0.1997 in Zh-En and 0.0623 in De-En.
aspect. ReFreeEval, combining the above three modules, achieves a higher correlation with human judgments, outperforming current SOTA referencefree metrics like YiSi-2, SentSim and BERTScoreMKD in most language directions.

heading: Limitations
text: In this section, we discuss some limitations of our method and future work based on the limitations.
First, the enhancement of the word-level module is not as strong as the remedy of the sentence-level module. Our word-level module solely achieves improvement compared with XBERTScore but doesn’t improve as much as the sentence-level module. The main reason is that the XBERTScore framework lacks sentence-level semantic knowledge. Besides, our word-level self-guided contrastive method doesn’t resort to external information and only consolidates the alignment already existing in the pre-trained language model. Second, ReFreeEval performs comparably with baseline models on language pairs involving German. We guess it is due to the evaluation of QE. Ma et al. (2019) mention that the evaluation results across all language pairs are unstable in “QE as a Metric” track and can’t explain yet.
In the future, we’ll further explore valuable external information on word level. And we’ll try to explore discrepancies among language pairs to optimize the results. In addition, our simple but effective data augmentation method - clause per-
mutation doesn’t rely on rules or toolkits, which is an initial attempt at modeling fluency. It could benefit from further refinement such as languagespecific knowledge, syntactic and semantic parsing to recognize clauses. We’ll conduct an in-depth investigation into further work.

heading: Acknowledgements
text: The research work described in this paper has been supported by the National Key R&D Program of China (2020AAA0108001) and the National Nature Science Foundation of China (No. 61976015, 61976016, 61876198 and 61370130). Wenjuan Han is supported by the Talent Fund of Beijing Jiaotong University (2023XKRC006). The authors
would like to thank the anonymous reviewers for their valuable comments and suggestions to improve this paper. We would like to express our sincere gratitude to Hui Huang for guidance before this research. We are also grateful to Chunyou Li, Yu Xiang and Yu Zhang for their assistance during internship.

heading: A Related Work

heading: A.1 Reference-based Evaluation for MT
text: According to matching features, reference-based evaluation methods can be categorized as follows: (1) n-gram(e.g. BLEU (Papineni et al., 2002) and CHRF (Popović, 2015)); (2) edit distance(e.g. TER (Snover et al., 2006) and EED (Stanchev et al., 2019)); (3) word embedding(e.g. YiSi (Lo, 2019) and BERTScore (Zhang* et al., 2020)); (4)predictor-estimator model (Kim et al., 2017)(e.g. COMET (Rei et al., 2020a)). n-gram matching
metrics are restricted to surface form and neglect semantic meaning.
Instead, embedding-based metrics adopt word embedding to explore word-level semantic meaning. WMDo (Chow et al., 2019) builds on Word Mover’s Distance (Kusner et al., 2015) to measure the similarity of candidate and reference. It also introduces a word order penalty to take fluency into account. YiSi-1 aggregates the weighted lexical similarity to evaluate translation quality. BERTScore calculates the token-level semantic similarity between candidate translation tokens and reference tokens. DA-BERTScore (Zhan et al., 2021) takes translation difficulty into account and assigns difficulty weightings to each token in reference.
COMET leverages contextual word embeddings of the source sentence, MT hypothesis, and reference (or human post-edition) extracted from pretrained cross-lingual models. The embeddings are combined and fed into a feed-forward network. It’s a quality estimation system and is trained with human assessments(DA, HTER, MQM).

heading: A.2 Reference-Free Evaluation for MT
text: As reference is costly to be collected in practice, reference-free metrics attract more attention. Recent studies have explored evaluating translation quality only based on the source text.
YiSi-2 calculates similarities between crosslingual word embeddings for aligned source and candidate translation words and outputs an Fmeasure statistic as the metric score. Zhao et al. (2020b) propose to re-align vector spaces and couple the semantic faithfulness scores with GPTbased fluency testing.
OpenKiWi-XLMR (Moura et al., 2020) and COMET-QE (Rei et al., 2020b) are quality estimation systems from “QE as a Metric” task (Mathur
et al., 2020). They remove reference at the input but still require human assessments to train.
As reference-based BERTScore has achieved outstanding performance, many recent referencefree evaluation methods build on BERTScore. XBERTScore (Leiter, 2021) adopts the crosslingual pre-trained language model to evaluate only based on source sentence without reference. SentSim (Song et al., 2021) combines semantic sentence similarity with token-level BERTScore. BERTScore-MKD (Zhang et al., 2022) also uses sentence embeddings to achieve cross-lingual word embedding alignment by multilingual knowledge distillation.

heading: B Case Study
text: From Table 4, we can see there is a significant difference between the golden truth DA of MT1 and MT2. And the quality of MT1 is much better than MT2. But XBERTScore evaluates incorrectly and assigns MT1 with a lower score than MT2. Though MT2 is translated word by word which means poor fluency, almost all words in MT2 can be aligned with source. As XBERTScore method is evaluated on word-level matching, it can be easily confused. The model trained with our holistic approach can make up for this shortage and discriminate the fluency problem.

heading: C Experimental Details

heading: C.1 Data Analysis
text: Following the data setting of awesome-align (Dou and Neubig, 2021), we use the following parallel corpora to fine-tune our model. The EnglishChinese(En-Zh) dataset is collected from the TsinghuaAligner webset and Englist-German(En-De) dataset is the Europarl v7 corpus. We only adopt a multilingual setting but use less data. We ran-
domly sample 20k parallel sentence pairs from each dataset and mix them together.
In the word-level faithfulness module, we directly use mixed data to train. In the sentence-level fluency and faithfulness module, as only the target is perturbed, we randomly select 1/3 mixed data and swap the source and target in order to attend to all three languages.
To evaluate our method, we choose segmentlevel evaluation datasets of WMT Metrics Task. Two types of human assessments are included. Segment-level Metrics datasets of WMT18/19 use daRR(direct assessment relative ranking) as ground truth and WMT21 use MQM(multidimensional quality metrics) as ground truth.

heading: C.2 Details of Sentence-Level Faithfulness
text: Before applying KL-divergence, the word-level and sentence-level similarity scores are processed as follows.
swpx, yq “ logp e swpx,yq
ř y1PYx eswpx,y 1q q (11)
sspx, yq “ e sspx,yq
ř y1PYx esspx,y 1q (12)

heading: C.3 Training Details
text: Our model is fine-tuned based on the 9th layer of XLM-Roberta-Base. We implement our model with Pytorch (Paszke et al., 2019), Transformers (Wolf et al., 2020) and BERTScore (Zhang* et al., 2020) package. We use AdamW optimizer (Loshchilov and Hutter, 2017). The model is trained on up to 3 GeForce RTX 2080 Ti GPUs.
For sentence-level training, the hyperparameter settings are displayed in Table 5. We mainly search α P t0, 1, 5, 10, 20, 30, 40, 50, 100, 500u. The training process is on a single GPU with gradient accumulation. We evaluate the model with classification accuracy every 100 steps and save the checkpoint with the highest accuracy.
For word-level training, the hyperparameter settings are displayed in Table 6. We search batch sizeP t8, 10, 15, 16, 24, 28, 32, 48u, learning rateP t1e´5, 5e´6, 3e´6, 1e´6, 2e´6, 5e´7, 1e´7u and c2 P t1e ´ 5, 1e ´ 10, 1e ´ 15, 1e ´ 20, 1e ´ 30, 1e ´ 50u. For dataset of WMT18/19 the training process is on 3 GPUs and the batch size on each GPU is 5. Specifically, for WMT21 MQM dataset the batch size is 32 and the learning rate is 2e-6. The training is on 4 GPUs and the batch
size on each GPU is 8. The code of this module is implemented based on awesome-align (Dou and Neubig, 2021)9. This word-level faithfulness training continues on the basis of the best checkpoint of sentence-level training.

heading: C.4 Details of Experimental Results
text: In Section 3, as we want to demonstrate the improvement of our ReFreeEval in multilingual setting of all language directions, we report results corresponding to the highest “average” of all language pairs for each dataset.
Table 7, 8, 9 are the best results of each language direction of WMT18/19/21 dataset.
9https://github.com/neulab/awesome-align/tree/ xlmr

heading: D Analysis

heading: D.1 Analysis of Data Augmentation
text: We compare our clause permutation with tokenlevel data augmentation methods shuffling and repetition. The results are displayed in Table 10.
For the fluency module alone, our clause-based augmentation method performs much better than the others, which suggests that our method provides more proper and valuable fluency information than others. As for sentence-level faithfulness, we compare the variation of sentence semantic similarity in Table 11. The disturbance caused by token shuffling is too great while our clause permutation is small. The obvious disturbance is easy to be distinguished and learned. While the disturbance caused by our method can hardly be distinguished by sentence similarity thus only this module is not enough. However, with the clause permutation method, the combination of both fluency and sentence-level faithfulness outperforms others a lot. This verifies that our clause-based augmentation method is effective.
Based on the linguistic definition of clauses, our clause permutation approach can effectively incorporate perturbation to continuity and smoothness, which constitute the essence of fluency. This approach is simple and intuitive, making it a suitable choice for the preliminary step for more in-depth investigations about realistic perturbations.

heading: D.2 Balance between Fluency Discrimination and Faithfulness Distillation
text: For sentence-level training, we adjust the hyperparameter α to balance fluency and faithfulness.
A small α means the sentence-level training mainly focuses on classification, which may neglect the semantic meaning of perturbed samples as we explained in section2.3. While a large α weakens the effect of hard classification labels, the soft similarity is also not enough for sentence-level training.
From Table 12, we can conclude that only by keeping the balance between hard fluency discrimination and soft faithfulness distillation can we achieve excellent experimental results.

heading: D.3 Control over Difficulty of Negative Samples in Word-Level Faithfulness
text: We experiment with different settings of threshold c2 in word-level faithfulness to observe the influence of the difficulty of negative samples.
A small c2 reduces the difficulty of contrastive learning. This setting includes negative samples whose unmatched relations can be easily distinguished. While a large c2 restricts the negative samples extremely, which may lose some useful information. The results in Table 13 indicate that properly controlling the difficulty of negative samples can lead to great performance on the whole. However, for En-De, a small threshold is beneficial to improve the results. This may be because negatives without strict limitations are harmful to contrastive learning due to specific language features of En-De.

heading: E Significance Test
text: Our experimental results above are based on the model training in a single run with random seed 42. In this section, we implement the statistical significance test following (Dror et al., 2018) to further compare the performance of our ReFreeEval with a strong baseline SentSim. We run both the models 10 times with random seed P t19, 27, 42, 55, 76, 80, 99, 153, 178, 200u. The p-value of the statistical test is displayed in Table 14. As we can see, the p-value on each language pair is well below the significance level of 0.05, which indicates that the results of our ReFreeEval are significantly better than SentSim.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section: Limitations
7 A2. Did you discuss any potential risks of your work? No potential risks

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section: Abstract and Section1: Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: Mainly in Section3: Experiments and Appendix D: Experimental Details. Also Section1: Introduction
text: and Section2: Approach mention models.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section Reference and footnotes of Section2 and Section 3.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section Reference and footnotes of Section2 and Section 3.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section2 and Section3
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Our data doesn’t have these information.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section3 and Appendix D
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section3 and Appendix D.1
C 3 Did you run computational experiments?

heading: Section3 and AppendixE / F

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section3 and Appendix D.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix F

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In this section, we discuss some limitations of our method and future work based on the limitations. First, the enhancement of the word-level module is not as strong as the remedy of the sentence-level module. Our word-level module solely achieves improvement compared with XBERTScore but doesn’t improve as much as the sentence-level module. The main reason is that the XBERTScore framework lacks sentence-level semantic knowledge. Besides, our word-level self-guided contrastive method doesn’t resort to external information and only consolidates the alignment already existing in the pre-trained language model. Second, ReFreeEval performs comparably with baseline models on language pairs involving German. We guess it is due to the evaluation of QE. Ma et al. (2019) mention that the evaluation results across all language pairs are unstable in “QE as a Metric” track and can’t explain yet. In the future, we’ll further explore valuable external information on word level. And we’ll try to explore discrepancies among language pairs to optimize the results. In addition, our simple but effective data augmentation method - clause per-
mutation doesn’t rely on rules or toolkits, which is an initial attempt at modeling fluency. It could benefit from further refinement such as languagespecific knowledge, syntactic and semantic parsing to recognize clauses. We’ll conduct an in-depth investigation into further work