Improving Translation Quality Estimation with Bias Mitigation
Year: 2023

Abstract:
State-of-the-art translation Quality Estimation (QE) models are proven to be biased. More specifically, they over-rely on monolingual features while ignoring the bilingual semantic alignment. In this work, we propose a novel method to mitigate the bias of the QE model and improve estimation performance. Our method is based on the contrastive learning between clean and noisy sentence pairs. We first introduce noise to the target side of the parallel sentence pair, forming the negative samples. With the original parallel pairs as the positive sample, the QE model is contrastively trained to distinguish the positive samples from the negative ones. This objective is jointly trained with the regression-style quality estimation, so as to prevent the QE model from overfitting to monolingual features. Experiments on WMT QE evaluation datasets demonstrate that our method improves the estimation performance by a large margin while mitigating the bias1.

heading: 1 Introduction
text: Quality Estimation (QE) aims to predict the quality of machine translation automatically in the absence of reference translations. State-of-the-art QE model mostly falls into Pre-Trained Model (PTM)based paradigm. In the latest QE evaluation tasks (Zerva et al., 2022), nearly all top-performing systems adopt Multilingual PTMs as backbone.
Good as the PTM based QE performance is, recent researches (Sun et al., 2020; Behnke et al., 2022) reveal that state-of-the-art QE models are biased. To be specific, the models largely rely on spurious monolingual features, such as the fluency of the target sequence, or the complexity of the source sequence, without really capturing the
†Contribution during internship at ByteDance Inc. ‡Corresponding Authors. 1Codes are available at https://github.com/
HuihuiChyan/AwesomeQE-contrast
bilingual semantic alignment. Such monolingual features do not have a casual impact on the translation quality, and bias the QE results to a large extent. For example, as shown in Figure 1, a fluent and uncomplicated translation might be assigned with a high quality score even it does not resemble the actual semantics of the source sentence, while an adequate translation with complicated structure might be assigned as bad translation.
Sun et al. (2020) recommends to counter with the bias by using a metric that represents adequacy well as labels. However, in their such annotated dataset, the bias is still striking, as revealed by Behnke et al. (2022). As an alternative, Behnke et al. (2022) explores several multitask architectures, to support the QE task and discourage the model from learning the bias. In spite of their success on alleviating the bias in QE, the overall estimation performance is degraded. In other words, they mitigate the bias at the cost of QE performance.
In this work, we present a new strategy to mitigate the bias of QE and meanwhile improve QE performance. Our method is based on contrastive learning between clean and noisy sentence pairs. Firstly, we add noise to the target side of the parallel sentence pair. We corrupt the target sentence with hand-crafted rules, and then use another mono-
2175
lingual pre-trained model to restore it. Secondly, with the original sentence pair as the positive sample and the noisy sentence pairs as the negative samples, contrastive learning is assigned to the QE model as an auxiliary task. In this procedure, the proposed method reassures the QE model to focus on the bilingual alignment in addition to monolingual features, therefore mitigating the bias while upholding the QE performance.
We perform experiments on MLQE-PE dataset (Fomicheva et al., 2020) and WMT19 QE evaluation dataset (Fonseca et al., 2019), including both high-, medium- and low-resource language pairs. Our method is confirmed to improve the QE accuracy by a large as well as margin mitigate the bias. In particular, we further provide in-detail analysis about the bias of QE by creating two adversarial test sets. Examination on these data reveals that our method strikes a compromise between QE performance and bias mitigation, avoiding bias mitigation from overriding the QE objective.
Our contributions can be summarized as follows: 1. We propose to use contrastive learning as a regularizer for QE training, to mitigate the bias and focus the model on bilingual semantic alignment.
2. We propose to create effective negative samples for contrastive learning by firstly corrupting the reference text and then reconstructing it with a pre-trained model.
3. Our bias mitigation method improves the QE performance by a large margin, while previous method would lead to performance degradation.
4. We provide in-detail and informative analysis about the bias mitigation of QE by creating two adversarial test sets.

heading: 2 Related Work
text: In contrast to the automatic MT evaluation metrics which is good at system level, QE is usually conducted in either sentence-level or word-level. In this work, we mainly concentrate on sentence-level QE, where the translation quality is measured with different schemes, such as Human-Targeted Error Rate (HTER) (Snover et al., 2006) or Direct Assessment (DA) (Graham et al., 2015), and the QE model is supposed to provide a quality score for each MT output with its source alongside.
Quality Estimation was proposed as early as in 2004 (Blatz et al., 2004). After the emergence of BERT, Pre-Trained Models (PTMs) become popular in the area of QE (Fonseca et al., 2019). By pre-
training on massive multilingual text, PTMs have learned various linguistic knowledge, and can be adapted to quality estimation task without further adjustment. In WMT21 and WMT22 QE evaluation tasks (Specia et al., 2021; Zerva et al., 2022), nearly all top-performing team build the system on multilingual PTMs, e.g. XLM-RoBERTa (Conneau et al., 2020), Multilingual BERT (Devlin et al., 2019), etc. PTM-based method has become the defacto paradigm.
Despite the breakthroughs made in QE, the prediction of QE model is revealed to be biased to spurious features. Sun et al. (2020) showed that QE models have a tendency to over-rely on spurious correlations, which is partially due to skewed label distributions and statistical artifacts in QE datasets. In particular, they show the existence of a partial input bias, i.e. the tendency to predict the quality of a translation based on just the target sentence (Poliak et al., 2018). To this end, they annotate and release a new dataset, but as shown in subsequent results of Behnke et al. (2022), the bias is still striking in their newly-released dataset.
The most correlated work with us is Behnke et al. (2022), who also aims to investigate the bias mitigation of QE model. They find that the model as well as the annotators tend to over-rate the quality of fluent but inadequate translations. Accordingly, they propose four auxiliary tasks to perform bias mitigation, two approaches use additional data to inform and support the main task, while the other two are adversarial to discouraging the model from learning the bias. Although their methods could alleviate the bias, the estimation accuracy (measured with Pearson Correlation Coefficient) of the QE model is degraded in most cases.
Another correlated work is Huang et al. (2021), who firstly propose to apply contrastive learning on QE. But the contrastive learning is solely performed in a zero-shot manner, and they did not apply their method to mitigate the bias of QE.

heading: 3 Approach

heading: 3.1 Contrastively Regularized QE
text: To compromise between bias mitigation and quality estimation, we propose Contrastively Regularized QE (ConRegQE), as shown in Figure 2.
The core idea of our method is the contrast between clean sentence pairs (deemed as positive) and noisy sentence pairs (deemed as negative). We start from parallel sentence pairs, and introduce
noise to the target side to create semantic disalignment. Notice that the noising scheme can be applied to the same positive pair multiple times, leading to multiple negative pairs according to each positive pair. After that, the positive pairs and the negative pairs are all fed to the QE model, which is trained to distinguish them with InfoNCE (Oord et al., 2018) objective defined as:
LCL = es(q,k
+)/τ
es(q,k+))/τ + ∑n
i=1 e s(q,k−i )/τ
(1)
where τ is a temperature coefficient, n is the negative sample number, (q, k+) is the positive pair and (q, k−) is the negative pair, and s(·, ·) denotes the predicted logit for a sentence pair provided by the QE model as follows:
s(q, k) = FCCL(Φ(q, k)) (2)
where FCcl is a fully-connected layer, and Φ is the pre-trained XLM-RoBERTa.
This contrastive objective is jointly trained with the regression-style QE objective as follows:
LMSE = ∥FCreg(Φ(q, k))− l(q, k))∥2 (3)
Ltotal = LMSE + λ× LCL (4) where FCreg is a fully-connected layer, and l(q, k) denotes the human annotated score, and λ is a factor to balance the two loss functions. Notice we use two separate classification heads to perform the contrastive and regression training, to avoid them from disrupting each other.
Without this contrastive regularizer, the encoder would only accept one single src-mt pair as input, and is trained to assign a quality label in a regression style, in which it would leverage every possible feature to fit the annotation, such as monolingual complexity, fluency, etc. Since current PTMs are mostly trained with monolingual data, therefore it is much easier for the model to capture monolingual features than bilingual alignment, leading to the bias. But in the meantime, the features which could be utilized to finish estimation is quite limited, especially when only thousands of training samples are provided. Therefore, strictly filtering all spurious monolingual features would undoubtedly lead to performance degradation (as can be seen in the results of Behnke et al. (2022)). Our contrastive regularizer claims a decent compromise in this dilemma, and therefore making the most of bias mitigation as a supplement.

heading: 3.2 Negative Sample Generation
text: To create negative samples for contrastive learning, we propose the method of Denoising Reconstruction, as shown in Figure 2. Our method starts with parallel sentence pairs, and the reference is noised by the following two steps:
1. Randomly corrupt the reference sentence by the combination of different human-crafted rules, including masking, insertion, deletion, infilling and replacement, etc2;
2. Restore the corrupted reference with monolingual pre-trained models;
We introduce two kinds of pre-trained reconstructors, namely encoder-only model (such as BERT (Devlin et al., 2019)), and encoder-decoder model (such as BART (Lewis et al., 2020)) to recover the target sequence. Both models are pretrained with first corrupt the text and then reconstruct it, making them naturally adapted to perform the reconstruction. Since the input information is corrupted, the recovered version would unavoidably contain noise which is unaligned with the source sentence. Meanwhile, the reconstructions are generated by the language model, thus the results will not be unnatural or outrageous. This is in line with the real noise distribution. While most of previous works rely on hand-crafted rules or machine translation (Wu et al., 2020; Briakou and Carpuat, 2020; Tuan et al., 2021) to create negative samples for contrastive training in natural language processing, this does not apply to our scenario, since both rule-based corruption and MT decoding have specific patterns and can be easily detected3.
To further imitate the noise distribution, we
2More detailed illustration is presented in the Appendix A. 3An example is presented in the Table 8 of the Appendix.
resort to knowledge distillation (Kim and Rush, 2016) to transfer the decoding space of the to-beevaluated MT model to the reconstructor, as shown in Figure 3. We first use the MT model to translate text in the source language, and then the pre-trained reconstructor is further tuned on the generated target sequences. The generated sequence would contain the decoding patterns of the to-be-evaluated model, and after knowledge distillation, the reconstructor could introduce noise with more consistent distribution. This is also helpful to regularize the model to focus on quality-related features.

heading: 4 Experiments

heading: 4.1 Setup
text: We mainly work with the MLQE-PE dataset (Fomicheva et al., 2020), which formed the basis for the WMT21 QE evaluation task. Seven language pairs are involved, including high-, mediumand low-resource languages4. The translations were generated using Transformer-based Neural MT models, and each source sentence is accompanied with a human post-edited reference. For each language, train, dev and two test sets (Test20 and Test21) were annotated on two different scales:
• Task1: Direct Assessment (DA) Prediction;
• Task2: Human-Targeted Error Rate (HTER) Prediction;
We also experiment on the WMT19 QE dataset (Fonseca et al., 2019), which includes HTER prediction data for two language pairs5.
We mainly compare with the work of Behnke et al. (2022), which is build based on M-TransQuest (Ranasinghe et al., 2020), and explore the following four strategies to mitigate the QE bias:
• bilingual: train with different language pair (Romanian-English) which is less biased;
• augmented: train with additional translations, which are shuffled to form “bad” translations;
• adversarial: train to predict the score based on only target-input with gradient reversed;
• focal: train with revised debiased focal loss;
4English-German, English-Chinese, Romanian-English, Estonian-English, Russian-English, Sinhala-English and Nepali-English.
5English-German and English-Russian.
We also compare with two competitive systems of OpenKiwi (Kepler et al., 2019b) and COMET (Rei et al., 2020), both are based on multilingual pre-trained models. To make a fair comparison, we implement all systems based on the same pretrained model (XLM-RoBERTa-base or Multilin-
gual BERT) with their released codes6. We use monolingual BERT (Devlin et al., 2019) for the backbone of the encoder-style reconstructor7. For Chinese, we also tried encoder-decoder style pre-trained model CPT (Shao et al., 2021)8.
To apply knowledge distillation for the reconstructor, we randomly sample 500k sentences from WikiMatrix (Schwenk et al., 2019) for English and CC100 (Conneau et al., 2020) for other languages. Notice our proposed method only entails monolingual data, therefore we are able to perform knowledge distilation even for low-resource languages.
Pearson Correlation Coefficient (PCC) between the prediction and the human annotation is taken as the major metric, and Spearman’s Rank Corre-
6It should be addressed that we did not use any released checkpoint provided by these quality estimation systems, since we want to make a fair comparison in the same data setting, and it is not clear what data augmentation technique is used in training their checkpoints. We train all systems based on the same pre-trained model and the same data, and we use their default settings (we also tried to tune the hyper-parameters of their systems but found no gain). Therefore, our comparison is fair and can be used to verify the effectiveness of our proposed method.
7https://huggingface.co/{bert-base-cased, hfl/chinese-bertwwm-ext, dbmdz/bert-base-german-cased, DeepPavlov/rubert -base-cased}
8We also tried mBART (Liu et al., 2020), but to our surprise, the model can hardly perform complex reconstructions.
lation Coefficient (SRCC) is also reported. All experiments are run with five different random seeds and we report the averaged results.
The temperature τ in InfoNCE loss is set as 0.3, and each positive sample is contrasted with 20 negative samples. For more detailed settings about contrastive learning and negative sample generation, please refer to the Appendix A.

heading: 4.2 Main Results
text: As shown in Table 1 and 2, we can see that our proposed method could improve the estimation accuracy by a large margin, consistently among different language pairs and annotation flavors. On the contrary, the bias mitigation methods proposed by Behnke et al. (2022) could lead to little improvement or even degradation in most cases. This indicates that the biased features should not be harshly restricted or even ruled out, since the translation quality is a whole and can not be simply decoupled. In contrast, our method applies a softer restriction to the representation, focusing it on the semantic alignment while not directly disturbing the regression-style prediction, therefore making the most use of bias mitigation as a supplement.
We also report the model performance in crossannotation scenario, to demonstrate their robustness and generalizability. In MLQE-PE dataset, each sentence pair has two different quality annotations, namely DA (Task1) and HTER (Task2). While they focus on different aspects of translation quality, they are both evaluation metrics and are inherently correlated. Therefore, we believe a well-trained model on one annotation could also function on another annotation. We apply different models on the test set with different annotations, and the results are shown in Table 3.
As can be seen, our model improves the crossannotation robustness of both models on both tasks. By contrast on noised parallel sentences, our method force the model to focus on semantic alignment, making it more general in different quality annotations, while the baseline system relies too much on spurious monolingual features and can not generalize well. And the methods proposed by Behnke et al. (2022) again lead to degradation in most cases, showing that their methods are too restrictive and deviate from the QE objective.

heading: 5 Analysis and Discussion

heading: 5.1 QE model bias: an illustration
text: As discussed in previous sections, the major bias of QE model is heavily based on monolingual features (e.g. complexity and fluency), without modeling the bilingual alignment. We further investigate this issue by constructing two adversarial test sets on the basis of MLQE-PE dataset:
1) test-adv1 This adversarial test set is randomized by adjacent sample shuffling. We create this test set by two steps: i) Sort the src-mt pairs according to quality scores in ascending order, ii) Switch the srcs of every two adjacent pairs while keep the mt and quality score unmoved. In this case, all translation pairs are unrelated, therefore the QE results would be in random (with a minimum correlation with the quality score).
2) test-adv2 This adversarial test set is perfected with post-edit results. We create this test set by simply substitute the mt in test set with its corresponding post-edit. In this case, all translations could be regarded as fully fluent and adequate, and the QE score would possibly reach the maximum value (and also with a minimum correlation with the quality score).
We train the QE model on the original training
set and evaluate on three test sets, one original and two adversarial. As shown in Figure 5, the QE model could claim even higher correlation score on test-adv1, despite all sentence pairs are unrelated and the estimation results should have fallen into random. We attribute this to the fact that two adjacent pairs should have roughly the same complexity and fluency after sorting with respect to quality scores, which are captured as the major classification feature by the biased QE model. This demonstrates the QE model is biased towards monolingual features (complexity, fluency, etc) while ignoring the bilingual semantic alignment.
Meanwhile, the QE model could provide a strong correlation score on test-adv2, especially on TASK1 (84.25% on ENDE and 85.43% on ENZH). This demonstrates that the monolingual complexity is a major bias for QE model, since in test-adv2, all target sequence are fluent and adequate, and the only feature that can be utilized now is the complexity in both sides.
In a nutshell, the bias of QE can be deems as a multi-aspect notion influenced by a lot of factors, for example, the complexity of the syntactic structure, the amount of low-frequency words, the fluency of the target sequence, and so on. However, none of these monolingual factors has a casual effect on the translation quality. The QE model is expected to be able to handle such cases as the MT model provide a decent translation for a complicated sentence, or the translation result is fluent but unadequate and should be classified as low quality.

heading: 5.2 Compromise in bias mitigation
text: Based on the discussion in Section 5.1, we report the results on test-adv1 as a measurement of bias mitigation. We compare our methods with the methods proposed by Behnke et al. (2022), and
the results are shown in Table 4.
As can be seen, our method do mitigate the bias by a large margin. Although we do not achieve the minimal correlation compared with some versions of Behnke et al. (2022), we would like to deem this as a compromise between bias mitigation and estimation accuracy. Our model do not over emphasize bias mitigation and exclude the monolingual features since they (such as fluency) are important factors in translation quality. We verify this by adjusting the extent of bias mitigation with different λ in Equation 4, and the variation of PCC on the original and adversarial sets is shown in Figure 6.
As can be seen, as the correlation with adversarial set is decreasing, the correlation with the original set would increase first and then decrease. Bias mitigation, to a certain extend, is helpful to avoid overfitting and obtain higher accuracy, but too much bias mitigation would harm the modeling of monolingual featues and eventually do harm
to the estimation accuracy. We believe claiming a zero-correlation with our adversarial test set is not the final objective. Rather, the final objective of bias mitigation is also to improve the model performance, and our method is supplementary to achieving more accurate estimation, obtaining a compromise between bias mitigation and QE.

heading: 5.3 Contrastive learning vs. data augmentation
text: In contrastive learning, each sentence pair would be augmented with multiple negative samples, which may make people deem that it is the data augmentation rather than the contrastive objective taking effect. To verify the necessity of contrastive learning, we use the generated synthetic data directly as data augmentation on MLQE-PE Task2. The noised reference is deemed as synthetic mt, and the HTER score between mt and pe is calculated with the official provided scripts9, leading to 140K (src-mt-hter) triplets for each direction. Then the original training set is mixed with the synthetic data, to be used for regression-style training. Notice the original training set is upsampled to make sure the synthetic and real data have roughly the same amount.
As shown in Table 5, the results would be degraded if directly use the augmented data as the regression objective. This is because the subtle distribution produced by MT decoding and crowdsourced human annotation, which is hard to be imitated by automatic data augmentation methods. We can not create an unbiased objective for regression automatically, but the noised pair is undoutedly
9http://www.cs.umd.edu/ snover/tercom/
worse translation, therefore the learning objective of contrastive learning is unbiased. Another problem is, for other annotations such as DA, there is no automatic script to calculate the quality score. Despite QE being a generally-agreed data-sparse task, data augmentation is not so easy to be directly applied on it.

heading: 5.4 Different ways for negative sample generation
text: As discussed in Section 3.2, while most of previous works rely on hand-crafted rules or machine translation to create negative samples for QE, we propose to generate synthetic data by Denoising Reconstruction, both by encoder-only model and by encoder-decoder model. For both models, we choose to apply knowledge distillation, to transfer the noise pattern from the to-be-evaluated NMT model to the pre-trained reconstructor.
Table 6 provides a comparison of different negative sample generation methods. The results show that both rule-based and MT-decoded negative samples are disruptive and would lead to performance degradation, since both of them have specific patterns and can be easily detected (Examples are provided in Table 8 in the Appendix). Especially for MT-decoded samples, most of them are correct translations with different syntactic structures, or else to say, they are not really “negative”.
It is also noticed that for PTM-based negative samples, knowledge distillation plays an important
role. This is because different models have different decoding space, leading to different noise distribution. Without knowledge distillation, the decoding space of the reconstructor would deviate from the to-be-evaluated MT model, which would be utilized as spurious features for contrastive learning, leading to performance degradation.

heading: 6 Conclusion
text: In this paper, we propose to improve translation quality estimation with bias mitigation. We first use pre-trained model to generate contrast samples, and then the QE model is trained to distinguish positive and negative samples. While previous methods mitigate the bias at the cost of estimation accuracy, our method achieves a compromise between bias mitigation and quality estimation.
While current state-of-the-art QE models being proved to be biased to monolingual features, the bias could not be simple ruled out for the sake of overall estimation accuracy. In the future, we will dig deeper into this problem, to improve the robustness and generalizability of QE in real applications.
Limitations
Our work still has some limitations: 1) Due to the lack of research about the bias mitigation of QE, there is only one directly related work in this area, which serves as the main baseline in our experiments. Since the bias of QE is a conspicuous problem, we hope there will be more related work in the future. 2) Although our experiments are on WMT QE datasets, we do not implement the complicated data augmentation or model ensemble techniques as described in Specia et al. (2021) and Zerva et al. (2022), therefore our results can not compete with the best results of the WMT QE evaluation tasks. 3) Also, our method requires reference as the positive sample. Although most QE data includes reference, there are still chances that the QE data is annotated without the absence of reference, and our method would be hard to apply to such cases.

heading: Acknowledgements
text: This work is supported by National Key RD Program of China (2020AAA0108000), National Natural Science Foundation of China (62276077, U1908216), Key RD Program of Yunnan (202203AA080004) and Shenzhen College Stability Support Plan (No. GXWD20220811170358002). Muyun Yang is
also partially supported by a joint project with Global Tone Communication Technology Co., Ltd.

heading: A Hyperparameters of Contrastive Learning
text: Previous research on contrastive learning finds that the amount of negative samples has a significant impact on the contrastive learning performance (He et al., 2019; Chen et al., 2020). In contrastive learning, the positive sample is pushed apart from all negative samples, and introducing more contrast samples could help to learn a uniform representation space, and also possibly incorporating harder contrast to learn more complicated semantics. Therefore, previous research often set a large batch size (sometimes leveraging the memory bank) for contrast. Also, an adjustable temperature τ is also believed conducive to contrastive learning (Wang and Isola, 2020). A lower temperature value could generate peaky logit distribution and punish the model more on harder samples. We tune both hyperparameters on MLQE-PE Task2.
As shown in Figure 7, while too few negative samples would lead to performance degradation, the model could not get further improvement after more than 20 negative samples. We think this is because our carefully choreographed noising scheme, enabling us to introduce harder contrast samples without a large batch size. Besides, as shown in
Table 7, the temperature does not have a significant influence on the result. We think it is because we are using contrastive learning in a multi-task architecture, therefore the loss would not drastically change when tuning the temperature value. In the end, we decide to set negative sample number as 20 and temperature as 0.3 in all experiments.

heading: B Hyperparameters of Data Generation
text: Algorithm 1 Text Corruption Input: Input sentence x with N tokens, mask ratio
rm ∈ [0, 1], random ratio rr ∈ [0, 1], insertion ratio ri ∈ [0, 1], and deletion ratio ri ∈ [0, 1]. Output: Corrupted sentence x′. 1: Draw J text spans from x with totally M to-
kens, where M = N × rd. 2: for i = 1, 2, ..., J do 3: Delete i-th text span. 4: end for 5: Draw K positions from x, where K = (N +
1)× ri. 6: for i = 1, 2, ...,K do 7: Generate a random number f ∈ [0, 1]. 8: if f > rr then 9: Insert i-th position with MASK token.
10: else 11: Insert i-th position with a random token. 12: end if 13: end for 14: Draw L positions from x with totally M to-
kens, where M = N × rm. 15: for i = 1, 2, ..., L do 16: Generate a random number f ∈ [0, 1]. 17: if f > rr then 18: Replace i-th text span with MASK token. 19: else 20: Replace i-th text span with a random token. 21: end if 22: end for
In this section, we would elaborate on the detailed hyperparameters for the data generation. As depicted in Section 3.2, we use denoising reconstruction to create negative samples, where we first use rules to corrupt the sequence, and then use a pre-trained reconstructor to restore it.
For the corruption of the text, we use the combination of five rules, including masking, replacement, insertion, deletion and infilling. Detailed
corruption procedure is depicted in Algorithm 1. Notice “replacement” is actually masking with a random token, and "infilling" is actually insertion with MASK token.
We try out different combinations of hyperparameters on MLQE-PE Task2, and the results are shown in Table 10. As can be seen, both the insertion/deletion and the replacement/infilling operation is helpful, since they can generate more diverse noise compared with only masking. Also, when set the noise ratio too high or too low, the model performance would degrade, since too much noise would make the reconstructed text outrageous and deviate from real MT noise, while too little noise would make the reconstruction too easy and the generated negative samples might be actually positive.
Current pre-trained models are mostly based on subword segmentation. As discussed in previous research (Cui et al., 2021), corruption on whole word level might be more consistent with the semantic structure and therefore draw further gain. When performing masking, replacement and deletion operation, we try three corruption strategies on subword level, word level and span level respectively (with length drawn from a Poisson dis-
tribution). As shown in Table 9, the result is the best when performing corruption on subword level, which is beyond our expectation. It is possibly because subword-level corruption can generate more diverse noise, providing more contrast examples.
In a nutshell, when generating negative samples for contrastive learning, the primary concern is to keep the noise distribution both consistent and diverse.
C Is target fluency the largest bias?
Behnke et al. (2022) claims that the major bias in QE is partial input bias, where the model relies too much on target fluency. We think this claim is not
accurate, and to verify this, we conduct three sets of experiments on only the target side of the data.
1) train-mt: Train on the original training set and infer on the original test set (only mt);
2) train-mt-bow: Train on the Bag-of-Words style training set and infer on the original test set. We shuffle each mt sentence on token level, therefore the fluency information is excluded. An example is as follows:
mt A man is fishing on the bank .
mt-bow is bank a fishing on man the .
3) train-pe: Train on the pes of training set and infer on the original test set. We simply substitute the mt in training set with its corresponding pe.
To make the most of partial input, we use monolingual BERT model for German10 and Chinese11. As shown in Figure 8, the QE model could claim strong results on both mt-BOW and pe scenarios, in both cases fluency is excluded and can not be utilized as feature12. This again demonstrates that fluency is not the major factor when performing estimation. The estimation can still be performed when there is no fluency information. Besides, it can also be noticed that with the help of powerful monolingual pre-trained models, we can achieve comparable or even higher estimation accuracy solely relying on the target side.
To draw a conclusion, target fluency is a major bias, but not the major bias.
10http://huggingface.co/dbmdz/bert-base-german-cased 11http://huggingface.co/hfl/chinese-bert-wwm-ext 12The only exception is TASK2-enzh trained with pe, where the model can achieve little correlation on test set, which deserves our future exploration.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
3 A2. Did you discuss any potential risks of your work? Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: co/hfl/chinese-bert-wwm-ext 12The only exception is TASK2-enzh trained with pe, where the model can achieve little correlation on test set, which deserves our future exploration. ACL 2023 Responsible NLP Checklist