DSEA-KoBE: Quality Estimation Based on Distantly Supervised
Entity Alignment
Ming Zhu, Min Zhang, Junhao Zhu, Yanqing Zhao, Zhanglin Wu, Hao Yang, Song Peng, Shimin Tao{Zhuming47,zhangmin186,zhujunhao,zhaoyanqing,wuzhanglin,yanghao30,pengsong2,taoshimin}@huawei.com
Huawei Technologies Co., Ltd.
Beijing, China

ABSTRACTThe interpretability of quality estimation without reference has
always been a challenging issue in the evaluation of machine translation. Google’s KoBE [1] method, which evaluates translation quality based on the matching rate of bilingual entities, has provided
strong explainability. However, this approach faces challenges suchas varying granularity in bilingual entity recognition and low map
coverage, which limit its practical applicability. To address theselimitations, we propose a DSEA-KoBE method that replaces the
KoBE’s entity link module with the Distantly Supervised Entity
Alignment (DSEA) module. This distantly supervised approacheffectively mitigates the problems related to graph coverage andvarying granularity in bilingual entity recognition. We conduct
experiments using KoBE’s public data for English to Chinese (enzh) and Chinese to English (zh-en) translations, and the results
demonstrate the effectiveness of our proposed method.

KEYWORDS
quality estimation; interpretability; KoBE; DSEA-KoBE; distantly
supervised
ACM Reference Format:
Ming Zhu, Min Zhang, Junhao Zhu, Yanqing Zhao, Zhanglin Wu, Hao Yang,Song Peng, Shimin Tao. 2023. DSEA-KoBE: Quality Estimation Based on
Distantly Supervised Entity Alignment. In Proceedings of December 8–9,
2023 (International Joint Conference on Knowledge Graphs ). Tokyo, Japan,
5 pages.

1

INTRODUCTION

Machine translation quality evaluation is an important part of natural language processing, which can be mainly divided into referencebased quality assessment and reference-free quality estimation.
There are three ways for reference-based quality assessment, including N-gram-based similarity, edit distance matrix-based, andword embedding-based methods. BLEU[2], CHRF[3], and othermethods are based on N-gram. BLEU calculates the similarity bymeasuring the word-level N-gram overlap between machine translation and reference translation, while CHRF is a character-levelN-gram method. TER[4], WER[5], and PER[5] methods are typicaledit distance-based approaches, and the main difference among
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
International Joint Conference on Knowledge Graphs , Tokyo, Japan,© 2023 Association for Computing Machinery.

these three methods lies in the definition of "errors" and the considered types of operations. TER considers insertion, deletion, substitution, and shift operations, and WER considers insertion, deletion,and substitution; in contrast, PER only considers insertion anddeletion. Word embedding-based methods include BERTScore[6],BLEURT[7], COMET[8], YiSi[9], etc. BERTScore, BLEURT, and YiSi
analyze and align at the lexical level between reference translationand machine translation using pre-trained models to calculate thesimilarity, while COMET is an end-to-end scoring model trainedwith manually scored data.
Compared with quality assessment with reference, quality estimation without reference is more challenging. Currently, quality
estimation without reference mainly uses CometKiwi[10], KoBE,
KG-BERTScore[11] and other methods. KoBE is more interpretable
than CometKiwi and KG-BERTScore, and is easier to apply. Methods
such as CometKiwi and TransQuest[12] use bilingual translation
data with Direct Assessment (DA) scores to directly train a scoringmodel in an end-to-end manner. Those methods are feasible but not
explainable. In contrast, the KoBE method creatively converts thetranslation quality problem into the translation accuracy problem
of entities. KoBE identifies bilingual entities, links them to the same
multilingual entity library, and then determines whether the entity
translation is accurate by comparing the IDs of the linked entities,
which shows a high degree of explainability. However, we find thatin existing entity recognition, there is no uniform granularity crosslanguage entity recognition dataset, so the granularity of entities
varies with languages. As shown in Figure 1, different entity granularities result in differences in entity links. In addition, the coverageof the multi-language knowledge graph is not high. Those facts
result in incorrect entity links. To address this issue, we propose aDistantly Supervised Entity Alignment (DSEA) model and apply
it to the entity matching module of KoBE. Experiments show thatour proposed DSEA-KoBE method can solve this problem and ishighly correlated with human scores.
Note that our DSEA method can also be used to build a crosslanguage graph. The knowledge graph is a highly explainable source
for KoBE; however, entity linking at different granularities also
introduces semantic models with lower explainability. As our model
does not involve entity linking, we believe that our approach iscompetitive with KoBE in terms of explainability.
Our main contributions include:• We find two problems of KoBE: (1) Different languages havedifferent granularities of entity recognition. (2) The entity
coverage of its knowledge graph is low, and test corpora
cannot be well covered.
• We propose DSEA-KoBE, which is a quality estimation method
based on KoBE. DSEA-KoBE alleviates the problem of map

Figure 1: Problems caused by different entity granularities
The real label of’救援队’ is Rescue team.

coverage and the problem of different granularity of entity
recognition in different languages.
• We perform experiments on KoBE’s zh-en and en-zh datasetsand validate our method.

2

METHODS

Figure 2 shows the overall architecture of DSEA-KoBE. It consistsof three modules: Named Entity Recognition (NER), DSEA, andscore.

2.1

NER Module

Algorithm 1: DSEA-KoBE evaluation process
Input : all source sentences 𝑠𝑘 ∈ 𝑆 and machinetranslations 𝑡𝑘 ∈ 𝑇 of 𝑛 sentence pairs
Output : a system-level score 𝐹// 𝑚𝑎𝑡𝑐ℎ(𝑠𝑟𝑐, 𝑚𝑡) the number of entities thatare successfully matched.
1 𝑚𝑎𝑡𝑐ℎ(𝑠𝑟𝑐, 𝑚𝑡) = 0 for each sentence pair {𝑠𝑘 ,𝑡𝑘 }∈ {S,T} do// 𝑒𝑛𝑡𝑖𝑡𝑦𝑠𝑘 indicates entities in the source
sentence 𝑠𝑘 , 𝑒𝑛𝑡𝑖𝑡𝑦𝑠𝑘𝑖 indicates the ith
entity in the 𝑒𝑛𝑡𝑖𝑡𝑦𝑠𝑘
2
𝑒𝑛𝑡𝑖𝑡𝑦𝑠𝑡𝑘𝑖 = 𝐴𝑙𝑖𝑔𝑛(𝑒𝑛𝑡𝑖𝑡𝑦𝑠𝑘𝑖 , 𝑡𝑘 )𝑒𝑛𝑡𝑖𝑡𝑦𝑠′𝑘𝑖 = 𝐴𝑙𝑖𝑔𝑛(𝑒𝑛𝑡𝑖𝑡𝑦𝑠𝑡𝑘𝑖 , 𝑠𝑘 )3
if 𝑒𝑛𝑡𝑖𝑡𝑖𝑒𝑠𝑠′𝑘𝑖 == 𝑒𝑛𝑡𝑖𝑡𝑖𝑒𝑠𝑠𝑘𝑖 then4
𝑚𝑎𝑡𝑐ℎ(𝑠𝑟𝑐, 𝑚𝑡)+ = 1
5else
6
do nothing
7
end
8 end
// 𝑐𝑜𝑢𝑛𝑡 (𝑒𝑛𝑡𝑖𝑡𝑖𝑒𝑠𝑆 ) is the number of source
entities

NER is the basis of the entire DSEA-KoBE process, and all scores
are based on the matching rate of entities. To better compare ourmethod with KoBE, we directly use KoBE entity mentions and thenperform entity alignment based on these entity mentions.

2.2

DSEA Module

Figure 3 shows the architecture of the DSEA module. In this module, we transform the entity alignment task into an NER task thatextracts the corresponding entity in the parallel language for agiven entity. We follow the basic model of CNN-Nested-NER[13],and change the input form for relation extraction.
We combine the source entities with the target sentences, and implement cross-language entity extraction using the multi-language
capability of the XLM[14] pretrained model. For the alignment
model, we use some bilingual entity pairs in the wiki and annotate
parallel corpus through distant supervision.

2.3

Score Module

The scoring process is shown in Algorithm 1. First, we obtain thesource entity through NER. Note that for better comparison with
KoBE, we use the KoBE entity. We combine the source entity withthe translation as the input of the alignment model, and then obtainthe translation entity through the alignment model. Consideringthat the model based on distant supervision has some misjudgment,
we use reverse verification technology, i.e., using the obtained translation entity and the source sentence as the input of the alignment
model, to find the corresponding entity of the translation entity
in the source sentence. We check whether the inferred entity is
the same as the entity identified by NER (only the same text is
considered, and the index information is not considered). If the twoentities are the same, the matching is successful. The final score is
obtained as follows:
𝑚𝑎𝑡𝑐ℎ(𝑠𝑟𝑐, 𝑚𝑡)
𝐹𝑠𝑐𝑜𝑟𝑒 = Í𝑛𝑐𝑜𝑢𝑛𝑡 (𝑒𝑛𝑡𝑖𝑡𝑖𝑒𝑠𝑠𝑘 )𝑘=1

9𝑚𝑎𝑡𝑐ℎ (𝑠𝑟𝑐,𝑚𝑡 )𝐹𝑠𝑐𝑜𝑟𝑒 = Í𝑛 𝑐𝑜𝑢𝑛𝑡 (𝑒𝑛𝑡𝑖𝑡𝑖𝑒𝑠 )
𝑆
𝑘=1

10

3

EXPERIMENTAL SETUP

In order to comprehensively evaluate the effectiveness of our proposed method, we conduct experiments on two distinct tasks: entityalignment and system-level Quality Estimation (QE) sharing. For
the entity alignment task, we employ two datasets to thoroughly
assess the model’s performance. The first dataset, referred to as
the distant supervision dataset, is derived by mining 100,000 crosslanguage entity pairs from the Wikipedia corpus1 . We extensively
annotate this dataset using a large parallel corpus2 , obtaining a totalof 50,000 instances. Out of these instances, 40,000 are designatedas the training set, while the remaining 10,000 are allocated fordevelopment and testing purposes. The second dataset, denoted as
KoBE-100, consists of a random sample of 100 data instances from
KoBE’s Chinese and English data. These instances are manuallylabeled, yielding a total of 734 instances with alignment relationships.
To evaluate the proposed DSEA method, we compare its performance against two baseline methods: Fast-Align[15] and FastAlign (boundary)[16]. The Fast-Align method represents a standardalignment algorithm, while Fast-Align (boundary) incorporates thebilingual entity boundary information into the alignment process.
Additionally, we compare our method against the entity linking
method employed by KoBE itself, utilizing the KoBE-100 dataset.
Moving on to the system-level QE task, we conduct experiments
using the zh-en and en-zh pairs extracted from the dataset provided
by the WMT19 shared task. This dataset serves as the basis for
1 https://dumps.wikimedia.org/
2 https://www.statmt.org/wmt18/translation-task.htmldownloadsrc_sentence: 校内科学家正致力研究母非洲狮的生殖系统。

DSEA

tgt_entity':[生殖系统-reproductive system
科学家-scientists
非洲狮-African lions]lines:[[生殖系统,
,The scientists of the university were ... ],[科学家,'
', The scientists of the university were ...],[非洲狮, '
', The scientists of the university were ...] ...]

lines:[[reproductive system,
, 校内科学家正致力研究母 ... ],[scientists,'
', 校内科学家正致力研究母 ...],[African lions, '
', 校内科学家正致力研究母 ...]]

src_entity: [生殖系统、科学家、非洲狮]DSEA

src_entity':[reproductive system-生殖系统，scientists-科学家，
African lions-非洲狮]

NER
SCORE
tgt_sentence: The scientists of the university were committing themselves tostudying the reproductive system of female African lions.

src_sentence: 校内科学家正致力研究母非洲狮的生殖系统。
QE_score

Figure 2: DSEA-KoBE
methodPrecision Recall
F1
Fast-Align
0.613
0.568 0.590
Fast-Align(boundary)
0.765
0.651 0.703
DSEA
0.961
0.961 0.961
Table 1: Entity Alignment F1 on Distantly Supervised Dataset

methodPrecision Recall
F1
Fast-Align
0.660
0.375 0.478
Fast-Align(boundary)
0.782
0.451 0.572
KoBE
0.960
0.560 0.694
DSEA
0.923
0.914 0.918
DSEA(validation)
0.961
0.770 0.845
Table 2: Entity Alignment F1 on KoBE-100 dataset

Figure 3: DSEA: Distantly Supervised Entity Alignment

evaluating the performance of our proposed method in the context
of system-level QE.
Our model uses the CNN-Nested-NER architecture, and the pretraining model is XLM-R-Large. Five epochs are trained. The learning rate is 7e-6, cnn_dim is 200, and biaffine_size is 400. n_head is
4 and batch_size is 32.

4

RESULTS

Table 1 presents the results obtained on the distantly supervised
dataset, indicating that the Fast-Align method achieves an F1 scoreof 0.590. We observe that the Fast-Align method may introduce
entity alignment errors due to potential misalignment of subwords

Metricen-zh zh-en
BLEU
0.901 0.899
YiSi-2-0.097 0.94
YiSi-2_srl-0.118 0.947
BERTScore-0.127 0.728
KoBE
0.216 0.907
KG-BERTScore 0.077 0.908
DSEA-KoBE
0.375 0.912-(validation)
0.105 0.613
Table 3: System-level Pearson correlation with human judgements for zh-en and en-zh language pairs from the WMT19
metrics-without-references shared task. Best QE results aremarked in bold.

within the entities. To address this limitation, we propose an approach that incorporates the entity word boundaries in the target
language as alignment constraints. This modification leads to a significant improvement, with the F1 score increased to 0.703. Furthermore, we evaluate our method on the manually labeled KoBE-100src
未通过备案审核的班次不得招生培训腾讯新六大事业群mt
The classes that have notpassed the archival filingand examination shall not
enroll for training.
Tencent six new businessgroups.

KoBE[班 次, /m/0gz84v],[training, /m/014jg3],[archival, /m/01tygv]

DSEA-KoBE[班次, classes]

Golden[班次, classes][腾 讯, /m/0403vtn], [腾 讯, Tencent], [腾 讯, Tencent],[事
业群, [事 业 群, business [事 业 群, business/m/025w401], [Ten- groups]
groups]cent, /m/0403vtn],[businessgroups,/m/0d06sy]
Table 4: Comparative example of matching results between KoBE and DSEA-KoBE

dataset, as depicted in Table 2. The entity link method employed by
KoBE achieves a high precision of 0.960. Notably, precision appears
to be more accurate than the recall in this case. However, the entity
link method only achieves a recall of 0.560. Through analysis, we
identify that the discrepancy in bilingual entity granularity is a key
contributing factor to this problem, as exemplified in Figure 1. In
contrast, the proposed DSEA method achieves higher precision andrecall rates, benefiting from its ability to handle varying bilingual
entity granularities.
To further improve entity alignment accuracy, we introduce
a DSEA(validation) method based on reverse verification. Thismethod leverages backtracking from the identified target entities,
improving the alignment accuracy. DSEA(validation) achieves aprecision of 0.961, surpassing the KoBE’s entity-linking method,while considerably outperforming the entity-linking method inrecall.
In our final set of experiments, we replace KoBE’s entity link
module with our DSEA and evaluate its performance on the zh-en
and en-zh datasets of the reference-free system-level QE sharedtask from WMT19[17]. DSEA-KoBE surpasses KoBE in both zh-en
and en-zh settings, as illustrated in Table 3. However, it is noteworthy that despite KoBE’s lower recall rate, its method remains
highly competitive in system-level QE. To further investigate this,we analyze KoBE’s scoring mechanism and propose that the strong
correlation between KoBE and DA scores in system-level QE relieson the high precision of KoBE’s entity link methods. Even witha lower recall rate, the relative recall rate of machine translationfrom different systems tends to be consistent within the same domain data, thereby preserving the relative ranking in system-level
scores. Consequently, we conduct additional experiments without
reverse validation, confirming that the correlation between QE
scores and DA scores is highly dependent on precision of entity
matching. Although the recall rate has a minimal impact on thePearson correlation of the DA score in system-level QE, it does influence the credibility of the score. When the number of alignment
entities to be recalled is small, it becomes challenging to assess thereasonableness of the recall rate.
We present two examples in Table 4, showcasing KoBE’s entity
link example and our DSEA example. In Example 1, we observethat the term "班次" is identified as an entity in Chinese, whereas
it is not recognized as an entity in English. This difference in entity
granularity across languages accounts for the disparity. Instead,two additional entities are identified in English. In Example 2, wefind that both the DSEA and KoBE’s entity links successfully match"腾讯" and "Tencent" However, the entities "事业群" and "businessgroup" are not included in the knowledge graph. Consequently, theentity link associates them with a similar entity, although the link’s"id" reveals that the pair of entities does not match. Conversely, ourproposed DSEA method aligns these two entities. These findings
indicate the advantageous aspects of our approach compared toentity linking in KoBE.

5

CONCLUSION

In this paper, we analyze the shortcomings and advantages of
the KoBE method, and propose a distantly supervised alignment
method, which can avoid the difference of entity granularity among
different entities to a certain extent, and mitigate the problem of low
map coverage. Moreover, we analyze the effect of entity alignment
accuracy and recall rate on QE scores, and verify our conjecture.
In our method, the position of entity alignment is not considered.
In the future work, we will consider using supervised methods toimplement entity alignment with position information.

REFERENCES[1] Zorik Gekhman, Roee Aharoni, Genady Beryozkin, Markus Freitag, and Wolfgang
Macherey. Kobe: Knowledge-based machine translation evaluation. In Findingsof the Association for Computational Linguistics: EMNLP 2020, pages 3200–3207,
2020.
[2] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method
for automatic evaluation of machine translation. In Proceedings of the 40th annual
meeting of the Association for Computational Linguistics, pages 311–318, 2002.
[3] Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In
Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–
395, 2015.
[4] Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John
Makhoul. A study of translation edit rate with targeted human annotation. In
Proceedings of the 7th Conference of the Association for Machine Translation in theAmericas: Technical Papers, pages 223–231, 2006.
[5] Franz Josef Och. Minimum error rate training in statistical machine translation.
In Proceedings of the 41st annual meeting of the Association for Computational
Linguistics, pages 160–167, 2003.
[6] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert. In International Conference on
Learning Representations, 2019.
[7] Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics
for text generation. In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 7881–7892, 2020.
[8] Craig Stewart, Ricardo Rei, Catarina Farinha, and Alon Lavie. Comet-deploying
a new state-of-the-art mt evaluation metric in production. In AMTA (2), pages
78–109, 2020.

[9] Chi-kiu Lo. Yisi-a unified semantic mt quality evaluation and estimation metricfor languages with different levels of available resources. In Proceedings of the
Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1),
pages 507–513, 2019.
[10] Nuno M. Guerreiro Chrysoula Zerva Ana C. Farinha Christine Maroti José G. C.
de Souza Taisiya Glushkova Duarte M. Alves Alon Lavie Luisa Coheur André F.
T. Martins Ricardo Rei, Marcos Treviso. Cometkiwi: Ist-unbabel 2022 submission
for the quality estimation shared task. In Proceedings of the Seventh Conference
on Machine Translation (WMT), 2022.
[11] Junhao Zhu Zhanglin Wu Hao Yang Song Peng Shimin Tao Ming Zhu, Min Zhang.
Kg-bertscore: Incorporating knowledge graph into bertscore for reference-freemachine translation evaluation. In Proceedings of the 11th International Joint
Conference on Knowledge Graphs, 2022.
[12] Tharindu Ranasinghe, Constantin Orǎsan, and Ruslan Mitkov. Transquest: Translation quality estimation with cross-lingual transformers. In Proceedings of the
28th International Conference on Computational Linguistics, pages 5070–5081,
2020.
[13] Xiaonan Li Xipeng Qiu Hang Yan, Yu Sun. An embarrassingly easy but strong
baseline for nested named entity recognition. In Proceedings of the 61st Annual
Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),

pages 1442–1452, 2023.
[14] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer,
and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale.
In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 8440–8451, 2020.
[15] NA Smith C Dyer, V Chahuneau. A simple, fast, and effective reparameterizationof ibm model 2. In Proceedings of the 2013 Conference of the North American Chapterof the Association for Computational Linguistics: Human Language Technologies,
pages 644–648, 2013.
[16] Min Zhang, Song Peng, Hao Yang, Yanqing Zhao, Xiaosong Qiao, Junhao Zhu,
Shimin Tao, Ying Qin, and Yanfei Jiang. Entityrank: Unsupervised mining ofbilingual named entity pairs from parallel corpora for neural machine translation.
In 2022 IEEE International Conference on Big Data (Big Data), pages 3708–3713,
2022.
[17] Erick Fonseca, Lisa Yankovskaya, André FT Martins, Mark Fishel, and ChristianFedermann. Findings of the wmt 2019 shared tasks on quality estimation. In
Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared
Task Papers, Day 2), pages 1–10, 2019.