言語処理学会 第30回年次大会 発表論文集（2024年3月）

ChatGPT as a Translation Engine: A Case Study on
Japanese-English
Vincent Michael Sutanto1 Giovanni Gatti De Giacomo1 Toshiaki Nakazawa2 Masaru Yamada3
1 Yaraku, Inc. 2 The University of Tokyo 3 Rikkyo University{vincent, giovanni}@yaraku.com, nakazawa@nlab.ci.i.u-tokyo.ac.jp,
masaru.yamada@rikkyo.ac.jp

Abstract
This study investigates ChatGPT for Japanese-English
translation, exploring simple and enhanced prompts andcomparing against commercially available translation engines. Performing both automatic and MQM-based human evaluations, we found that document-level translation outperforms sentence-level translation for ChatGPT.
On the other hand, we were not able to determine if enhanced prompts performed better than simple prompts in
our experiments. We also discovered that ChatGPT-3.5
was preferred by automatic evaluation, but a tradeoﬀ exists
between accuracy (ChatGPT-3.5) and ﬂuency (ChatGPT4). Lastly, ChatGPT yields competitive results against two
widely-known translation systems.

1

Introduction

Recently, ChatGPT has emerged as a versatile tool,ﬁnding applications in several domains due to its multifunctional capabilities. Beyond its extensive utility, ChatGPT extends its prowess to include translation tasks, showcasing its adaptability in bridging language barriers [1].
Interestingly, ChatGPT, as a machine translation tool,oﬀers more than just standard translation; it can be further
customized by ”enhancing” the prompt, producing resultsthat are preferred by professional translators [2]. Thisfeature empowers users to curate translations that alignmore precisely with their intended domain, purpose andtone [2] [3].
However, a comprehensive investigation into the useof ChatGPT as an MT system is still required to unveil
its applicability. Several questions remain unclear: doesa general-purpose model like ChatGPT perform as well
as specialized translation engines trained speciﬁcally fortranslation tasks? Is translating the entire document atonce better than translating sentence by sentence? How do
simple and enhanced prompting techniques aﬀect the translation results? Is there any discernible diﬀerence between
ChatGPT-3.5 and ChatGPT-4? Answering these types ofquestions will help us understand and provide valuable insights into ChatGPT’s role in translation technology.
In this paper, we employed ChatGPT-3.5 and ChatGPT-4
APIs 1）to answer the previously mentioned research questions. Additionally, we evaluate the performance against
the two most widely used commercial systems for JA-EN
translation. Our investigation reveals the following:
1. ChatGPT makes better translations when it handles
whole documents instead of sentence by sentence,
likely due to better context awareness as a translation
system.
2. We could not ﬁnd conclusive evidence with our experiments that enhanced prompts lead to higher qualitytranslations. However, this could be due to the smallscale of our human evaluation and the bias towards
gold data in the automated evaluations. Therefore, we
believe larger scale human evaluations are required toaccurately assess the quality diﬀerence.
3. ChatGPT-3.5 emerged as the favored model compared
to ChatGPT-4 in automatic evaluation, but humanevaluation reveals that the end user can choose between accuracy (ChatGPT-3.5) or ﬂuency (ChatGPT4).
4. For the JA-EN pair translation, both ChatGPT 3.5
and 4 performed competitively against two commercial MT systems, solidifying their position as viabletranslation system options.

1） gpt-3.5-turbo and gpt-4 models― 2096 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 1 Simple Prompt for ChatGPT
Translate this document from <src language> to <tgt
language>:<text>

2.4

Human Evaluation

We also conducted a human evaluation to appraise machine translation quality using the Multidimensional Qual-

Table 2

Enhanced Prompt for ChatGPTYou are a translation engine. Perform the following
steps carefully:
1. - Translate the following <src language> document
delimited by triple backticks to <tgt language>
2. - The translation is expected to be in the ﬁeld of<ﬁeld>.
3. - The expected output is expected to have <style>style.
4. - Change the translation tone into <tone>.
“‘<text>“‘
After ﬁnishing those steps, return the ﬁnal result only!

ity Metrics (MQM) framework2）. The evaluation was conducted by two professional translators, both with background and experience in JA-EN translation.

For thehuman evaluation, one sample from the document-leveltranslation of each of the following datasets were used:
ParaNatCom [4], Novels [6], and WMT News [8].
To facilitate the annotation process, we designed an evaluation tool for the MQM framework that we are open sourcing3）. We hope that the tool will aid researchers and practitioners in conducting similar human evaluations for theirmachine translation studies. Finally, the tool also comes
pre-conﬁgured with categories and weights selected by lin-2

Methodology

guists for evaluating both Japanese and English sentences.

2.1

Datasets3

We used several publicly available JA-EN datasets forour experiments: ParaNatCom [4], FLORES [5], Novels

Results and Discussion

3.1 Document-Level vs.
Translation

Sentence-Level[6], KFTT [7], and WMT News [8]. These datasets were

From Table 3, we calculate percentages indicating whenhandpicked to investigate translation quality on a variety

the document-level score exceeds the sentence-level score.

of domains, lengths, and styles. Due to budget constraints

This is done by counting where Doc is greater thanfor the API, we sampled ﬁve documents from each dataset

the Sent column for each row and metric. Speciﬁcally,

for the automatic evaluation. Additional details about the

60% favor document-level in BLEU, 100% in COMET,

datasets are provided in Appendix A.

while DA-BERT dissents with only 15% preferring the

2.2

document-level. Thus, our automatic evaluation indicates

Promptingthat document-level is better than sentence-level for Chat-

In this work, we investigated two prompting strategies:

GPT. This could be attributed to the fact that document-

Simple and Enhanced prompts. The Simple prompt simply

level translation preserves context more eﬀectively whencompared to sentence-level translation (See Appendix C).

directs ChatGPT to translate text based on source and target
languages (Table 1). On the other hand, the Enhanced
prompt, motivated by [2], instructs ChatGPT to consider
category, style, and tone as well as the source and target
languages for a more customized output (Table 2).

3.2

Simple vs. Enhanced Prompt

Similar to Subsection 3.1, we calculate the percentages where the enhanced prompt outperforms the simple

With the help of professional translators, we assigned

prompt. However, as the results vary, we cannot make a

distinct categories, styles, and tones to each dataset, ensur-

direct conclusion. BLEU and COMET remain neutral at

ing the translation output aligns with the speciﬁc charac-

50%, while DA-BERT indicates that the enhanced prompt

teristics of each dataset (Appendix B).

is better (60%). We observe that enhanced prompts can

2.3

cause the translation output to deviate from the gold data.

Automatic Evaluation

This may lead to lower scores, even for more appropriate

We computed three commonly used automatic evaluation metrics to measure overall translation quality: BLEU[9], COMET [10], and DA-BERT [11].

translations, simply because it diﬀers from the reference.
2） https://themqm.org/
3） https://github.com/yaraku/he-tool― 2097 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 3

DatasetParaNatCom

FLORES

KFTT

NovelsWMT News

BLEU, COMET, and DA-BERT results per dataset (higher is better).
BLEU
COMET
DA-BERT
System
Doc
Sent
Doc
Sent
Doc
Sent
ChatGPT-3.5 Simple
36.39 36.07 0.906 0.903 0.190 0.193
ChatGPT-4 Simple
37.55 38.09 0.907 0.903 0.181 0.185
ChatGPT-3.5 Enhanced 37.64 34.11 0.907 0.905 0.178 0.200
ChatGPT-4 Enhanced
38.25 36.45 0.901 0.891 0.184 0.199
Commercial System A
49.14 49.32 0.911 0.912 0.172 0.156
Commercial System B
47.60 47.76 0.911 0.911 0.148 0.148
ChatGPT-3.5 Simple
31.82 33.70 0.903 0.884 0.198 0.180
ChatGPT-4 Simple
31.82 35.43 0.904 0.870 0.187 0.188
ChatGPT-3.5 Enhanced 34.07 27.84 0.899 0.858 0.187 0.211
ChatGPT-4 Enhanced
29.28 27.52 0.891 0.856 0.194 0.214
Commercial System A
30.19 32.62 0.895 0.899 0.221 0.199
Commercial System B
41.43 32.94 0.893 0.773 0.176 0.217
ChatGPT-3.5 Simple
18.52 16.69 0.846 0.835 0.031 0.026
ChatGPT-4 Simple
17.52 18.51 0.843 0.836 0.018 0.024
ChatGPT-3.5 Enhanced 17.79 14.60 0.848 0.837 0.024 0.062
ChatGPT-4 Enhanced
16.62 15.23 0.845 0.830 0.018 -0.002
Commercial System A
19.51 19.36 0.842 0.840 0.027 0.062
Commercial System B
20.32 20.23 0.844 0.843 0.062 0.058
ChatGPT-3.5 Simple
12.79 13.26 0.860 0.840 0.272 0.274
ChatGPT-4 Simple
14.65 14.19 0.857 0.837 0.271 0.272
ChatGPT-3.5 Enhanced 13.69 10.48 0.865 0.787 0.274 0.297
ChatGPT-4 Enhanced
14.90 12.07 0.829 0.799 0.272 0.279
Commercial System A
15.49 12.97 0.864 0.855 0.280 0.293
Commercial System B
14.64 14.85 0.847 0.845 0.267 0.268
ChatGPT-3.5 Simple
25.84 26.69 0.819 0.819 0.185 0.199
ChatGPT-4 Simple
26.87 28.05 0.820 0.819 0.187 0.191
ChatGPT-3.5 Enhanced 24.06 25.30 0.822 0.816 0.186 0.198
ChatGPT-4 Enhanced
17.55 16.96 0.800 0.787 0.139 0.176
Commercial System A
16.75 24.53 0.702 0.805 -0.047 0.174
Commercial System B
41.02 41.55 0.832 0.832 0.189 0.188

and sentence-level.

Table 4

MQM scores for each system (lower is better).
SystemMQM Accuracy Fluency
ChatGPT-3.5 Simple
84.50
55.67
26.00
ChatGPT-4 Simple
95.83
82.67
11.83
ChatGPT-3.5 Enhanced 83.17
59.67
17.00
ChatGPT-4 Enhanced
103.2581.00
15.50
Commercial System A
96.25
57.00
30.83
Commercial System B
83.67
63.33
10.00

In terms of automatic evaluation,

we ﬁnd that ChatGPT-4 is not superior to ChatGPT-3.5.
Speciﬁcally, for document-level, COMET and DA-BERT
preferred ChatGPT-3.5, at 30% and 40% respectively,
while BLEU remained neutral at 50%.

Similarly, forsentence-level, BLEU preferred ChatGPT-4 at 80%, while
COMET and DA-BERT preferred ChatGPT-3.5, at 40%

Additionally, even with the human evaluation scores weand 20% respectively.

could not ﬁnd conclusive evidence that enhanced prompts

For the human evaluation (Table 4), the overall MQM

produce higher quality translations (See Table 4). Thisscores align with the ﬁndings in automatic evaluation:

could be due to the small-scale of our human evaluation

higher scores were observed for ChatGPT-4, suggesting

and the bias towards gold data in the automated evaluations.
Therefore, we believe larger scale human evaluations arethat it is not superior to ChatGPT-3.5. Moreover ChatGPT3.5 is better in terms of accuracy, meaning that the trans-

required to accurately answer the question of simple vs.

lation is reﬂecting the source text better.

enhanced prompts.

ChatGPT-4 oﬀers better ﬂuency, meaning that it could

3.3

understand. However, the decision of which one is bet-

Conversely,

generate translations that feel more native and easier to

ChatGPT-3.5 vs. ChatGPT-4

Once again, we compute the percentages where
ChatGPT-4 outperforms ChatGPT-3.5 for both document

ter depends on the preferences of the end users. For tasks
where conveying information without ambiguity is crucial,― 2098 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 5

Average score of ChatGPT vs Commercial MT Systems
Document Level
Sentence Level
SystemBLEU COMET DA-BERT BLEU COMET DA-BERT
ChatGPT-3.5 Simple
25.07
0.867
0.175
25.28
0.856
0.174
ChatGPT-4 Simple
25.68
0.866
0.169
26.85
0.853
0.172
ChatGPT-3.5 Enhanced 25.45
0.868
0.170
22.47
0.840
0.193
ChatGPT-4 Enhanced
23.32
0.853
0.161
21.65
0.832
0.173
Commercial System A
26.21
0.843
0.130
27.76
0.862
0.177
Commercial System B
33.01
0.865
0.169
31.46
0.841
0.176

ChatGPT-3.5 might be more appropriate. On the other
hand, for translations that include creative writing, e.g. advertisements, where a natural ﬂow is essential, ChatGPT-4

4

Conclusion
In our study comparing ChatGPT-3.5 and ChatGPT-4

may be the preferred choice.

for JA-EN translation, using simple and enhanced prompts,

3.4

we uncovered key insights. Firstly, translating entire doc-

Comparison to Commercial MT Systems

We averaged each metric across all datasets and organized it in Table 5. With document-level translation, we observe that all ChatGPT settings surpass System A’s scores,
while also performing on par or better than System B in
COMET and DA-BERT. On the other hand, the BLEU
scores show the reverse trend, with all ChatGPT settings
performing worse than both commercial systems. At thesentence-level, we found similar trends to the ones observed at the document-level. With BLEU, ChatGPT performs worse than both System A and System B. Conversely,
all variants of ChatGPT perform competitively against bothuments proved more eﬀective than translating sentence bysentence, potentially due to enhanced contextual preservation. Secondly, the question of whether enhanced prompts
are superior remains uncertain, as both automated and human evaluations provided inconclusive results. Thus, weleave it open for future researchers to answer this question through larger scale experiments. Thirdly, automaticand human evaluations indicated an overall preference for
ChatGPT-3.5, but we also point out that the end user may
want to tradeoﬀ between accuracy (ChatGPT-3.5) and ﬂuency (ChatGPT-4). Finally, in JA-EN translation, all investigated ChatGPT settings proved to be strong competitors
against two commercial MT systems in both automatic andcommercial systems in COMET and DA-BERT.
Overall, ChatGPT shows impressive translation capa-

human assessments.

bilities with two of the three evaluation metrics investigated, when compared to commercial translation engines.
Additionally, this interpretation is also supported by the
MQM scores (Table 4), where three of the four investigated ChatGPT settings perform better than System A andcompetitively against System B. Speciﬁcally, ChatGPT-3.5
achieved better accuracy scores than System B. In terms of
ﬂuency, all ChatGPT settings achieved scores within therange of the well-regarded translation engines. These are
evidence that ChatGPT, with scores on par with commercial translation systems, can be used to generate translations
that remain faithful to the source text.
Another important factor that we observed is that ChatGPT is slower than both commercial engines. We wereable to mitigate this issue by using Azure’s Japan East
servers for ChatGPT, but it was still noticeably slower.

― 2099 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, andDekang Lin, editors, Proceedings of the 40th Annual

References[1] Tom Kocmi, Eleftherios Avramidis, Rachel Bawden,
Ondřej Bojar, Anton Dvorkovich, Christian Federmann,
Mark Fishel, Markus Freitag, Thamme Gowda, Roman
Grundkiewicz, Barry Haddow, Philipp Koehn, BenjaminMarie, Christof Monz, Makoto Morishita, Kenton Murray,
Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja
Popović, and Mariya Shmatova. Findings of the 2023 conference on machine translation (WMT23): LLMs are here
but not quite there yet. In Philipp Koehn, Barry Haddow,
Tom Kocmi, and Christof Monz, editors, Proceedings ofthe Eighth Conference on Machine Translation, pp.
1–42, Singapore, December 2023. Association for Computational Linguistics.
[2] Masaru Yamada. Optimizing machine translation throughprompt engineering: An investigation into chatgpt’s customizability. Proceedings of Machine Translation
Summit XIX Vol. 2: Users Track, pp. 195–204, 9 2023.
[3] Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo
Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Towards making the most of ChatGPT for machine translation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational
Linguistics: EMNLP 2023, pp. 5622–5633, Singapore,
December 2023. Association for Computational Linguistics.
[4] Masao Utiyama.
Paranatcom — parallel englishjapanese abstract corpus made from nature communications articles.
https://www2.nict.go.jp/astrecatt/member/mutiyama/paranatcom/, 2019.
[5] James Cross Onur Çelebi Maha Elbayad Kenneth Heaﬁeld
Kevin Heﬀernan Elahe Kalbassi Janice Lam Daniel Licht
Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood Bapi Akula Loic Barrault Gabriel
Mejia Gonzalez Prangthip Hansanti John Hoﬀman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon
Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti
Bhosale Sergey Edunov Angela Fan Cynthia Gao Vedanuj
Goswami Francisco Guzmán Philipp Koehn Alexandre
Mourachko Christophe Ropers Saﬁyyah Saleem Holger
Schwenk Jeﬀ Wang NLLB Team, Marta R. Costa-jussà.
No language left behind: Scaling human-centered machinetranslation. 2022.
[6] Masao Utiyama and Mayumi Takahashi. English-japanesetranslation alignment data. https://www2.nict.go.jp/astrecatt/member/mutiyama/align/download/index.html, 2003.
[7] Graham Neubig.
The Kyoto free translation task.
http://www.phontron.com/kftt, 2011.
[8] Jörg Tiedemann. Parallel data, tools and interfacesin OPUS. In Nicoletta Calzolari, Khalid Choukri,
Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard,
Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios
Piperidis, editors, Proceedings of the Eighth Interna-

Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA,

July 2002. Association for Computational Linguistics.
[10] Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon
Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang
Liu, editors, Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing(EMNLP), pp. 2685–2702, Online, November 2020. Asso-

ciation for Computational Linguistics.
[11] Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S.
Chao. Diﬃculty-aware machine translation evaluation. In
Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli,
editors, Proceedings of the 59th Annual Meeting of
the Association for Computational Linguistics andthe 11th International Joint Conference on Natural
Language Processing (Volume 2: Short Papers), pp.

26–32, Online, August 2021. Association for Computational Linguistics.

tional Conference on Language Resources and Evaluation (LREC’12), pp. 2214–2218, Istanbul, Turkey, May

2012. European Language Resources Association (ELRA).
[9] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-

― 2100 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

A

Document-level Translation:

Datasets• ChatGPT-3.5 Simple: He praised the ramen and riceball with sea bream, saying, ”The Thai broth is well

Table 6

DatasetParaNatCom
FLORES
NovelsKFTTWMT News

Description of Datasets
Avg.Num
Direction
Type
Src.Sent
EN → JA Abstracts
1179.2EN → JA
Article600.6
EN → JA
Novels1640.6JA → EN History
742.6
JA → EN
News
375.0

Avg.Num
Src.Sent
7.2
5.0
16.2
15.0
7.0

ﬂavored. It’s delicious.”• ChatGPT-4 Simple: After sampling the ramen andTaimeshi onigiri, he added, ”The Thai broth is very
eﬀective. It’s delicious,” giving it a stamp of approval.
• ChatGPT-3.5 Enhanced: He praised the taste of theramen and taimeshi rice ball after trying them, stating,”The Thai broth is well accentuated. It’s delicious.”

B Enhanced Prompts: Categories,
Styles, and Tones• ChatGPT-4 Enhanced: After a taste test of the ramen and tai-meshi onigiri, he heartily approved, stat-

For categories and its style mapping, the options are as

ing: ”The snapper soup really comes through. Tastes

follows:
1. ﬁnance, economy, judicial aﬀairs: contracts, prospec-

great!” The new products will also be available atstores in neighbouring Aichi and Gifu prefectures.

tus, ﬁnancial reports, research reports, articles of in-• System A: Tasting the ramen and taimeshi rice balls,

corporation, certiﬁed copies, business letters, press

he said, ”The Thai dashi (soup stock) is very eﬀective.

releases.

They are delicious.

2. medicine: application documents (CTD), package
inserts, medical records, regulatory guidelines, aca-

• System B: He sampled the ramen and taimeshi riceballs and praised them, saying, “The Thai soup stock

demic papers, literature.

works well.It’s delicious.”

3. industry, science technology: instruction manuals,
catalogs, brochures, design documents, drawings,

Sentence-level Translation:

proposals, regulations, academic papers, literature.

• ChatGPT-3.5 Simple: I tasted ramen and an onigiri

4. patent, intellectual properties: application documentswith sea bream and praised it, saying, ”The Thai broth(speciﬁcations), gazettes, prosecution, IDS (Informa-

is very ﬂavorful. It’s delicious.”tion Disclosure Statement).

• ChatGPT-4 Simple: They sampled the ramen and5. other:catalogs, brochures, operations manuals,

guidebooks, article, manga, novel.
For tones, the options are as follows: spoken formal,
spoken casual, written formal, written casual, plain tone
(da & de aru), and polite tone (desu & masu).

C

rice balls with sea bream and aﬃrmed, ”The Thai
broth is very well done. It’s delicious.”• ChatGPT-3.5 Enhanced: I tried tasting ramen andtai meshi onigiri and gave it my stamp of approval,
saying ”The Thai broth is really eﬀective. Delicious!”• ChatGPT-4 Enhanced: I taste-tested Ramen andTai-meshi rice balls, and I gotta say, the Thai ﬂa-

Translation Samples

vors are really coming through. It’s super tasty, big
We sampled one source sentence from WMT News, thecorresponding gold translation, and the translation outputs• System A: After tasting the ramen and taimeshi riceof all ChatGPT settings and of the two commercial trans-

balls, the chef gave his stamp of approval, saying,lation systems for both document and sentence-level.

”The Thai dashi (soup stock) is very eﬀective andthumbs up from me!

Source Text: ラーメンとたいめしのおむすびを試食

delicious.

し「タイのだしがよく効いている。おいしい」と太• System B: He sampled the ramen and taimeshi rice

鼓判を押した。

balls and praised them, saying, “The Thai soup stock
works well.It’s delicious.”

Gold: He sampled the ramen and rice balls and said, ”You
can taste the sea bream soup stock and it’s delicious.” giving it a thumbs up.

― 2101 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).