Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors
Authors: Nikita Mehandru
Sweta Agrawal
Yimin Xiao
Ge Gao
Elaine C Khoong
Marine Carpuat
Niloufar Salehi
Venue: EMNLP 2023 Main (2023)
Keywords: medical machine translation
clinical harm
human-centered NLP
TL;DR: We conduct a human study with physicians to test how certain interventions affect their ability to detect machine translation errors in high-stakes medical settings.

Abstract: A major challenge in the practical use of Machine Translation (MT) is that users lack information on translation quality to make informed decisions about how to rely on outputs. Progress in quality estimation research provides techniques to automatically assess MT quality, but these techniques have primarily been evaluated in vitro by comparison against human judgments outside of a specific context of use. This paper evaluates quality estimation feedback in vivo with a human study in realistic high-stakes medical settings. Using Emergency Department discharge instructions, we study how interventions based on quality estimation versus backtranslation assist physicians in deciding whether to show MT outputs to a patient. We find that quality estimation improves appropriate reliance on MT, but backtranslation helps physicians detect more clinically harmful errors that QE alone often misses.

Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors
Nikita Mehandru', Sweta Agrawal,, Yimin Xiao²,
Elaine C Khoong', Ge Gao", Marine Carpuat", Niloufar Salehi'
'University of California, Berkeley ?University of Maryland
University of California, San Francisco
mmehandru@berkeley edu, sweagraweumd. edu, yxiaoêumd. edu elaine, wwwong@ucsf.ff.ccmmoomoo edu, gegaolumd. edu marine@umd. edu, ssalehi@berkeley.oo
Abstract
cluding in high-stakes contexts such as hospitals and courtrooms (Vieira et al., 2021), MT errors in those settings can be particularly harmful. In our setting, incorrect discharge instructions could lead a patient misunderstanding their diagnosis or taking medications incorrectly, with potentially fifetthreatnning consequences.
major challenge in the practical use of Ma- chine Translation (MT) that users lack guid- ance to make informed decisions about when to rely on outputs. Progress in quality estima- tion research provides techniques to automati- cally assess MT quality, but these techniques have primarily been evaluated vitro by com- parison against human judgments outside of a specific context of use. This paper evalu- ates quality estimation feedback vivo with a human study simulating decision-me decision-making in high-stakes medical settings. Using Emergency Department discharge instructions, we study how interventions based on quality estimation versus backtranslation assist physicians in de- ciding whether show MT outputs patient. We find that quality estimation improves ap- propriate reliance on MT, but backtranslation helps physicians detect more clinically harmful errors that QE alone often misses.
Research shows that people tend to over-rely on systems (Buçinca et al., 2021), and that explain- ability techniques that aim to address this issue can instead increase blind trust in incorrect predictions (Bansal et al., 2021). In the case of MT, most user studies have focused on human translators (Stew- art et al., 2020; Castilho et al., 2019; Green et al.. 2013), who have the expertise evaluate MT faith- fulness and correct outputs when needed. Decid- ing how rely on MT is much more challenging for people who use to communicate in a language that they do not know. Zouhar et al. (2021) find that providing quality feedback to people using MT in travel scenarios has mixed effects, and can make them feel more confident in their decisions without improving their actual task performance.
1 Introduction
Empowering people to decide when and how to rely on NLP systems appropriately is a critical, al- beit challenging, endeavor. Appropriate reliance is a moving target because it is difficult to opera- tionalize, and depends context and application domain. Research in this space has far relied evaluations that are abstracted from real world use cases (Doshi Velez and Kim, 2017; Narayanan et al., 2018; Boyd-Graber et al., 2022). We build on this work, and study appropriate reliance close to the actual decision that users have to make on the ground: whether or not to rely on a model output.
This work evaluates the impact of quality esti- mation (QE) feedback on physicians' reliance on MT, building on decades of MT research on auto- matically estimating the quality of MT without ref- erence translations (Blatz al., 2004; Quirk, 2004; Specia al., 2018; Fonseca et al., 2019; Han al., 2021). However, QE systems are primarily trained to score overall translation quality outside spe- cific context of use. It is unclear whether people will know how to interpret QE scores given prior evidence that they struggle to use probability and confidence estimates in explanations (Miller, 2018; Vodrahalli al., 2022). Additionally, even when interpreted correctly, it is not clear whether seeing QE score will make users better at deciding when to rely on an MT output and when not to, e.g. due to clinically relevant errors. We compare QE with
We study this question in the context of physi- cians deciding when to rely Machine Translation (MT) in an emergency room when communicating discharge instructions speak their language (Mehandru et al., 2022). MT represents an example of an NLP system used by millions of people in daily life (Pitman, 1021), in-
patient who does not

---

a) Quality estimation
1I Input sentence from discharge instructions.
2) Chinese translation via. Google Translate
3) Quality estimation model provides eveluztion the of translation.
Your diag is opilepoy breakthrough se zures we recommend resuming sonisamide and ca carbamacopine 1 week donazepam bridge.
155 Wi:2 de (4 SRER 11
G
Carbomazzeinnel.... 1, @e Wi.i
ZonisamideRi
transiatio preserves only some of the meaning the but misses signlficnt parts.
Input sentence discharge instructions.
Chinese translation viz. Google Translate
3) Back-translation to Eng Google
b) Back translation
Your diagnos's epilopay with breaithhouuhh seioures we recommend resumiig zonisamide and carbomatepiine week of conazepam bridge.
13:99 10 #hH., 32 32119 Zontsamide-0
diagnosis epilepey of epileptic seizures. It recommended that we restore Zonisamide and Carbamazspine and chlorolsi AM Bridge.
Carbamassine #8
Figure I: Physicians who participate the study are shown sentence from real discharge instruction, and translation to Chinese via Google Translate. They are asked to decide whether they would hand this translation to patient who only speaks Chinese. Participants are randomly assigned to one of two conditions: a) Quality estimation: A quality estimation model provides an evaluation of the translation, the backtranslation of the text to English using Google Translate. We found that participants who were provided the quality estimation were better at detecting when to rely on a translation. For the most severe errors, backtranslation provided more reliable assistance. The sentence shown was marked as not adequate and life-threatening by bilingual physician annotators. The human reference translation for the Google Translate output states: Your diagnosis is seizure of seizure, suggest we resume Zonisamide and Carbamazepine and one week of Clonazepam-am-briee
Backtranslation: The participant shown
a method commonly used by lay users, including physicians, to estimate the quality of translation: backtranslation into the source language using the same system (Mehandru et al., 2022).
ical case translations, have led research systems that produce translations that are more appropri- ate with in domain terminology than generic MT systems for diverse languages (Neves et al., 2022). However, in practical settings, clinicians turn to widely available MT systems such as Google Trans- late or Bing Translator (Randhawa et al., 2013; Khoong and Rodriguez, 2022). When translat- ing Emergency Department discharge instructions from English into Chinese and Spanish, Google Translate was found to produce majority of accu- rate outputs, however, a small number of inaccurate translations presented a risk of clinically signifi- cant harm (2% of sentences in Spanish and 8% of sentences in Chinese) (Khoong et al., 2019a). Our work evaluates tools that are used in practice by physicians and purposefully over-samples from error- prror-prone sentences present useful evaluation framework that can also be used to evaluate ded- icated clinical MT systems, complementing stan- dard reference-bant based evaluation metrics which do not directly account for the potential clinical impact of MT errors (Dew et al., 2018).
We conduct a randomized experiment with 65 English-speaking physicians to test how each of these interventions impacts their ability to decide when to rely imperfect MT outputs (Figure 1). We find:
The QE treatment group had a significantly higher confidence-wedd weighted accuracy in their overall decision to give or not give a transla- tion to a patient.
The BT treatment group more effectively de- tected critical translation errors, those rated as having higher clinical risk.
In sum, both interventions improve physicians' ability to assess clinical risk and their confidence in their decisions, but for complementary reasons.
2 Background
We situate this work in the MT literature before motivating our medical use case in Section 3.
Quality Estimation Quality Estimation (QE), the task of automatically assessing the quality of MT outputs without access to human references, is a long- standing area of research MT (Specia et al., 2018). State-of ofatee-o-the-aas the-art QE systems devel-
Clinical MT MT shared tasks motivated by med- ical applications, such as scientific abstracts or clin-
'Code and data to reproduce our findings are released at https:// //github.bbbbbb.. com/n mewnndr//hyssiccinnn git.

---

oped for evaluating MT such as OpenKiwi (Kepler al., 2019). TransQuest (Ranasinghe al., 2020), or COMET-sre (Rei et al., 2020) are built on top of large scale multilingual models like BERT (De- vlin et al., 2019) or XLMMRoBERTA (Conneau et al., 2020). They are trained to predict direct as- sessment scores collected by crowdsourced work- ers post-editing efforts measured by HTER (Snover et al., 2006). QE systems have been primar- ily evaluated "in vitro" by measuring the correla- tion of their scores with generic human judgments quality collected independently from the context of use. QE has also proved successful at guid- ing human translation workflows (Stewart et al.. 2020). However, it remains unclear how useful QF is for non-professional end-users in practice. In this work, we present assessments derived from the state-of-the-art COMET-sre to physicians who do not speak Chinese help them decide when rely on Boglish-Chinese MT.
of backtranslation in a clinical decision-making context and compare it to a QE model output.
3 MT for Cross-lingual Communication Between Patients and Physicians
Cross-lingual communication is imperative in the presence of language barriers between physicians and patients. We focus on specific high-stakes context: helping physicians communicate dis- charge instructions to patients in the Emergency Department (ED). While medical interpreters of- facilitate conversations between clinicians and Limited English Proficiency (LEP) patients, this has been found to be insufficient in helping pa- tients recall what they are supposed to do after get- ting discharged (Hoek et al., 2020). Further, com- prehension of emergency discharge instructions is known to be an important contributor to patient non-adherence (Clarke et al., 2005).
Our prior work has found that MT frequently used in practice for tasks such as automatically translating discharge instructions (Mehandru et al., 2022), and thus provides a written record for pa- tients to aid in comprehension at discharge time as well as recall and adherence. A key challenge is that it is difficult for physicians to ensure that patients comprehend written discharge instructions when they cannot verify the accuracy of machine- generated translation. As an added complication, limited health literacy and discharge plan complex- ity can lead patients to overestimate comprehension (Glick et al., 2020)-
Backtranslation We focus on scenarios where the imput text is translated into language unknown to the text's author. The onus to decide whether the output is acceptable therefore falls on the au- thor, even though they do not have the expertise to assess translation quality. In these settings, peo- ple routinely use intuitive feedback mechanism: backtranslation, which consists of using MT again translate the original MT output in the input lan- guage, so they can compare it with the original in language they understand. This practice has been decried the MT literature (Somers, 2005) as backtranslation provides a very noisy signal by lumping together errors from the forward and back- ward MT pass, and potentially hiding errors that the backward pass recovers from. Nevertheless, people backtranslation routinely, perhaps couraged by interfaces that let them switch MT translation direction with single click. However, little is known about the usefulness of backtransla- tion in these practical settings. Zouhar et al. (2021) conduct a user study evaluating the impact of back- translation by lay users for the practical task of booking hotel room in language that they do not speak. They hypothesize that backtranslation feed- back can help people craft the content of a message it correctly interpreted by the recipient. Back- translation feedback was found to greatly increase user confidence a translation, without improving the actual translation quality. We study the impact
Designing MT for effective pyysician-pationn communication involves many stakeholders. This work focuses on physicians as a starting point, findings can inform their training and strategies for cross-lingual communication maximize impact in the short-term. We leave to future work the equally important question of helping diverse LEP patient populations rely on MT output adequately.
4 Methods
We conducted randomized controlled experiment to test how quality estimation and backtranslation impact physicians appropriate reliance on MT.
4.1 Emergency Department Discharge Instructions Data
Source Text English source text for our experi- ment is drawn from de-identified Emergency De- partment (ED) discharge instructions that were writ-

---

ten between the years 2016 to 2021 at the Univer- sity of California, San Francisco (UCSF). We select six discharge instructions, and a total of twenty- eight sentences from these notes, to ensure that they present expected key elements in a discharge instruction to a patient, including presentation of the problem (chief complaint), actual diagnosis, medication list, follow-up items, and 24/7 call- back number with the referring provider (DeSai al., 2021; Halasyamani et al., 2006). Addition- ally, sentences were selected based on the source text complexity, and the presence of certain words having multiple meanings.
4.2 Experimental Design
We conduct a between-subjects experiment with participants randomly assigned to one of the two treatment conditions: backtranslation (BT) and quality estimation (QE). We added baseline con- dition to assess participant responses MT in the absence of feedback, which both groups completed first wwithin-subjects).
Participants We used convenience sampling to recruit sixty-five physicians to participate in our study. Medical residents were in training programs in the United States, and practicing physicians worked in multilingual settings. Their specialties included: internal medicine, cardiology, emergency medicine, neurology, surgery, family medicine, pe- diatrics, allergy and immunology, intensive care, obstetrics and gynecology, infectious discases, mil- itary medicine, and psychiatry. 45% of physicians reported interacting with LEP patients daily, while another 45% responded interacting with them ei- ther two three times per week bi-weekly. 17% of physicians reported using Google Translate on a monthly basis when writing discharge instructions. while 14% responded using it bi-weekly or two to three times a week.
MT The subset of selected English sentences were automatically translated into simplified Chi- nese by Google Translate.? We chose English- Chinese, because it is high-demand language pair that is often needed in clinical settings in the United States, and Google Translate is known to be used by physicians (Khoong et al., 2019b; Mehandru al., 2022). While general translation quality is expected to be reasonably strong, on FLORES, the translation quality as measured by BLEU is 38.50 for the English-Chinoss (Simplified) devtest (Goyal et al., 2022). Medical texts are typically out of the training domain, and as result, clinically harmful translation errors have been documented for this specific language pair (Khoong et al., 2019b).
We randomized physicians into each condition for a total of thirty-five physicians in the backtrans- lation group and thirty in the QE group. Physicians were screened to ensure they were fluent in English, and had no knowledge of Chinese. This inclusion criteria for the experiment was in the demographic section of the survey, and was included in our re- cruitment emails. We further manually filtered out any participants who may have missed this state- ment and reported that they spoke Chinese in the pre-survey.
Gold Annotation Three bilingual physicians in- dependently annotated each MT output along two dimensions: translation adequacy and clinical risk. Adequacy defined whether the Chi- nese translation accurately conveyed the meaning of the English source text (Turian et al., 2003). Physicians rated clinical risk based on the transla- tion presented to them according to five categories: clinically insignificant, mildly clinically significant, moderately clinically significant, highly clinically significant, and life-threatening (Nápoles et al., 2015). During the annotations, physicians kept in mind how if a monolingual patient were read the translation, whether or not the patient would understand the discharge instruction sentence, The three physicians then met with the lead author to discuss disagreements in the clinical risk ratings, and agree on final label for each sentence.
This study was approved by our Institutional Review Board (IRB), and physicians were com- pensated for taking the time to participate in our experiment.
Survey Design Physicians were first presented with the baseline condition. They were asked to read discharge instruction, and then were pre- sented with a sentence from the note and its respec- tive Chinese MT translation. After the baseline condition, physicians were presented with one of the two treatment interventions, and the same set of twenty-eight English sentences from the six dis- charge instructions, each associated with its Chi-
We use the Google Sheets Translation API.

---

ADEQUATE
CLINICAL RISK
YES
No
INSIGNIFICANT
MODERATE LIFE-TREATTEING
CONSISTENT
10
MOST PRESERVED
4 6
3
2
0
SOME PRESERVED
1
Table 1: Each column represents the physicians' true labels clinical risk (e.g., high clinical risk implies that the translation meaning was not similar to the original source). Each row represents the QE's assessment of the translation with consistent meaning that the translation preserved the original source's meaning. This chart shows QE quality labels are imperfect. but good enough to be potentially useful. The system err by overestimating translation quality compared to human assessments of adequacy and clinical risk on our dataset.
nese MT translation except also accompanied by one of two quality feedback types.
to define quality labels, motivated by those used to assess translation quality in human evaluation of MT (Freitag et al., 2021; Kocmi et al., 2022).
In all conditions, after seeing a Chinese trans- lation, participants were asked: 1) whether they would give the translation to patient who only reads Chinese (binary question, yes no), and 2) their confidence in this decision five-point Lik- ert scale (I= Not Confident and 5= Very Confident). the treatment conditions, participants were ad- ditionally asked to assess whether a monolingual Chinese patient would understand the discharge instruction sentence after reading the Chinese MT. using five point Likert scale (I= Patient would un- derstand none of the meaning and 5= Patient would understand all of the meaning).
To define these labels in data-driven fashion, we collect small development dataset by asking bilingual physician to answer two questions about the quality of 125 sentences sampled from related patient- physician conversational dataset: a) is the translation accurate? b) can the translation error pose a clinical harm? (Farcez et al., 2022). We identify thresholds for QE scores on this develop- ment set based on the ROC curves for adequacy and risk prediction. The translation pairs are then labeled according to Table 2.
Acknowledging the real-works context in which physicians would have to make these decisions, participants were asked to assume that a medical interpreter had already reviewed the discharge in- struction with the Chinese-speaking patient. To ensure that they focused on patient comprehension of the discharge instruction as it pertains to clin- ical outcomes, physicians were also instructed to respond to questions without concern for lawsuits and regulatory requirements around showing an im- perfect translation to a Chinese- -preekin patient.
QE SCORE OUR QF
[0.101:1]
The meaning of the translation is consistent
with the source.
[0.072,0.101]
The translation retains most of the meaning of the source.
1;00072]]
The translation preserves only some of the meaning of the source but misses signifi- cant parts.
Table Thresholds for QE scores and their assigned labels.
Table 1 shows the breakdown of QE labels for clinical risk and adequacy assessment the gold annotation data. COMET-QE achieves an accuracy of 66% and 73% on detecting adequate and clini- cally insignificant translations, respectively. How- ever, COMET-QE was not effective at detecting incorrect translations that caused clinical risk. Of the nine translations that are deemed by bilingual experts to cause moderate to life-threatening harm to users (columns "moderate". "high", and "life- threatening"ing in Table 1), COMET-QE rated 2/9 having "consistent" translations. The QE labels provided are thus accurate enough to be potentially
4.3 Treatment Conditions Details
Quality Estimation We adopt the state-of-the- art quality estimation (QE) system, Comet-QE (wmt21-comed qemmettt---mm->> to design this treatment condition (Rei et al., 2021). Each source and MT pair is passed through the trained QE predictor to generate score in the range of [), A posi- tive score indicates that the translation quality of the sentence is better than average, while a nega- tive score indicates below-average quality. While model-based predictions correlate well with human judgments (Freitag et al., 2022), they are hard to interpret. Hence, we partition the [-1, 1] interval
"This dataset provides simulated patient-physician inter- actions in English across six medical cases, which are closer to discharge instructions than c.g., clinical notes aimed at other physicians rather than patients. The conversations are automatically transcribed and manually post-edited.
Chttps://gittaen wtwp:ttuwb.cmm//ubbbbel////EEEE

---

useful, yet realistically imperfect, as expected of automatically generated reliability measures. Given the QE labels extracted using the above strategy, participants were then presented with a description of the quality estimation system, and the output labels they expect to see. They were asked to decide, based on the information provided, whether they would provide the translation to patient and their confidence in the assessment.
The Confidence Weighted Accuracy (CWA)
for each condition (BT, QE) for each physician (p) given by is:
CWA A
Σ
sign(s)
Cx
5
BES
(2)
1,
if s*
== Sp
sign(s)
-1, otherwise
Backtranslation In the backtranslation treatment condition, physicians were presented with the source English sentence, and were told that the Google Translation system generated Chinese translation for the given sentence. Participants were then presented with text explaining that Google Translate translated the previous text back into En- glish and were then shown the output translation.
Correctness We define a correct decision by comparing the physician's decision with the ad- equacy of the translation deemed by our physician annotators.
Adequate Translation. An adequate translation is one in which the discharge instruction sentence was passed through Google Translate and annotated by bilingual English-Chiness Chinese physicians as correctly conveying the meaning of the English source text. An accurate decision in this context would be a physician giving the discharge instruction sentence to a monolingual patient.
4.4 Measures
Our goal this work is to study whether physi- cians can more accurately rely on an MT output they are provided with an evaluation of the quality by a QE model, or by seeing the backtranslation by the same MT system. Our outcome metric is as close to possible the actual decision that physi- cians make in practice. We asked physicians to decide whether they would share translation with a patient and also their confidence in that deci- sion. To measure their overall performance, we use confidence weighting (Ebel, 1965), a common met- ric in cognitive psychology that measures whether the participant made the correct decision weighted by their confidence in that decision. Intuitively, confidence- weighted accuracy provides way to encapsulate the properties of appropriate reliance in one metric: making accurate decisions and calibrat- ing confidence in the model appropriately. other words, if the participant makes an error with high confidence, this metric penalizes them more than they make the same error but with less confidence.
Inadequate Translation. An inadequate translation is one in which the discharge instruction sentence was passed through Google Translate and anno- tated by bilingual English-Chiness physicians as incorrectly conveying the meaning of the English source text. An accurate decision in this context would be physician not giving the discharge in- struction sentence to monolingual patient.
5 Results
We will show that physicians in the quality es- timation (QE) condition had significantly higher confidence-weightee accuracy (CWA) than their baselines compared to those in the backtranslation (BT) condition. We also found that while the QE in- tervention increased their overall CWA, physicians in the QE treatment group significantly worse than those in the BT treatment group detecting errors, especially those labeled with higher clinical risk. We end with a discussion of the potential com- plementary roles that these two interventions can play for informed reliance on MT in high-stakes settings.
Reliance Metrics Given P physicians, in- stances (sentences), let s* be the correct answer for sentence s, and 6, is the answer selected by the p-th physician for this sentence with confidence es on a scale of to 5. Our experiment uses the following measures: The Physician Accuracy (%) for each condition
QE Treatment Group has Higher Confidence- Weighted Accuracy Physicians in the QE treat- ment group (M 19.6, SD 13.7) had signifi- cantly higher CWA than the baseline for that group (M = 11.2, SD 15.3, t(58) 2.1,p = 0.03,
(BT, QE) for each physician (p) is given by is:
Accuracy 1 Σ 11s = spl $ES
(1)
Figure 2). We did not find any difference

---

ACCURACY (%)
DISCHARGE INSTRUCTION
CLINICAL RISK
BT
QE
We believe this was due to something called vasovagal response which causes people feel faint or lose consciousness. Please try to drink fluids to make sure you hydrated today. sure to avoid taking NSAIDs (ibuprofen, naproxen) and As- pirin for pain.
moderately significant
80
23
moderately significant
89
40
mildly significant
31
The orthopedic surgeons drained your knee. You may take norco for pain, however, do not drink drive as it cause drowsiness.
highly sigmificant
91
moderately significant
71
43
Your diagnosis is epilepsy with breakthrough seizures recom- mend resuming zonisamide and carbamazepine clonazepam bridge.
life-threatening
97
80
week of
Please START taking and Doxycycline 100mg twice per day for seven days Keflex 500mg four times per day for seven days
highly significant
Table sample of the sentences used our study, the clinical risk of their translation determined by bi-lingual physicians, and participant accuracy rates across conditions for that sentence.
fied clinically harmful errors at a much higher rate. For example, consider the last sentence in Table 3, which gives medication instructions to patient. The MT was annotated by bilingual physicians not adequate and inducing highly clinically signif- icant risk. Physicians in the BT treatment group correctly decided not to give the MT output to a Chinese patient with 94.3% accuracy, while the QE treatment group did so with only 53.5% accuracy.
Bazeline
Confidence Weighted Accuracy (CWA)
Expertment
20
15
10
Post-Survey Analysis In a post-survey, we asked physicians to rate their confidence in using the QE and Google Translation systems in clinical work- flow scale of to 5. Physicians in the QE group reported higher levels of confidence in the quality estimation system (M = 2.6) compared physicians in the BT (M = 1.6). A Mann-W Whitney test revealed a significant difference in responses between the groups, U 229, p< .001. We saw a similar difference in the response to whether they would like to use each tool for clinical decision- making. Appendix Figure 5 has full post-survey results.
BT
QE
Treatment Group
Figure 2: The QE treatment group had a significantly higher seffidence-weighhed accuracy score than the baseline condition for that group well as the BT treat- ment group. However, we did not find any significant difference between the BT treatment group and the base- line condition for that group.
ns: not significant; #; significant with p-value < 0.05; p-value 0.001.
CWA between the physicians in the BT treatment group (M 8.6, SD 11.3) and their baselines (M 7.9, SD 11.4, f(68) 0.2, p 0.7). This means that physicians significantly improved in their ability rely appropriately on MT when presented with the QE evaluation, but not when presented with the BT. The difference between the QE and BT treatment groups was significant (t(63) 3.4, < 0.001).
Discussion
We found that QE and BT may play a comple- mentary role: QE can assist physicians in their decision to provide translated written discharge in- structions to patients, while BT was more effective in detecting critical errors in MT systems. Combin- ing aspects of BT and QE feedback may therefore benefit MT users in high-stakes settings. Our sults show that, contrary to common MT wisdom, BT should not be entirely discounted as a quality feedback method for people, aligning with recent
For discharge instruction sentences that were la- beled inducing higher clinical risk (moderately clinically significant, highly clinically significant, or tifetthreatennng), the BT treatment group identi-

---

Condition
Physician Accuracy (%6
threstonningg
NgNy sonificanc
moderately significant
midy sigiicann
inlgnificant
Clinical Risk
Figure 3: BT enables physicians to detect errors with high clinical risk at higher accuracy. *; p-value of 0.05;
Beseline
60
Baseline
Confidience Weighted Accuracy (CWA)
Confidence Weighted Accuracy (CWA)
Experiment
60
40
30
ns
20
20
10
-10
-20
-20
BT
QE
BT
QE
Treatment Group
Treatment Group
(a) Correct Identification of Adequate Translations
(b) Correct Identification Inadequate Translations
Figure 4: The QE treatment group had an overall higher CWA in correctly relying on MT systems when (a) treatment group had higher CWA in judging (b) inadequate translations.
translations were adequate whereas ns: not significant:*:: significant with p-valuc of 0.01; p-value of 0.0001.
findings that BT-based metrics can complement off-the-shelf supervised QE systems in automatic QE settings (Agrawal et al., 2022). However, re- mains to be seen whether presenting the BT output itself needed, or whether providing finer- finer-grained QE feedback could also play that role.
potential to assist physicians in writing discharge instructions (Arora and Arora, 2023), and the de- velopment of dedicated clinical language models, such as GatorTrro which trained on 90 billion words from electronic health records (Yang et al., 2022). Our survey asked physician respon- dents how they perceived the use of ChatGPT their workflows (Homolak, 2023; Dave et al., 2023; Li et al., 2023). We found 77% of physicians would use ChatGPT in their clinical workflows, bar- ring regulatory requirements and legal implications. More specifically, 78% responded they would use ChaiGPT to summarize patient notes, including symptoms and treatment plans, 60% responded they would use it to answer patient questions, 32% for clinical decision support and to make evidence- based recommendations, and 12% vowed to never use it in their actual clinical workflows. This further highlights the urgent need for NLP work that de-
In the post-survey, we asked physicians how they would want translation quality estimates to be pro- vided. Respondents reported that they would most want a binary indicator of whether the translation was correct, an explanation of where the errors oC- curred, and error categories relevant for medical purposes. Our study showed the potential of QF using one specific scheme, and motivates future work refining the presentation of QE feedback to best support physician needs.
Our study took place in the midst of many dis- cussions around the use of large language mod- els (LLMS) in clinical settings, including their

---

velops appropriate mechanisms for people to make appropriate use of language generation tools be- yond MT clinical settings.
needs explore how MT can be used for other settings, Additionally, our study focused on physi- cians' reliance on MT, but successful communi- cation naturally requires taking into account the patient's 's perspective as well, which we will con- sider in future work.
7 Conclusion
We conducted a randomized experiment with 65 Engliih-speaking physicians to test how quality es- timation and backtranslation interventions impact their ability to decide when to rely on imperfect MT outputs. We found that the QE group had a signifi- cantly higher accuracy in their overall decision to give not give a translation BT group detected critical translation errors more effectively. This study paves the way for future work designing methods that combine the strengths of QE and BT, and contributes a human-centered evaluation design that can be used to further im- prove MT in clinical settings.
Ethics Statement
This work was conducted in line with the EMNLP Ethics Policy. Models, datasets, and evaluation methodologies used are detailed in our Methods section. The discharge instruction sentences went through a rigorous de-identification process to en- sure no patient information was compromised. The use of these discharge notes were approved to be released by the university hospital that we obtained them from. Our study was approved by our univer- sity's Institutional Review Board (IRB). Physicians gave their consent participate in the study, and were compensated for their time.
patient, while the
More broadly, this work provides support for the usefulness of explanations for helping people rely on AI tools effectively in a real-world task. Our ex- planations focus on providing actionable feedback rather than explaining the internal workings of the MT model, aligning with recent calls to rethink ex- plainable AI beyond interpretability (Miller, 2023). It remains to be seen how to provide feedback that gives users more agency in appropriately using imperfect MT outputs, and how to design for ap- propriate reliance on other NLP tools in clinical settings, including large language models.
Acknowledgements
We would like to thank the physician participants for their time and the anonymous reviewers for their constructive feedback. We would also like to thank Coye Cheshire for early feedback on the study design. This work is supported by NSF Fair- ness in AT Grant 2147292.
References
Limitations
Sweta Agrawal, Nikita Mehandru, Niloufar Salehi, and Marine Carpuat. 2022. Quality estimation via back- translation at the wmt 2022 quality estimation task. In Proceedings of the Seventh Conference on Ma- chine Translation (WMT), pages 593-596.
The study naturally limited to specific exper- imental conditions. For example, we evaluated translation for a single language pair, and MT sys- tem. While this is motivated by real-world com- munication needs in hospitals in the United States, it is unclear how these findings would generalize other language pairs, including translation into, from, or between languages that are underrepre- sented in the MT training data which would likely lead to lower translation quality, as well as transla- tion between more closely related languages where users might be able to exploit cognates or other cues to assess translation quality even if they only speak one of the two languages involved. Future research will consider the combined use of multiple interventions and larger sample sizes of sentences. Emergency department discharge instructions represent one of many forms of communication between physicians and patients, and future work
Anmol Arora and Ananya Arora. 2023. The promise of large language models in health care. The Lancet, 401 (10377):641.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Ray- mond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Ffffect of AT Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors Computing Systems, CHI '21, pages 1-16, New York, NY, USA. Association for Computing Machinery.
John Blatz, Erin Fitzgerald, George Foster, Simona Gan- drabur. Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Uefting. 2004. Confidence estimation for machine translation. In COLING 2004: Pro- ceedings of the 20th International Conference Computational Linguistics, pages 315-321, Geneva. Switzerland, COLING.

---

Jordan Graderr Samuel Carton, Shi Feng, Q. Vera Liao, Tania Lombrozo, Alison Renter and Chenhao Tan. 2022. Human-centered evaluation of explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistiex: Human Language Tech- nologies: Tutorial Abstracts, pages 26-32, Seattle, United States. Association for Computational Lin- guistics.
Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Interpretable Machine Learning.
Robert L Ebel. 1963. Confidence weighting and test reliability. Journal of Educational Measurement, 2(1):49-57.
Faiha Fareez, Tishya Parikh, Christopher Wavell, Saba Shahab, Meghan Chevalier, Scott Good, Isabella De Blasi, Rafik Rhouma, Christopher McMahon, Jean-Paul Lam, et al. 2022. A dataset of simulated patient- physician medical interviews with focus respiratory cases. Scientific Data, 911:3113.
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forc- ing functions can reduce overreliance on ai in ai- assisted decision-making. making. Proceedings of the ACM on Huuman-Compan Computer Interaction, S(CSCW (:SCWI))1-22 1-21.
Erick Fonseca, Lisa Yankovskaya, André F. T. Mar- tins, Mark Fishel, and Christian Federmann. 2019. Findings of the WMT 2019 shared tasks on quality estimation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Pa- pers, Day 2), pages 1-10, Florence, Italy, Association for Computational Linguistics.
Sheila Castilho, Federico Gaspari, Joss Moorkens, Maja Popović, and Antonio Toral. 2019. Editors' foreword the special issue on human factors in neural ma- chine translation. Machine Translation, 33(1):1-7.
Collin Clarke, Steven Mare Friedman, Kevin Shi, Tamara Arenovich, Jose Monzon, and Christopher Culligan. 2005. Emergeney department discharge instructions comprehension and compliance study. Canadian Journal of Emergency Medicine, 7(1):5- 11.
Markus Freitag. George Foster, David Grangier. Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transac- tions of the Association for Computational Linguis- rics, 9:1460 1-1460-1474.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo. Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Tirth Dave, Sai Anirudh Athaluri, and Satyam Singh. 2023. Chatgpt in medicine: an overview of its ap- plications, advantages, limitations, future prospects, and ethical considerations. Frontiers in Artificial Intelligence, 6.
Alexander F Glick, Jonathan S Farkas, Rebecca E Rosenberg, Alan L Mendelsohn, Suzy Tomopoulos, Arthur H Fierman, Benard Dreyer, Michael Mig- otsky, Jennifer Melgar, and H Shonna Yin. 2020. Accuracy of parent perception of comprehension of discharge instructions: role of plan complexity and health literacy. Academic pediatrics, 2664):516-523.
Charisma DeSai, Keri Janowiak, Beatrice Secheli, Bleanor Phelps, Sam McDonald, Gary Reed, and Andra Blomkalns. 2021. Empowering patients: sim- plifying discharge instructions. BMJ Open Quality, 1003)x001141.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Spence Green, Jeffrey Heer, and Christopher D. Man- ning. 2013. The efficacy human post-editing for language translation. In Proceedings of the SIGCHI Conference Human Factors in Computing Systems, CHI 13, pages 439-448, Paris, France. Association for Computing Machinery.
Kristin N. Dew. Anne M. Turner, Yong K. Choi, Alyssa
Bosold, and Katrin Kirchhoff. 2018. Development of machine translation technology for assisting health communication: A systematic review. Journal of Biomedical Informatics, 85:56-67.
Halasyamani, S Kripalani, E Coleman, J Schnipper, C Van Walraven, J Nagamine, P Toreson, T Bookwal- ter, T Budnitz, and D Manning. 2006. Transition of

---

care for hospitalized elderly patients development of discharge checklist for hospitalists. Journal of Hospital Medicine, 16)):354-30.
Tim Miller. 2018. Explanation in Artificial Intelligence: Insights from the Social Sciences.
Tim Miller. 2023. Explainable ai is dead, long live explainable ai! hypothesisddriven decision support using evaluative ai. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Trans- parency, pages 333-342.
Lifeng Han, Alan Smeaton, and Gareth Jones. 2021. Translation quality assessment: A brief survey on manual and automatic methods. In Proceedings for the First Workshop on Modelling Translation: Trans-
latology in the Digital Age, pages 15-33, online. As- sociation for Computational Linguistics.
Anna M Nápoles. Jasmine Santoyo- -Ossoon,,, Leah Kar- liner. Steven E Gregorich, and Eliseo J Pérez-Stable. 2015. Inaccurate language interpretation and its clini- cal significance in the medical encounters spanish- speaking latinos. Medical care, 53(1 1):940.
Amber E Hock, Susanne CP Anker, Ed F van Beeck, Alex Burdorf, Pleunie PM Rood, and Juanita A Haagsma. 2020. Patient discharge instructions in the emergency department and their effects on com- prehension and recall of discharge instructions: a systematic review and meta-analysis. Annals of emer- gency medicine, 75(3):435-444,
Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2018. How do humans understand explanations from ma- chine learning systems? an evaluation of the human- intepretabbtyy of explanation. arXiv preprint arXiv:1802.00652
Jan Homolak, 2023. Opportunities and risks chat- gpt in medicine, science, and academic publishing: modern promethean dilemma. Croatian Medical Journal, 14
Mariana Neves, Antonio Jimeno Yepes, Amy Siu. Roland Roller, Philippe Thomas, Maika Vi- cente Navarro, Lana Yeganova, Dina Wiemann, Gior- gio Maria Di Nunzio, Federica Vezzani, Christel Gerardin, Rachel Bawden, Darryl Johan Estrada, Salvador Lima-lopez, Eulalia Farre- madull. Mar- tin Krallinger, Cristian Grozea, and Aurelie Neveol. 2022. Findings of the WMT 2022 Biomedical Trans- lation Shared Task: Monolingual Clinical Case Re- ports. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 694-723, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117-122 Florence, Italy, As- sociation for Computational Linguistics.
Elaine C. Khoong and Jorge A. Rodriguez. 2022. Research Agenda for Using Machine Translation in Clinical Medicine. Journal of General Internal Medicine, 375):127
Jeff Pitman. 2021. Google Translate: One billion installs, one billion stories. https:/blog.goooll/ppooo wowgleppoootttttrrasttttoooo-ooo installs/.
Elaine C. Khoong, Eric Steinbrook, Cortlyn Brown, and Alicia Fernandez. 2019a. Assessing the Use of Google Translate for Spanish and Chinese Transla- tions of Emergency Department Discharge Instruc- tions. JAMA Internal Medicine, 179(4):580-582
Christopher B. Quirk. 2004. Training a sentence-level machine translation confidence measure. In Proceed- ings of the Fourth International Conference on Lan- guage Resources and Evaluation (LREC"04) Lisbon, Portugal. European Language Resources Association (ELRA).
Elaine C Khoong, Eric Steinbrook, Cortlyn Brown, and Alicia Fernandez. 2019b. Assessing the use of google translate for spanish and chinese translations of emer- gency department discharge instructions. JAMA in- lernal medicine, 179(4):500-582
Tom Koemi, Rachel Bawden, Ondfej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, et al. 2022. Findings of the 2022 conference machine translation (wmt22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1-45.
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020. TransQuest: Translation quality esti- mation with cross-lingual transformers. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 5070-5081, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Jianning Li, Amin Dada, Jens Kleesick, and Jan Eg- ger. 2023. Chatgpt in healthcare: A taxonomy and systematic review. medRxiv, pages 2023-03.
Gurdeeshpal Randhawa, Mariella Ferreyra, Rukhsana Ahmed, Omar Ezzat, and Kevin Pottie. 2013. Us- ing machine translation in clinical practice. Cana- dian Family Physician Medecin De Famille Cana- dien, 5984)3382-383
Nikita Mehandru, Samantha Robertson, and Niloufar Salehi. 2022, Reliable and safe use of machine trans- lation in medical settings. In 2022 ACM Conference on Fairness, Accountability and Transparency, pages 2026-2025.
Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie.

---

2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Pro- ceedings of the Sixth Conference Machine Trans- lation, pages 1030-1040, Online. Association for Computational Linguistics.
Vilém Zouhar, Michal Novák, Matúš Zilinec, Ondrej Bojar, Mateo Obregón, Robin L. Hill, Frédéric Blain, Marina Fomicheva, Lucia Specia, and Lisa Yankovskaya. 2021. Backtranslation feedback im- proves user confidence in MT, not quality. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 151-161, Online. Association for Computational Lin- guistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-2702, Online. Association for Computational Linguisties.
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of trans- lation edit rate with targeted human annotation. In Proceedings of the 7th Conference the Association for Machine Translation in the Americas: Technical Papers, pages 223-231 Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
Harold Somers. 2005. Round-trip trip translation: What is it good for? In Proceedings of the Australasian Language Technology Workshop 2005, pages 127- 133.
Lucia Specia, Carolina Scarton, and Gustavo Henrique Paetzold. 2018. Quality estimation for machine trans- lation. Synthesis Lectures on Human Language Tech- nologies, 1162
Craig Stewart, Ricardo Rei, Catarina Farinha, Alon Lavie. 2020, COMET deploying a new state-of- the-art MT evaluation metric in production. In Pro- ceedings of the 14th Conference of the Association for Machine Translation in the Americas ( (Volume 2: User Track), pages 78-109, Virtual. Association for Machine Translation the Americas.
Joseph Turian, Luke Shen, and Melamed. 2003. Evaluation of machine translation and its evaluation. In Proceedings of Machine Translation Summit IX: Papers.
Lucas Nunes Vieira, Minako O' O'Hagan, and Carol O'Sullivan. 2021. Understanding the societal im- pacts of machine translation: critical review of the literature on medical and legal use cases. Informa- tion, Communication & Society, 24(11))1515-1532
Kailas Vodrahalli, Tobias Gerstenberg, and James Zou. 2022. Uncalibrated Models Can Improve Human-Al Collaboration. In Advances in Neural Information Processing Systems.
Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona Flores, Ying Zhang. et al. 2022. Gatortron: A large clinical language model to unlock patient information from unstruc- tured electronic health records. arXiv preprint arXiv:2203.33540.

---

A
de Survey Design
We provide an example of one sentence from single discharge note in each of the two interventions: BT and QE. Physicians in both treatment groups received the baseline questions prior to receiving the same sentence accompanied by their quality feedback.
A.1 Baseline
Instruction Google Translate is translation system that allows users to automatically translate text from one language to another. Automatic translation systems like Google Translate; however, can sometimes make errors.
Please respond to the following questions barring lawsuit and regulatory requirements around showing imperfect translation to Chinese-speaking patient. You are solely focused patient comprehension of discharge instructions it pertains elinical outcomes.
Please answer each question assuming that you have already reviewed the sentence from the discharge instruction with a patient using a medical interpreter. Use your best judgment when answering questions, and do not consult outside sources.
You will be shown 28 sentences from a total of six discharge instructions.
Discharge Note The following sentence passed through Google Translate: You in the hospital because you had a loss of consciousness after getting IV placed. Google Translate generated the following Chinese translation for the sentence above: 13 OF B *
Question 1:
Would you give the Chinese translation to a patient who only reads Chinese?
How confident are you in your decision give (or not give) the above translation to patient? Likert scale: Confidence level (1=Not Confident; V= Confident)
Question 2:
A.2 BT
Instruction In the following section, you will be shown Chinese translation of an English sentence that passed through Google Translate. You will then be shown the English translation of that same Chinese sentence after it was passed through Google Translate again.
Please respond to the following questions barring a lawsuit and regulatory requirements around showing an imperfect translation to a Chinese-speaking patient. You are solely focused on patient comprehension of discharge instructions as it pertains to clinical outcomes.
You will be shown 28 sentences from a total of six discharge instructions.
Discharge Note The following sentence was passed through Google Translate: You in the hospital because you had loss of consciousness after getting an IV placed. Google Translate generated the following Chinese translation for the sentence above: f0 LE HH
RiR Google Translate translated this text back into English as: You are the hospital because you lose consciousness after the vein injection.
* T
Question 1:
Would you give the Chinese translation to a patient who only reads Chinese?
How confident you in your decision give (or not give) the above translation to patient? [Likert scale: Confidence level (1=Not Confident; 5=Very Confident)]
Question 2:
Question a monolingual Chinese patient were to read the previous sentence, do you think the patient would understand the discharge instruction sentence? [Likert scale: Comprehension ability (I=Patient would understand none of the meaning; 5=Patient would understand all of the meaning)]

---

A.3 QE
Instruction A quality estimation system can estimate the quality of the translated text by comparing the machine-generated translation (e.g. Chinese) to the original text (e.g. English). This model is trained to try to estimate how a human would rate the quality of translation.
The system will generate one of the three ratings:
The meaning of the translation is consistent with the source.
The translation retains most of the meaning of the source.
The translation preserves only some of the meaning of the source but misses significant parts.
Please respond to the following questions barring lawsuit and regulatory requirements around showing an imperfect translation to Chinese-speaking patient. You are solely focused on patient comprehension of discharge instructions it pertains to clinical outcomes. You will be shown 28 sentences from a total of six discharge instructions.
Discharge Note The following sentence was passed through Google Translate: You were in the hospital because you had a loss of consciousness after getting an IV placed. Google Translate generated the following Chinese translation for the sentence above: B B
R A quality estimation automatic translation system estimated the translation quality as: The meaning of the translation is consistent with the source.
Question 1: Would you give the Chinese translation a patient who only reads Chinese?
Question 2: How confident are you in your decision to give (or not give) the above translation to a patient? [Likert scale: Confidence level (I=Not Confident; 5= Confident)]
If monolingual Chinese patient were read the previous sentence, do you think the patient would understand the discharge instruction sentence? [Likert scale: Comprehension ability (I=Patient would understand none of meaning; 5=Patient would understand all of the meaning)]
Question 3:

---

Iaisegree strongly disegree somewhet.
am neutral about somewhet.
lagree strongly
lm confident in the Quality Estimation system. feel that it
3%
16.7%
43.3
36,
The Quality Estimation system is very reliable. can count correct all
NJ5.3
26.7
40.0
20.0
would use systems in clinical practice endorsed by medical board.
3.3 %
16.7
46.7
33.3
like using the Quality Estimation system for decision makingg
30.0
33.3
23.3 %
13.3 %
(a) QE
disagree strongly.
Jagrce strongly
disagron gre somewhat.
lagroe somewhat
am confident Google Translate. feel that it works well.
22
14.3
34.3
28.6%
Google Translate is very rellable. I count on correct time.
57.1 %
25.7
14.3 N
would use Al systems in clinical practice endorsed by medical
2.9
14.3 %
20.0 %
37.1 %
25.7%
like using Google Translate decision-making.
45.7 %
20.0
28.6 %
5.7%%
(b) BT
Figure 5: Post-Survey Physicians' Trust Analysis on use of QE and MT systems Clinical Workflows.


Reviews:
[{"review_id": "YemumKFPO7", "reviewer": "Reviewer_m7Bm", "summary": "", "questions": "* You note that physicians were screened for knowledge of English and Chinese in Line 301f. How did the screening work?\n* There is one sentence I can't entirely follow in Line 373f: ‘Of the nine translations that are deemed by bilingual experts to cause moderate to life-threatening harm to users, COMET results in six false positives.’ I tried hard but failed to relate this information to the numbers presented in Table 1 – could you explain where exactly the six false positives are in the table?\n* What is XAI (Line 379)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your time and encouraging feedback. Addressing your comments below:\n\n- In terms of screening, we used Qualtrics to administer our experiment. After providing context on the goal of the study, we included a demographic information section in our survey form where we asked physician participants if English was their first language, and how many other languages they spoke. We stated in bold if they spoke Chinese, they were ineligible to participate in the study. This inclusion criteria for the experiment was also included in our recruitment emails, and we manually filtered out any participants who may have missed this statement and wrote they spoke Chinese in the “other languages” text box. \n\n- In reference to Table 1, we updated our paper to further clarify the COMET-QE labels of 6/9. We added this text: “Table 1 shows the breakdown of QE labels for clinical risk and adequacy assessment on the gold annotation data. COMET-QE achieves an accuracy of 73% and 66% on detecting adequate and clinically insignificant translations, respectively. However, COMET-QE was not effective at detecting incorrect translations that caused clinical risk. Of the nine translations that are deemed by bilingual experts to cause moderate to life-threatening harm to users (columns “moderate”, “high”, and \"life-threatening\" in Table 1), COMET-QE rated 6/9 as having “consistent” or “most preserved” translations. The QE labels provided are thus accurate enough to be potentially useful, yet realistically imperfect, as expected of explainable artificial intelligence (XAI) interventions.”\n\n- We have clarified in the paper that XAI refers to explainable artificial intelligence. \n\n- In reference to “pointing out Google Translate's BLEU on FLORES…” we included this number to provide a rough indicator of the translation quality of Google Translate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is strongly relevant to the NLP community and the MT community in particular. Automated decision support aids such as QE receive a lot of attention, and QE in particular is mainly evaluated in comparison to human judgements of MT quality for ‘general’ texts. Outside of professional translation, QE (and related) methods clearly have a vast potential in informing decisions about whether or not to use MT in real-life tasks, but there is little to no published evidence that NLP-based decision support is effective in practice. This paper contributes a neatly designed case study in a field where MT is used frequently and where errors can have severe consequences. It also shows that ‘simple’ methods such as BT can complement ‘fancier’ models such as SOTA QE, and that the research community is well-advised to not drop the former off the radar.", "reasons_to_reject": "TL;DR: This is a very strong paper. I will fight to have it accepted.\n\nOther reviewers might be tempted to suggest that this paper would be better suited for publication at another venue, such as a human–computer interaction (HCI) conference. However, I strongly advise the (S)ACs and PC members to accept it for publication at EMNLP. As rightly pointed out, for example, by Stanford NLP's Diyi Yang in the ACL 2023 panel on The Future of Computational Linguistics in the LLM Age, NLP technology is easier than ever to use by people, so we – the NLP community – should care about how people use, and can be enabled to make the best possible use of, NLP technology. Earlier EMNLP conferences also featured papers that focussed on the perception of MT. For instance, https://aclanthology.org/D18-1512/ garnered significant media attention and continues to be cited regularly.", "questions_to_authors": "* You note that physicians were screened for knowledge of English and Chinese in Line 301f. How did the screening work?\n* There is one sentence I can't entirely follow in Line 373f: ‘Of the nine translations that are deemed by bilingual experts to cause moderate to life-threatening harm to users, COMET results in six false positives.’ I tried hard but failed to relate this information to the numbers presented in Table 1 – could you explain where exactly the six false positives are in the table?\n* What is XAI (Line 379)?", "paper_topic_and_main_contributions": "This paper explores the effectiveness of quality estimation (QE) and backtranslation (BT) to support subject matter experts in deciding when (not) to rely on machine translation (MT). Specifically, the authors designed and conducted a between-subjects experiment in which 65 physicians were randomised into two groups and, with the support of either QE or BT, indicated with what confidence they would or would not give a translation to a patient. Results show that the two decision support aids are complementary: the use of QE lead to significantly higher overall accuracy in deciding whether or not to use MT, while the use of BT lead to significantly higher accuracy in detecting critical translation errors.", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "First off: congratulations on the crystal clear writing.\n\nSome (mostly minor) suggestions for improvement:\n* I'm not sure if pointing out Google Translate's BLEU on FLORES is very helpful since – as with every black box system – it cannot be ruled out that these test data are part of the training data.\n* Line 362f: Revise punctuation/bracketing"}}, {"review_id": "seYJXmrwgv", "reviewer": "Reviewer_fR6t", "summary": "", "questions": "I suppose that the double-one in (1) is in fact the same as sign(s) in (2)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our paper. We appreciate you taking the time, and have addressed your comments below:\n\n- We chose 28 sentences across 6 discharge instructions to use in our study. Our focus in this paper is on physician perception of BT and QE interventions, and thus we prioritized a larger human subject sample in our experiment to make more robust conclusions about the effects of these techniques on physician users. We ensured; however, through our bilingual physician gold annotation process, that there was a diverse sample of sentences with varying levels of clinical risk in our experiment that represent a range of potential real-world use cases. This contrasts to typical MT evaluation methods where the goal is to assess the quality of an MT system and there are a larger set of text samples presented to a small number of human subjects. Expanding the dataset is an important area for future work that we have added to the paper.\n\n- Thank you for pointing out the labeling of translation pairs missing after line 368. We have fixed the placing and reference to the table in our paper.\n\n- We agree that an interesting intervention to study would be to combine QE and BT. In this paper, we specifically focused on the individual contributions of each technique on appropriate physician reliance on machine translation.  Our decision to focus on these two interventions was influenced by prior work suggesting that BT would not be effective, and our ability to recruit enough practicing physicians. We have emphasized the complementary effects of each intervention as an important future direction of work. \n\n- In (1), the function used is an indicator function that takes on a value of 1 if the physician classified the sentence correctly, and 0 if the physician was incorrect. This results in a classical definition of accuracy (i.e. percentage of correct answers). The downside to this metric is that it only considers correct answers and does not account for confidence. In our definition of Confidence Weighted Accuracy (CWA), (2), we use a sign function, that also indicates the sign for a correct or incorrect response and weighs it by the confidence of that answer. Confidence Weighted Accuracy is a commonly used metric in studies like ours.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study is interesting because it shows how to improve a real use of a NLP tool in a real world situation. It is well made and presents convincing results.", "reasons_to_reject": "There is only 6 discharge instructions and 28 sentences for 5 important elements. So, very few sentences for each element. Is it enough to drive definitive conclusions on the use of QE and BT in this setting?\n\nThe labelling of translation pairs seem to be missing after line 368. It makes table 1 hard to interpret.\n\nI don't understand why a third experimental situation has not been studied: showing both QE and BT to physicians. This could (or not…) have enhanced both results. I was on the verge of seeing this as a blocking point.\n", "questions_to_authors": "I suppose that the double-one in (1) is in fact the same as sign(s) in (2)?", "paper_topic_and_main_contributions": "This paper presents a study on the use of machine translation in US emergency rooms to translate discharge instructions to patients not speaking English. It compares the use of a quality evaluation tool vs. displaying the text back translated into English using the same machine translation tool.\nWhile the test data is limited in size, the statistic evaluation seems to be correct. The study finds that QE is efficient at helping practicians decide to deliver the translation to the patient, while BT helped to detect dangerous errors.\n\nSuggestion: use a different machine translation tool for back translation than the one used for the initial translation. This would allow reducing the risk of cancelling the translation errors. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Missing text after line 368 and thus table 1."}, "extra_text": {}}, {"review_id": "jx3cAgUqZg", "reviewer": "Reviewer_6psY", "summary": "", "questions": "1. Why do you think this paper should be addressed at the EMNLP conference? The practicality is acknowledged in terms of focusing on investigating LM's impact on the medical domain. However, the paper only addresses the influence of QE and BT on physicians' decision-making. This is considered more appropriate to be addressed in the medical domain.\n\n2. What is the significance of the paper? What takeaways do the authors want to reveal?\n\n3. Why the authors only deal with English-Chinese pairs? There are many possible language pairs.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback; it has definitely strengthened our paper. We have addressed your comments below:\n\n1. We believe that this paper is well-suited for the Human-Centered NLP track included in the EMNLP 2023 Call for Papers.  We view our work as aligning with a growing trend toward NLP research that accounts for the people who interact with, and are impacted by, NLP technology, as can be seen for instance in the papers highlighted in the Human-Centered NLP Special Theme at NAACL 2022, and in the Human-Centered NLP courses by Diyi Yang at Stanford or by Sherry Tongshuan Wu at CMU, which include many ACL/EMNLP publications. The long-term goal of this line of work is to build and continually improve the design of the systems to meet the concerns of users, and move beyond solely performance and accuracy measurements.\n\n- We argue that our findings are relevant to a broad segment of the MT/NLP community, beyond the sub-community working on medical applications. We studied the effectiveness of two distinct NLP techniques for giving lay users feedback on the quality of MT. Characterizing the impact of MT errors rigorously requires situating our study in a specific context of use: otherwise, it is unclear how to reliably decide whether an error is critical or not.  We selected physician’s decision-making as a case study, because it corresponds to a real-world need,  in addition to providing a context where the consequences of MT errors are clear. Our study paves the way for future work evaluating QE mechanisms in a more diverse range of settings. \n\n2. There are a few key takeaways we want to highlight with this work. We found that the physicians exposed to the QE intervention had a significantly higher Confidence Weighted Accuracy in their overall decision to give or not give a translation to a patient, while the BT group detected critical clinically harmful translation errors more effectively. These findings not only provide insight into how currently imperfect MT systems can be leveraged by physicians in medical settings when interacting with patients with limited English proficiency, but it also serves to motivate further technical work in developing QE and MT systems that are informed by the needs of users based off of our human-centered evaluation. \n\n- More broadly, this research advances our understanding of explainability in NLP and more broadly AI systems. Our work provides support for the usefulness of technical interventions that help people rely on AI tools effectively, in a real-world task. Our interventions focus on providing actionable feedback rather than explaining the internal workings of the MT model, aligning with recent calls to rethink explainable AI beyond interpretability. Future work can investigate how best to provide feedback that gives users more agency in appropriately using imperfect MT outputs (our work provides a foundation for this), and how to design for appropriate reliance on other NLP tools in clinical settings, including large language models. Our study examined commercialized MT systems because such systems are being widely used by physicians in clinical settings. LLMs are gradually being deployed in the medical context, and future work could investigate how these models such as ChatGPT are being used by physicians in practice. \n\n3. We chose English-Chinese, because Chinese is a high-demand language pair that is often needed in clinical settings in the United States (Khoong et al. 2019), and is a linguistically interesting pair involving translation across distant languages. While the general translation quality is expected to be reasonably strong for this high-resource language pair, the FLORES English-Chinese (simplified) devtest has a translation quality as measured by BLEU as 38.5 (Goyal et al. 2022). This pair is also well-studied in the MT literature, which influenced our motivation to use Chinese. \n\n4. In reference to comments around reproducibility, we are committed to releasing: a) the Emergency Department (ED) discharge instructions, b) the translated sentences that were shown to the 65 physicians in our experiment, c) the survey responses from physicians on whether to give a translation to a patient or not as well as their confidence in doing so, and d) the bilingual physician gold annotations labeling adequacy and clinical risk. We will release this data upon acceptance of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-motivated. Especially in the medical domain, where human health and even life are directly related, even a single translation error carries significant risk. To prevent safety issues caused by simple translation errors, the practical analysis of the roles of information (QE or BT) for physicians is valuable.", "reasons_to_reject": "1. The motivation of this paper is well-established; however, there are doubts about whether the content of this paper is suitable for the EMNLP conference. This paper is not about how translation errors of language models appear in the medical field or how to produce correct translations through rewriting, etc. The paper does not explore the technical aspects and focuses only on the practical aspects of the utilization of Commercialized MT and QE systems, lacking in-depth technical details.\n\n2. Lines 515-539 cover the content related to LLM in this paper. Recently, LLM has shown outperforming performance, and the importance of its wide societal utilization has been emphasized by an increasing number of users. It is regrettable that this paper only utilizes the Commercialized MT system (Google Translate), not containing LLM. Although experiments related to ChatGPT are presented in those lines, they are not comprehensive and only rely on simple sociological statistics.\n\n3. The data statistics are not provided in the paper (Section 4.1). Due to the possibility of different evaluation or interpretation depending on the data, I consider it important to provide statistics about the data in experiments. Especially since the authors conducted new annotations, this aspect is further emphasized.\n\n4. This paper only deals with English-Chinese language pairs. The motivation presented by the authors could be applicable to all language pairs, but there is no specific reason given for choosing Chinese, and they did not cover multiple languages. In fact, it is understandable that experimentally dealing with multiple languages could have been challenging. However, there should have been a reason or explanation for utilizing Chinese in the study.", "questions_to_authors": "1. Why do you think this paper should be addressed at the EMNLP conference? The practicality is acknowledged in terms of focusing on investigating LM's impact on the medical domain. However, the paper only addresses the influence of QE and BT on physicians' decision-making. This is considered more appropriate to be addressed in the medical domain.\n\n2. What is the significance of the paper? What takeaways do the authors want to reveal?\n\n3. Why the authors only deal with English-Chinese pairs? There are many possible language pairs.", "paper_topic_and_main_contributions": "This paper experimentally investigates the impact of QE and BT on the decision-making process of physicians when providing translations for discharge instructions to non-English-speaking patients. The authors provide QE or BT results with discharge instructions to a group of 65 English-speaking physicians and analyze their final decision results. \n\nAccording to the experimental results, the physician group that referred to QE information reported higher accuracy in making decisions about providing translation results to patients. When utilizing BT, they more effectively detected cases with higher risk. The authors also suggest the possibility of complementary utilization of QE and BT.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how physicians might or might not trust the outputs of machine translators to provide these to their patients. A human study was carried out to see how two different tools, quality estimation and back translation, compare in helping physician decide if they will use the MT outputs. 

As one of the reviewers has pointed out, this work makes an important contribution in showing how tools such as QE can be used in real applications in more technical fields where there might be severe consequences for any translation errors. All reviewers agreed that this paper has strong soundness, and some reviewers found it strongly exciting.  Still, some suggestions have been made on how to further improve the writing of the paper. 

Some of the limitations of this work include the fact that only 6 discharge instructions with 28 sentences were used. Also, the fact that only the English to Chinese translation pair was considered in the study. In a way, the authors need to consider these to these two factors before making any strong claims on the generality of their study. Also, there seems to be a missing opportunity to study the usage of QE and BT together. Adding this condition would have made the study even stronger.
 
Overall, this paper is good, and I am recommending acceptance to the Main conference.
Decision: Accept-Main