SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel J. Liebling
601 N. 34th St.
Seattle, WA 91803 USA
dliebling@google.com
Samantha Robertson*
253 Cory Hall
Berkeley, CA 94720 USA
samantha robertson@berkeley.edu
Katherine Heller
1600 Amphitheatre Parkway
Mountain View, CA 94043 USA
kheller@google.com
Wesley Hanwen Deng†
5000 Forbes Avenue
Pittsburgh, PA 15213 USA
hanwend@andrew.cmu.edu
Machine translation models are embedded in larger user-facing systems. Although model evaluation has matured, evaluation at the systems level is still lacking. We review literature from both the translation studies and HCI communities about who uses machine translation and for what purposes. We emphasize an important difference in evaluating machine translation models versus the physical and cultural systems in which they are embedded. We then propose opportunities for improved measurement of user-facing translation systems. We pay particular attention to the need for design and evaluation to aid engendering trust and enhancing user agency in future machine translation systems.
Evaluation of Machine Translation (MT) system performance seeks to close the gap between automated measurements and human judgments of translation quality. The MT evaluation literature contains a variety of work to quantify how closely human and automated measurement are linked. The models being evaluated in this literature are often complex combinations of traditional NLP and neural components. In this paper, we consider these systems as major but separate components of larger user-facing translation systems embedded in larger social contexts. These translation systems serve diverse user needs that go beyond simple translation of text to text. Grounded in research across domains, we motivate new research in three specific
areas. First, translation system users, their motivations, and usage practices deserve further inquiry. Second, we argue that end-users (both individual and institutional) evaluate system performance differently from the system developers. End-users necessitate evaluation of the entire system, rather than just the model. Finally, we argue that significant research is necessary around nurturing trust in MT.
2 Machine translation users
Web- and app-based translation services, as well as translation APIs, are widely available. People who use these systems span geographic, socioeconomic, and occupational categories. We first discuss the contexts of consumers, a group of people who encounter machine translation as part of transacting their daily lives, but not as part of their vocation. Then we discuss the needs of translation professionals, people who use machine translation as part of their occupational workflows.
2.1 Primary consumers
We consider consumers as individuals who have specific needs for communicating across languages. As an additional factor, we consider individuals with "high-stakes" communication needs as a specific subset.
2.1.1 General public
Lack of language proficiency in a target language can be temporary (e.g. while traveling) or more persistent. In the former case, MT-based systems for travel translation have existed in various forms from personal digital assistants (PDAs) (Isotani
* Work while the author was an intern at Google Research.
† Work while the author was a student at University of California, Berkeley.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 229 - 241
et al., 2002) to unidirectional offline speech-to-speech translation devices, to bidirectional devices embodied as earbuds that stream audio and translations to cloud services. Although these products view international travelers as their target market, the language needs of travelers are narrower than general purpose translation. Travelers' actual reported translation needs are lower than their perceived needs (Liebling et al., 2020).
Students use translation systems to complete homework assignments, especially for language-learning. Though educators hold varying opinions about this, translation systems are frequently adopted by second-language learners (Niño, 2009). The use of translation systems can improve students' lexical diversity (Fredholm, 2019) and the students have complex, nuanced perceptions of system quality (Xu, 2021).
Immigrants with language needs often turn to online or app-based translation when they have access to the associated technology (phones, laptops, etc) (Liebling et al., 2020). Proficient individuals also use machine translation. Using a second language for academic writing, which is domain-specific and highly structured, can be difficult; writers turn to machine translation to verify their word choices or even translate larger blocks of text (Bowker, 2020). Importantly, primary consumers may be aided by other individuals. For example, librarians will introduce the use of translation web sites or apps to immigrants with limited proficiency in their new country's language (Bowker and Buitrago Ciro, 2015). In this scenario, the first party (librarian) evaluates the quality of the system based on repeated interaction, whereas the second party (library patron) evaluates the quality based on a single interaction.
2.1.2 High-stakes use cases
A "high-stakes" scenario speaks to the potential impact and consequences of its outcomes. There is growing evidence that MT is widely used in high-stakes contexts such as healthcare (Vieira et al., 2020), law (Vieira et al., 2020; Kit and Wong, 2008), immigration (Torbati, 2019), and policing (Vieira et al., 2020; Liebling et al., 2020; Berger, 2017). Policymakers may discourage the use of MT () because the quality and reliability of machine translation in these settings is less well understood and the potential for harm from mistakes is high. As such, it is difficult to know how widespread these practices really are. For instance, many examples of MT use in policing have come to light
only through journalism (Berger, 2017) and legal cases (Vieira et al., 2020).
The use of MT in the medical domain is comparatively better documented. Because language barriers in healthcare are associated with disparities in quality of care and health outcomes (Wilson et al., 2005), MT holds great potential for impact. When a language-concordant provider is not available, healthcare professionals are expected to use professional language services (Randhawa et al., 2013). For example, during COVID-19 mass vaccination in 2021, the City of Seattle (United States) used a third-party live medical interpreting service provided over videoconference (Mattmiller, 2021). However, professional interpretation or translation can be difficult to arrange, especially in emergent situations, at smaller clinics, and for less commonly spoken languages (Turner et al., 2013; Sentürk et al., 2021). In these situations, healthcare providers have increasingly turned to MT (Turner et al., 2013; Randhawa et al., 2013). We return to perceived medical translation quality in Section 4.2.
2.2 Translation professionals
Translation professionals take content written in a source language and adapt it for use in a target language. They interact with this content on a transactional basis. This differentiates them from literary translators whose work is seen as a creative, artistic act (Benjamin, 1996; Hatim and Mason, 1990). In this section, we focus on the former.
Professional translators use software tools to manage their workflows. These computer-aided translation (CAT) systems aim to increase translation throughput. Specifically, translation memory (TM) features track previous translations of terms and phrases and suggest the prior translation, creating efficiency and consistency. Some tools augment the workflow with machine translation, as in "post-editing" workflows. In this workflow, source text is machine translated first, then corrected by professionals. Although post-editing can improve throughput and lower cost, human translators have resisted this task (O'Brien, 2017, 2012) as it changes the nature of the work from translator to editor, thereby reducing the agency of the worker. Complex issues of agency, quality, and workplace demands lead to multifaceted perception of machine translation.
3 Evaluating translation systems
Evaluating translation systems used by humans is difficult because the task, system design, and model performance are intertwined.
3.1 Measurement of translation systems
A common problem in systems measurement is disentangling whether effects are due to the manipulation of experimental variables, or due to complex interactions within the systems (both technical and human) (Olsen, 2007). For user-facing translation systems, it is important to distinguish between the performance of the underlying text-to-text translation models and the systems which they compose.
3.1.1 Model evaluation techniques
MT model performance on a corpus is frequently measured using the BLEU score (Papineni et al., 2002), amongst other metrics. Large increases in BLEU correlate with improved human judgment of translation quality. This makes it useful for driving progress in model architectures and training regimes. However, marginal gains in BLEU scores cease to be meaningful indicators of overall improvement; Callison-Burch et al. (2006) argued that improvements in BLEU were "neither necessary nor sufficient" for material improvements in model quality. Because of these limitations, other metrics exist, such as METEOR (Banerjee and Lavie, 2005) and COMET (Rei et al., 2020).
Techniques like quality estimation provide tools for predicting the quality of model output on a source string. Projects such as OpenKiwi (Kepler et al., 2019) provide interactive visualization of quality at the phrase and term level, but it remains unresolved what the effects are of showing quality estimates to end-users (Turchi et al., 2015). Recent papers provide evidence that showing quality metrics to end-users does not significantly impact perception or behavior (e.g. Turchi et al. (2015)). This argues that the metrics favored by developers aren't meaningful to end-users. Other subfields of applied artificial intelligence have come to similar conclusions. For example, Konstan and Riedl (2012) showed that the user experience in which recommender systems algorithms were embedded, significantly affected users' value perceptions.
3.2 Scale of studies
Similar to translation systems, web search engines are another example of how algorithms — in this
case, ranking algorithms — are embedded in a larger system and context. Around the same time as the first web-based translation systems emerged, researchers published the first analysis of web search logs (Silverstein et al., 1999). Ranking metrics existed in the Information Retrieval (IR) community before web search engines (Salton and McGill, 1983), but the existence of search logs became a rich source of information on information-seeking behavior and use of search systems (Dumais et al., 2014; White, 2016). Various search log datasets were published, although widespread adoption of stricter privacy policies limited further releases. To our knowledge, there is no published research that uses machine translation logs to understand the behavior of online MT users. Instead, most research on how translation systems (versus models) comes from short-term user studies, ethnographic and qualitative investigation, and news reports. It seems unlikely that large-scale analyses will become available, so the burden of proof lies in small-scale studies.
3.3 Experimental protocols
Given that small-scale studies will likely compose the bulk of user-facing translation systems research, we now enumerate key challenges for reproducible systems research in this space. By systems evaluation, we mean studies where the MT model itself is fixed, but user interface affordances and interaction techniques are tested.
3.3.1 Participant selection
Second-language proficiency varies highly across individuals. In order to reduce experiment noise due to language ability, researchers need to find multilingual participants whose skills are roughly matched. A thorough experiment would also measure the proficiency of participants, but for simplicity, these skills are typically self-rated. Since model qualities vary across language pairs and direction of the translation, the ideal system setup will use language pairs with similar model quality.
Cultural backgrounds of those participants also vary. Wang et al. (2013) measured effects of machine translation on brainstorming between Chinese (Mandarin Chinese-speaking) and American (English-speaking) dyads. They measured quantity and frequency of idea generation in human-or MT-mediated dialogues. They noted that cultural bias towards or against quantity over quality might affect the results. The experimental design
was asymmetrical; conversations were mediated through English in both conditions. Fully exploring the space of multilingual interaction requires evaluating systems across languages and linguistic backgrounds, which adds cost and time.
Pituxcoosuvarn and Ishida (2018) designed a system to allow multilingual users with varying degrees of proficiency to converse with each other through text. Participants could communicate through their L2 proficiency or through machine translation. In this way, the researchers could evaluate quality (measured via communication difficulty and morbidity) using mismatched participants.
3.3.2 Task design
Researchers can use a variety of tasks to encourage dialogue between participants, in order to study multi-party communication in translation. Of particular importance is eliciting spontaneous speech, as that is closer to how humans would use translation systems for communication in the real world. These needs are mismatched with the transactional affordances of contemporary translation web sites. Liebling et al. (2020) found that people used speech translation apps in a variety of rich, complex interactions from simple quotidian purchasing transactions to sensitive dialogues such as negotiating working conditions with an employer.
To elicit spontaneous speech, some established protocols exist. Pituxcoosuvarn et al. (2020) developed a MT-mediated chat app with a desert-island survival task (Lafferty, 1974). In this task, participants must choose and discuss items that they would desire should they be stranded on a desert island. The items are meant to elicit spontaneous responses from co-participants. Both Hara and Iqbal (2015) and Gao et al. (2014) used a storytelling task where participants were given a word list and encouraged to construct a story using those words. The map task (Anderson et al., 1991) gives participants paired maps; one participant's map contains a route and that person's goal is to get the other participant to follow the same route. Gao et al. (2015) used this task to study grounding in MT-mediated conversation.
These tasks give the experimenters some control over the conversation topic, which can reduce sources of error in across-subjects that comes from translation quality varying across different domains. If participants are given entirely open-ended tasks, their selection of domain will introduce additional noise. The tasks also have natural endpoints: all
stimulus words are used, all differences are found, or route planning is complete. We argue that in the name of reproducibility, researchers should use existing tasks whenever possible, and share stimulus material resources as appropriate. Several of these conversation elicitation protocols have extensive validation in the social sciences. Because these tasks have been used for monolingual research as well, studies that employ them can serve as useful baselines when evaluating the effect of MT mediation on human conversations.
4 Trust in translation systems
Although some argue that MT has reached "human parity," many challenges remain. Output translations still contain errors, especially for lower resource languages. While minor errors in low-stakes contexts may be inconsequential, the unpredictability and lack of control over MT output raises concern regarding these systems' reliability in real world settings. Researchers and users have also documented systematic errors and biases in MT system output, which can result in longer term, representational harms to minoritized and oppressed groups (Stanovsky et al., 2019). In this section, we review important concerns in the design and use of safe, reliable, and useful machine translation systems. In the following section we identify open research questions at the intersection of HCI and MT that can begin to address some of these challenges.
Systematic errors and biases have been documented across NLP models and tasks. For example, Bolukbasi et al. demonstrated that word embeddings can learn stereotypical associations between gender and occupations (Bolukbasi et al., 2016). Machine translation models also struggle with semantic gender; for example, difficulties arise when translating a sentence from a language with gender-neutral pronouns to one with gendered pronouns while avoiding reinforcement of gender stereotypes (Stanovsky et al., 2019). In 2021, a major web translation site addressed this by allowing users to select from multiple target translations with different genders. This gives the user agency to control the output of the system, compensating for issues with the underlying model.
<sup>1</sup>https://ai.googleblog.com/2020/04/a-scalable-approach-to-reducing-gender.html
4.1 Design asymmetries
Typically, machine translation models are constructed by people with access to substantial computation resources, embedded in systems with simple user interface wrappers. Early speech-to-speech translation devices had interfaces which mirrored standard web interfaces, consisting of two text boxes and language selection (Waibel et al., 2003; Zhou et al., 2003). These systems were evaluated mainly on performance of the speech and translation models, rather than system performance from a user's perspective. Notable exceptions include the Verb Mobil project (Wahlster, 2000); designers identified many concerns that influenced system design, including recovery from error. Similarly, the BBN TransTalk device (Prasad et al., 2013), developed as part of the military-funded TRANSTAC research, also included a way for at least one party to estimate the quality of the translation. These systems are designed from a single perspective, and reflect the power dynamics in the environment (RISKU et al., 2021; Paullada, 2020). People with low socioeconomic status use machine translation systems to access social capital (Liebling et al., 2020), but do not appear to be consulted during system design.
For professional translators, translation tools are often designed with throughput in mind, rather than the usability for end-users. Lagoudaki surveyed users of translation memory (TM) systems and concluded that "end-users' demands seem to have been only of subordinate interest" (Lagoudaki, 2006, 2008). Moorkens and O'Brien (2017) reached a similar conclusions through a separate survey and structured interview process.
Translation systems research will therefore benefit from direct user involvement, using techniques like participatory design (Muller and Kuhn, 1993; Schuler and Namioka, 1993) and value sensitive design (Friedman, 1996; Friedman et al., 2002). For example, we recruited 9 immigrants to the United States (4 men, 5 women) for a multi-day participatory design exercise. Participants first articulated their language journeys. Then, working in teams of three, they constructed fake translation devices out of foam and paper. The variety of devices created spoke to the need for translation devices designed for use in various environments and highlighted how design can reflect the asymmetry of translation needs across conversation participants.
4.2 Use in High-Stakes Settings
As outlined in section 2.1.2, MT is used when human interpreters are not available in high-stakes settings such as healthcare and policing. These settings pose unique challenges because of the power dynamics between parties, domain-specific language, and ramifications of precise word choice.
Several studies investigated whether MT is reliable enough to be used in healthcare settings. Findings from the provider's perspective are mixed; while some studies show promising performance for languages like Spanish and Chinese (Khoong et al., 2019), others found that MT was not considered reliable for lower-resource languages (Das et al., 2019; Taira et al., 2021). Beyond translation accuracy, providers expressed concern that entering sensitive healthcare-related information into a free translation website may violate patient privacy (Vieira et al., 2020).
Less work has examined patients' perspectives. Turner et al. simulated emergency scenarios with English-speaking healthcare workers and low-English proficiency (LEP) Spanish- or Chinese-speaking patients, communicating via either Google Translate or an app that provided pretranslated phrases (Turner et al., 2019). Patient-participants found it difficult and unpleasant to communicate their needs and understand instructions due to translation inaccuracies and delays in using the interface. Panayiotou et al. (2020) worked with both elder-care providers and older people with LEP to understand their perceptions of and attitudes towards using MT. While many participants were excited about the potential to improve their communication, they also raised concerns about translation accuracy and the difficulty of learning to use the technology.
Patient needs for translation are not limited to patient-provider interactions. People frequently turn to the web for healthcare information prior to, and soon after visits to healthcare providers (White and Horvitz, 2013). Some LEP immigrants selected providers based on availability of interpreter services, but the ancillary processes of accessing healthcare (such as calling to make an appointment) was an unmet need and opportunity for MT (Liebling et al., 2020).
Although there is consensus that professional translation or interpreters are always preferable in healthcare settings, there also seems to be acceptance that MT is a valuable last resort. Several
studies, which included patient perspectives, recommend opting for tools that provide fixed, professionally translated phrases where possible, in order to guarantee accuracy and patient privacy (Spechbach et al., 2017; Turner et al., 2019; Panayiotou et al., 2019, 2020). When they choose to use MT, providers are urged to exercise caution. Some suggestions include using simple language (Khoong et al., 2019) and watching patients' facial expressions to detect confusion (Randhawa et al., 2013), although the effectiveness of these strategies has not been evaluated. This suggests a promising direction for future work is to develop systems that provide safeguards and guidance for generating high quality translations.
4.3 Methods for calibrating trust
Modern translation systems treat translation queries as a stateless, single-shot problem. While user insight can improve on this, popular interfaces do not allow users to introspect on system performance. The interfaces provide few, if any indicators of uncertainty or quality. Translation system providers frame support for language pairs as a binary encoding (supported or not) despite a wide distribution of quality across language pairs and directionality within a language pair (i.e. French  $\rightarrow$  Japanese vs. Japanese  $\rightarrow$  French).
Individuals understand that the outputs of translation systems are not perfect (Liebling et al., 2020), but strategies for overcoming system limitations vary. Bowker and Buitrago Ciro (2019) argue for Machine Translation Literacy, a set of strategies for understanding and working with the limitations of MT. In general, the lack of transparency with respect to system behavior makes it more difficult for end users to develop and validate such strategies. In light of this, some research has begun to explore how to surface more detailed information about system behavior to end-users.