SlowGuess/NAACL_2022_OCR huggingface dataset:
Pituxcoosuvarn et al. (2020) developed a system to warn MT users if a term could cause cross-cultural misunderstanding. The warnings weakly influenced awareness of translation sensitivity. However, the warnings prompted some participants to change their word choice. A similar intervention was proposed by Miyabe and Yoshino (2011). They tested several visualizations of accuracy, but found that indicating this did not change the performance of the participants in terms of time to repair poor translations. However, participants
anecdotally found some affordances more or less valuable. Picking the metric to visualize is important. Here, the visualization used a single value conflating accuracy and fluency. Research shows that fluency and adequacy are not equally considered (Martindale and Carpuat, 2018).
Moreover, users' perceptions need to be continually reassessed as underlying model architectures change. Literature from the Translation Studies discipline demonstrates that professional translators' impressions of MT are largely based on pre-neural models (O'Brien, 2012), but these impression still persist (Moorkens and O'Brien, 2017).
The interventions discussed so far focus on providing real-time feedback to end-users of MT systems. Complementing such approaches should be transparent information about overall model performance. Researchers have proposed various generic techniques for expressing model capabilities and limitations. For example, Mitchell et al. (2019) introduced model cards, intended to provide end-users and even indirect users with easily understandable information. One commercial computer vision  $\mathrm{API}^2$  shows examples of how low-quality lighting and visual occlusion can affect model performance.
Model-based performance metrics are not the only way to understand MT models and systems. Recent research questions the culpability of training and evaluation reference data as well (Freitag et al., 2020). Commonly-used corpora of reference translations can exhibit poor lexical and grammatical diversity (trending towards "translationese" (Graham et al., 2020)). Bommasani and Cardie (2020) demonstrated, in the context of summarization, how the evaluation of datasets can reveal the inconsistencies between the data's underlying properties and actual usages. One way to address this is to construct Datasheets for Datasets (Gebru et al., 2021). Inspired by standard datasheets for electronic components, Gebru et al. proposed documenting a dataset's motivation, composition, collection process, recommended uses, and so on.
We suggest that machine translation models can have similar model cards and datasheets. These would provide concrete examples of model and data boundaries while addressing folk-theories of model performance, which may or may not hold true. Additional research should evaluate the per
$^{2}$ https://modelcards.withgoogle.com/object-detection
ception of these cards and the effects, if any, on how people select models and use machine translation systems. The best development of cards-like reporting may require tweaking cards in the MT setting, and options should be thoroughly investigated. In the spirit of translation, it's important to have these resources accessible in the same languages that the systems support. This will help address the perception that access to the best apps and content is only available through English-language interfaces (Karusala et al., 2018).
4.4 Users' mental models
HCI has a long tradition of using one's mental model of a system to understand one's experiences of the system (Norman, 2013). As AI systems have become common in consumer-facing products, HCI researchers have studied these mental models of various AI systems (Kocielnik et al., 2019; Liao et al., 2020; Khadpe et al., 2020; Wu et al., 2019). For example, Wu et al. (2019) studied YouTube content creators' mental models of YouTube algorithms that rank, filter, and recommend content. Through one-on-one interviews, they identified users' three main algorithmic characterizations of the algorithm, and how these characterizations could serve as a conceptual framework to create new interactions.
End-users have mental models of how the systems use the text of suggestions that translation professionals provide. For example, King (2019) attempted to offer corrections to Google Translate. Google's own blog post about Google Translate indicates that it "learns" from user suggestions (Turovsky, 2016). With those expectations set, King submitted various corrections through the web interface and then later re-evaluated the system output for the corrected strings. From the author's perspective, the system did not change.
5 Implications on Future HCl+MT Research
In this paper, we have examined groups of translation users and the role that MT plays in their lives. Common needs that cut across these user groups include reliability, trust, and agency, and we examine these in more detail here.
5.1 Trust
Use of MT in high-stakes settings underscores the importance of ensuring these systems are safe
and reliable. Harms resulting from mistranslations range from inconvenience or embarrassment, to systematic representational harms. The most obvious way to reduce the frequency of such incidents is to improve the quality of the underlying translation model. Improvement in the accuracy of translation improves the user experience, and reduces the risk of harm. However, machine translation will never be perfect. We argue that improvement in the underlying models is neither sufficient for, nor necessarily indicative of, increasing the reliability and trust with these systems.
System safety engenders trust (Salem and Dautenhahn, 2015). Disentangling safety from model performance opens new questions for translation interface designers and human factors researchers. How can we design MT systems that are safer to use? An immediately feasible opportunity is to improve system transparency. Providing more information about system performance can help users calibrate their trust appropriately. Most MT systems remain opaque to end users, making it difficult for people to decide when they can rely on translation output.
At the model level, more detailed paradigms like Model Cards or Datasheets for Datasets could help resolve questions around how models perform, from where the data is sourced, and how systems incorporate user suggestions. These are effectively promises from translation services providers. Future research should investigate how users are influenced by and respond to these materials.
Trust is earned through repeated interaction. We argue that systems should provide more nuanced indications of uncertainty and quality. Those should be matched with user-facing affordances for acting on this information. For example, the user can see how alternate phrasing of their source text affects the output, but in a way that's comprehensible to someone with limited proficiency in the target language. This will require new designs and evaluation methods. For example, Zouhar et al. (2021) found that providing back-translations can improve user trust in outbound translations but did not correspond to an improvement in translation quality. As detailed in Section 3.3.2, careful task design will be necessary to prove the value of these affordances. Even if an affordance does not have a direct effect on language use in the system, it may still engender trust. Unfortunately, the direct effects of trust are difficult to measure in laboratory studies; similar
mixed effects are seen in human-robot interaction (Flook et al., 2019). Research on trust in machine translation should focus on specific user sets (e.g. students, immigrants, translation professionals) and circumstances.
Reliability issues are encountered when the machine translation system does not give expected results, with potentially negative consequences for the user. Understanding the limitations of machine translation and knowing when not to rely on the automated results are key to safe machine translation usage. Some of the "algorithmic disillusionment" (Eslami et al., 2018) users may experience with machine translation systems can also engender critical and vigilant usage of machine translation tools, especially in high-stakes scenarios.
5.2 Agency
We believe that carrying a sense of agency through the translation process will be a key part of future machine translation systems. We envision a world where individuals can effectively communicate across language boundaries without having to adapt their language use to suit the limitations of the system (e.g. by the use of "controlled language"). Although an utterance in one language may not have a precise equivalent in the target language, systems should provide affordances to help users understand those constraints. This will involve the design, implementation, and evaluation of new user experiences in translation systems. For example, current speech-to-speech translation systems assume that speech recognition is high-quality, and pass the transcripts directly to translation models. However, even human interpreters will ask the original speaker clarifying questions to best formulate a translation. Some MT models can incorporate longer context, but the translation UIs do not yet support ways to input context or clarify ambiguities in content or register. All these possibilities create large potential for human factors-informed research in the translation space.
For translators who intend to augment their work with machine translation technology, new interactions and interfaces are needed to establish an effective human-AI collaboration relationship. For example, Interactive Neural Machine Translation (Santy et al., 2019) provides human translators with on-the-fly hints and suggestions. However, recent research in human-AI collaboration showed that, when paired up with AI assistant, people might tend to over-rely on the AI system (Bucinca et al.,
2021; Bansal et al., 2021). How to maintain the agency of human translators while effectively making suggestions remains an open question in this line of research.
Our focus in this paper is on enumerating the contributions which HCI research can offer for safety and user-focused design, but it is equally important to recognize the limitations of what is achievable with improved design processes, user interfaces, or evaluations. In conclusion, we discuss limitations, highlighting complementary alternatives.
As we consider improving systems for use in high-stakes settings, we must also recognize fundamental concerns around the use and development of translation technology. Assuming that translation technology continues to improve, how will high-stakes users of machine translation understand the limitations? Under what conditions would machine translation be acceptable in medicine, policing, or immigration? How can we ensure the consent and center the needs of disempowered users in these interactions? Ethnographic and qualitative work with impacted communities, such as transnational migrants with low-language proficiency, can help establish consistent boundaries and good practices. Researchers must also consider how power dynamics shape the use and consequences of MT and avoid overly narrow conceptions of their users and those who are impacted.
Human communication is subtle and complex. For high-resource languages, machine translation models can successfully map many phrases across languages with high fluency and adequacy. We believe that the next frontier of research in machine translation is, through a human factors lens, to bring modern themes of trust and agency to the table. First, more robust qualitative and quantitative research is necessary to understand how people interact with MT systems today, and what improves the quality of those interactions. Machine translation users vary from professionals incidentally using MT as part of their workflow, to immigrants with low language proficiency who rely on MT systems to survive. From a design and implementation standpoint, we acknowledge that MT user interfaces have evolved very little. Given that people use machine translation to communicate in high-stakes scenarios, there is a great opportunity to
provide new affordances which ultimately engender trust and facilitate more authentic, trustworthy communication, and avoid algorithmic harm.
Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry S. Thompson, and Regina Weinert. 1991. The HCRC map task corpus. Language and Speech, 34(4):351-366.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA. Association for Computing Machinery.
Walter Benjamin. 1996. The Task of the Translator. Belknap Press of Harvard University Press, Cambridge, Mass., USA.
Yotam Berger. 2017. Israel arrests palestinian because facebook translated 'good morning' to 'attack them'. Haaretz.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075-8096.
Lynne Bowker. 2020. Chinese speakers' use of machine translation as an aid for scholarly writing in english: a review of the literature and a report on a pilot workshop on machine translation literacy. Asia Pacific Translation and Intercultural Studies, 7(3):288-298.
Lynne Bowker and Jairo Buitrago Ciro. 2015. Investigating the usefulness of machine translation for newcomers at the public library. *Translation and Interpreting Studies*, 10(2):165-186.
Lynne Bowker and Jairo Buitrago Ciro. 2019. Machine Translation and Global Research: Towards Improved Machine Translation Literacy in the Scholarly Community. Emerald Publishing Limited.
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1-21.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 249-256, Trento, Italy. Association for Computational Linguistics.
Prithwijit Das, Anna Kuznetsova, Meng'ou Zhu, and Ruth Milanaik. 2019. Dangers of machine translation: The need for professionally translated anticipatory guidance resources for limited english proficiency caregivers. Clinical Pediatrics, 58(2):247-249.
Susan Dumais, Robin Jeffries, Daniel M. Russell, Diane Tang, and Jaime Teevan. 2014. Understanding User Behavior Through Log Data and Analysis, pages 349-372. Springer New York, New York, NY.
Motahhare Eslami, Sneha R. Krishna Kumaran, Christian Sandvig, and Karrie Karahalios. 2018. Communicating algorithmic process in online behavioral advertising. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18, New York, NY, USA. Association for Computing Machinery.
Rebecca Flock, Anas Shrinah, Luc Wijnen, Kerstin Eder, Chris Melhuish, and Severin Lemaignan. 2019. On the impact of different types of errors on trust in human-robot interaction: Are laboratory-based HRI experiments trustworthy? Interaction Studies, 20(3):455-486.
Kent Fredholm. 2019. Effects of Google Translate on lexical diversity: vocabulary development among learners of Spanish as a foreign language. *Revista Nebrija de Lingüística Aplicada a la Enseñanza de Lenguas*, 13(26):98-117.
Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61-71, Online. Association for Computational Linguistics.
Batya Friedman. 1996. Value-sensitive design. Interactions, 3(6):16-23.
Batya Friedman, Peter Kahn, and Alan Bornig. 2002. Value sensitive design: Theory and methods.
Ge Gao, Bin Xu, David C. Hau, Zheng Yao, Dan Cosley, and Susan R. Fussell. 2015. Two is better than one: Improving multilingual collaboration by giving two machine translation outputs. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW '15, page 852-863, New York, NY, USA. Association for Computing Machinery.
Ge Gao, Naomi Yamashita, Ari MJ Hautasaari, Andy Echenique, and Susan R. Fussell. 2014. Effects of public vs. private automated transcripts on multi-party communication between native and non-native english speakers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '14, pages 843-852, New York, NY, USA. Association for Computing Machinery.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM (CACM), 64(12):86-92.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 72-81. Association for Computational Linguistics.
Kotaro Hara and Shamsi T. Iqbal. 2015. Effect of Machine Translation in Interlingual Conversation: Lessons from a Formative Study. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, pages 3473-3482, Seoul, Republic of Korea. Association for Computing Machinery.
Basil Hatim and Ian Mason. 1990. Discourse and the Translator. Routledge.
Ryosuke Isotani, Kiyoshi Yamabana, Shin-ichi Ando, Ken Hanazawa, S. Ishikawa, Tadashi Emori, Ken-ichi Iso, Hiroaki Hattori, Akitoshi Okumura, and Takao Watanabe. 2002. An automatic speech translation system on pdas for travel conversation. In Proceedings of the Fourth IEEE International Conference on Multimodal Interfaces (ICMI), pages 211-216.
Naveena Karusala, Aditya Vishwanath, Aditya Vashistha, Sunita Kumar, and Neha Kumar. 2018. "Only if you use English you will get to more things": Using smartphones to navigate multilingualism. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA. Association for Computing Machinery.
Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. 2019. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics-System
Demonstrations, pages 117-122, Florence, Italy. Association for Computational Linguistics.
Pranav Khadpe, Ranjay Krishna, Li Fei-Fei, Jeffrey T. Hancock, and Michael S. Bernstein. 2020. Conceptual metaphors impact perceptions of human-air collaboration. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2):1-26.
E.C. Khoong, E. Steinbrook, C. Brown, and A. Fernandez. 2019. Assessing the use of Google Translate for Spanish and Chinese translations of emergency department discharge instructions. JAMA Internal Medicine, 179(4):580-582.
Katherine M. King. 2019. Can Google Translate be taught to translate literature? A case for humanists to collaborate in the future of machine translation. *Translation Review*, 105(1):76-92.
Chunyu Kit and Tak Ming Wong. 2008. Comparative evaluation of online machine translation systems with legal texts. *Law Library Journal*, 100:299-321.
Rafal Kocielnik, Saleema Amershi, and Paul N Bennett. 2019. Will you accept an imperfect AI? exploring designs for adjusting end-user expectations of AI systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1-14.
Joseph A. Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User Modeling and User-Adapted Interaction, 22(1):101-123.
J. Clayton Lafferty. 1974. The Desert Survival Situation Problem: a Group Decision Making Experience for Examining and Increasing Individual and Team Effectiveness. Human Synergistics.
Elina Lagoudaki. 2006. Translation memories survey 2006: Users' perceptions around TM use. In Proceedings of ASLIB Translating and the Computer, volume 28.
Elina Lagoudaki. 2008. Expanding the Possibilities of Translation Memory Systems: From the Translator's Checklist to the Developer's Design. Ph.D. thesis, Imperial College, London.
Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020. Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, New York, NY, USA. Association for Computing Machinery.
Daniel J. Liebling, Michal Lahav, Abigail Evans, Aaron Donsbach, Jess Holbrook, Boris Smus, and Lindsey Boran. 2020. Unmet needs and opportunities for mobile translation AI. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, New York, NY, USA. Association for Computing Machinery.
Marianna Martindale and Marine Carpuat. 2018. Fluency over adequacy: A pilot study in measuring user trust in imperfect MT. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 13-25, Boston, MA. Association for Machine Translation in the Americas.
Michael Mattmiller. 2021. Personal communication.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* '19, page 220-229, New York, NY, USA. Association for Computing Machinery.
Mai Miyabe and Takashi Yoshino. 2011. Can indicating translation accuracy encourage people to rectify inaccurate translations? In Human-Computer Interaction. Interaction Techniques and Environments, pages 368-377, Berlin, Heidelberg. Springer Berlin Heidelberg.
Joss Moorkens and Sharon O'Brien. 2017. Assessing user interface needs of post-editors of machine translation. In Dorothy Kenny, editor, Human Issues in Translation Technology. Routledge.
Michael J Muller and Sarah Kuhn. 1993. Participatory design. Communications of the ACM, 36(6):24-28.
Ana Niño. 2009. Machine translation in foreign language learning: language learners' and tutors' perceptions of its advantages and disadvantages. *ReCALL*, 21(2):241-258.
Donald A. Norman. 2013. The design of everyday things. Basic Books.
Sharon O'Brien. 2012. Translation as human-computer interaction. Translation Spaces, 1:101-122.
Sharon O'Brien. 2017. Machine translation and cognition. In John W. Schwieter and Aline Ferreira, editors, The Handbook of Translation and Cognition, chapter 17, pages 313-331. Wiley, Hoboken, NJ, USA.
Dan R. Olsen. 2007. Evaluating user interface systems research. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST '07, pages 251-258, New York, NY, USA. Association for Computing Machinery.
A. Panayiotou, A. Gardner, S. Williams, E. Zucchi, M. Mascitti-Meuter, A.M. Goh, E. You, T.W. Chong, D. Logiudice, X. Lin, B. Haralambou, and F. Batchelor. 2019. Language translation apps in health care settings: Expert opinion. *JMIR mHealth uHealth*, (4).