On The Real-world Performance of Machine Translation: Exploring Social Media Post-authors’ Perspectives
Year: 2023

Abstract:
Many social networking sites (SNS) offer machine translation of posts in an effort to increase understanding, engagement, and connectivity between users across language barriers. However, the translations of these posts are still not 100% accurate and can be a cause of misunderstandings that can harm post-authors’ professional or personal relationships. An exacerbating factor is on most SNS, authors cannot view the translation of their own posts, nor make corrections to inaccurate translations. This paper reports findings from a survey (N = 189) and an interview (N = 15) to explore users’ concerns regarding this automatic form of machine translation. Our findings show that users are concerned about potential inaccuracies in the meaning of the translations of their posts, and would thus appreciate being able to view and potentially correct such translations. Additionally, we found that when users write posts in their native language, they write them for specific audiences, so they do not always want them translated. This underscores the urgency of providing users with more control over the translation of their posts.

heading: 1 Introduction
text: Social networking sites (SNS) allow users to connect with people from various language backgrounds, and a sizable proportion of SNS users writes posts in a language that not everyone in their audience understands [11, 12, 21]. To help people with diverse language backgrounds understand the posts of users who write in an unfamiliar language, SNS offer machine translation (MT) to automatically translate users’ posts from the original language into a language the reader understands [11, 12]. This may, in turn, increase users’ engagement with an increasingly global audience [12].
However, currently SNS do not give users any control over the translation of their own posts. For
example, Facebook 1 allows the readers of posts with options to select “Languages you’d like to have posts translated into”, “Languages you don’t want to be offered translations for”, and “Languages you don’t want automatically translated.” In contrast, the only settings that are available for post-authors are “A feature that lets you post multiple language versions of a status” [5] and “Turn off translations” for each post. However, it still does not allow post-authors to view or edit the translated version of their own posts, nor does it allow them to choose the languages they would like their post(s) to be translated into. Unlike Facebook, Instagram currently allows authors to read translations of their posts and stories but, like Facebook, offers no control over these translations. Other popular SNS (e.g., Twitter, LinkedIn) currently do not offer any controls over the MT feature, making it difficult for authors to judge the quality of the translations of their posts [5, 4]. In fact, the only way that users of these SNS are exposed to MT-translated posts is by reading the translations of other users’ posts.
Casacuberta et al. [3] find that machine translations are often not accurate and can inadvertently distort the intended meaning of a post, which can lead to misunderstandings among SNS users. As a result, SNS users report feeling insecure when they think about how their posts might be translated and whether their translated posts accurately convey their intended meaning [3]. Beyond this, there is very little research done regarding how users feel about the translation of their posts, the use of automatic MT, or the fact that, in most cases, users have no control over the translation that gets posted. The current study bridges this gap by addressing the following research questions:
• RQ1: How does awareness of MT influence authors’ posting behaviors? Does it change their tendency to post about sensitive topics?
1https://www.facebook.com/settings?tab= language
302
• RQ2: To what extent do authors appreciate MT? What are their concerns about MT?
• RQ3: To what extent do authors desire control over the MT of their posts?
Next, we discuss existing literature related to the concepts that ground our study. We then describe our methodology, the results obtained from data analysis, and the implications of our findings. Finally, we present limitations and future directions.

heading: 2 Related Work
text: In this section, we review research on MT methods, on tools that give authors control over their social media posts, and on authors’ considerations when posting sensitive information to a growing and increasingly diverse audience.

heading: 2.1 Machine Translation
text: Existing MT algorithms can be categorized into rule-based, statistical, hybrid, and neural MT [2]. While statistical MT is commonly used in SNS, it lacks user interaction [2]. Hybrid MT was proposed to improve translation quality by combining the fluency of statistical MT with the content preservation of rule-based MT, but it is not effective with misspellings or missing characters [2]. Neural MT systems are more noise-robust but often fail to accurately convey the original post’s meaning [7, 8, 15], leading to lower user engagement [11, 12].
To overcome these barriers, Lim et al. [11] presented a system called SenseTrans that includes an emotional and contextual explanation of the translated post generated by a combination of natural language analysis and MT. Their study found that SenseTrans provides a greater understanding of the posts and increases the willingness of the audience to engage with posts written in foreign languages [11, 10]. However, while SenseTrans helps readers understand the overall meaning of posts, it still fails if posts are mistranslated or possess inaccurate keywords. Further, Lim et al.’s work is more applicable to the readers than to the writers of SNS posts. To bridge this gap, we seek to specifically understand authors’ concerns regarding MT and investigate whether they would like to have more control over the translation of their own posts.

heading: 2.2 Post-authors’ Control
text: While various tools have been developed to improve post translation quality and audiences’ understanding of posts in unfamiliar languages, little
existing work considers providing authors control over the translation of their posts [5]. Gupta [5] points out that authors’ control over their posts can be increased by a) allowing them to decide whether they want their posts to be translated, b) giving them control over which languages their posts can be translated into, c) making them aware of privacy controls to manage which audiences can view their translated posts, and d) giving them controls to manage whether and how sensitive content in their posts will be translated [5]. Toselli et al. [20] corroborate the idea that the ability to edit the MT output before sharing a translated post is essential to obtaining high-quality translations between any two pairs of languages. They propose interactive MT, which would use interactive pattern recognition to learn from the iterative edits made by human translators to the MT of posts [20]. However, their idea has, to our best knowledge, never been implemented in the context of SNS post translation.

heading: 2.3 Sensitivity of Social Media Posts
text: SNS users sometimes write posts about sensitive topics, and low-quality MT could pose a privacy threat if their writing is mistranslated [13, 22]. At present, Facebook offers controls to restrict the audience of a post, but it has no option to disable or alter the MT of sensitive posts. In fact, both Facebook and Instagram only offer control over MT to the audience of a post, leaving no recourse for authors who want more control over the translation of posts they themselves share.
Reflecting upon this limitation, Gupta [5] suggests that authors should be given control over not only how they want their posts to be perceived but also who the audience should be. This could involve asking for permission before translating each post, or an “obfuscation option” to hide sensitive content written in the original language before making the translation public [9]. The lack of such features may put authors at risk of privacy invasion and career-ruining misunderstandings [9, 5].

heading: 3 Methodology
text: We conducted a survey study (N = 189) and an interview study (N = 15) to investigate SNS users’ awareness of, appreciation of, and desire for control over the MT of their posts. Here we discuss the development of our survey questions and interview script and outline participant recruitment and data collection for both studies. The studies were
approved by our IRB, and we used an exploratory approach to answer our research questions.

heading: 3.1 Survey Design
text: The purpose of the survey study was to examine users’ awareness and concerns regarding the translation of their SNS posts and their desire for control over the translation of their posts. A detailed breakdown of the survey has been uploaded to the OSF2 for reproducibility purposes. The survey consists of 39 items addressing five major categories: demographic information, SNS and language preferences, awareness of and experiences with MT and its quality, the perceived sensitivity of various categories of posts, and preferences for control over MT. Most of the survey questions were adapted from pre-existing surveys [19, 1, 18, 13, 22].
The first author ran a pilot study with 12 humancomputer interaction experts to help ensure the content of the questions aligned with the goals of the study, to assess the clarity of the questions, and to estimate the survey completion time. Feedback included changing the format of some questions to make them easier to comprehend and navigate.

heading: 3.2 Survey Participants and Procedures
text: We used the web-based recruiting platform Prolific to recruit participants in two iterations. We used strict pre-screening criteria to ensure that the recruited participants represented the intended audience (i.e. bi/multilingual participants). In iteration 1, we required that participants be 18+ years old, located in the United States, have an SNS account, and know one or more other languages in addition to English. In iteration 2, to acquire more relevant data, we added, “Participants were raised with two or more languages” to our recruitment criteria. However, we did not put any restrictions on the languages. Our recruitment strategy was reasonably successful: 48% of the participants reported to write posts on SNS in a language other than English—the reported languages included both rich-resourced languages (Chinese, Spanish, Arabic, German, and French) [6] and lowresourced languages (e.g., Japanese, Korean, Cantonese, Ukrainian, Creole, Nepali, Hindi) [14].
Participants took on 7 minutes and 30 seconds to complete the survey and were rewarded with $1.50 upon completion. We recruited 200 participants in
2https://osf.io/f5kcz/?view_only= 0fb3ff9458c74e87a0d4c7b7cceb6636
iteration 1 (age range: 18-64; 97 male, 98 female, 4 non-binary, 1 no answer; 177 born in the US, 2 in China, 1 each in Korea, New Zealand, Malaysia, the Philippines, India, Ukraine and Yemen, 4 no answer) and 41 participants in iteration 2 (age range: 18-54; 14 male, 26 female, 1 non-binary; 40 born in the US, 1 in Saudi Arabia), for a total of 241 participants. Of these, 189 responses met the inclusion criteria and were used for data analysis.

heading: 3.3 Interview Design
text: Our semi-structured interviews further investigated the factors that users account for when posting on SNS, collecting qualitative data to complement the quantitative data collected in the survey. The interview was guided by 20 open-ended questions3, probing four categories: language use while posting on SNS, users’ thoughts about the quality of MT, how users decide which kinds of posts are acceptable to be translated without prior permission, and additional controls which would make users feel safer when posting on SNS. Most of the interview questions were written as a means to further explore the questions asked in the survey study.

heading: 3.4 Interview Participants and Procedures
text: Participants were recruited among the survey participants, and additional participants were recruited through posts on various SNS, for a total of 15 participants. Original survey participants received a $5 gift card for the interview, while new participants received $10 for participating in both the survey and the interview. All interviews were conducted via Zoom. After instructing participants to change their usernames to avoid recording identifiable information, the first author obtained consent to record the interview for analysis purposes.

heading: 4 Results
text: This section presents the findings of our two studies. Analysis was conducted to explore the effect of users’ familiarity with MT (RQ1), attitudes towards MT (RQ2), and desire for control over the translation of their posts (RQ3). Both quantitative and qualitative data were collected from the survey; only qualitative data were collected from the interviews. We analyzed the quantitative data using correlation, regression, and t-tests, and we used thematic analysis for the qualitative data.
3https://osf.io/wqvrc/?view_only= 6b2bc034dcb940a19a62e1ec9a73d8ec

heading: 4.1 Familiarity with MT (RQ1)
text: About 85% of our participants were familiar with MT and about 80% of participants reported having not just knowledge of MT but encountered it on SNS. Given participants’ overall familiarity and experience with MT, we consider our sample to be a good representation of SNS users’ opinions regarding the translation of posts.
We subsequently explored the relationship between participants’ familiarity with MT and their posting frequency on SNS. Fig. 1 shows that users who are more (less) familiar with MT are more (less) likely to post on SNS (r = 0.2, p = 0.006).

heading: 4.2 Attitudes towards MT (RQ2)
text: Fig. 2 shows that a majority of participants found the quality of MT to be good in general. Given these results, we examined the relationship between participants’ perception of MT’s ability to accurately convey the meaning of their posts and their posting frequency. We found that participants who post on SNS more frequently find MT of posts to be more accurate in terms of actual meaning (i.e.,
dictionary translation) and connotative meaning (i.e., pragmatic intention), but only the former was significant (actual meaning: r = 0.16, p = 0.03; Fig. 3; connotative meaning: r = 0.14, p = 0.06).
We further examined the relationship between participants’ prior experience with MT and their perceptions of MT’s ability to accurately convey the meaning of their posts. We found that participants who have experience with MT find it to be less accurate in terms of connotative meaning (t(19.94) = 3.3, p = 0.004; Fig. 4) but not actual meaning (t(18.49) = −0.54, p = 0.59).
We also examined the relationship between participants’ perception of the accuracy of MT in terms of connotative meaning and their frequency of writing posts in their native language (i.e., not in English). We found that if participants who find MT translations more (less) accurate in terms of connotative meaning tend to post more (less) often in their native language (r = 0.27, p = 0.01).
The interview results regarding participants’
opinions about the quality of MT on SNS aligned with these findings. Firstly, almost all participants reported that they found Google Translate to be more accurate than the MT on SNS, for example:
Using Google translate in the very beginning, its infancy was interesting because I learned a little bit, but in the last few years [...] it is improved greatly over what it was. And there are words that it’s finding from the Internet, where people have input or given feedback. That really makes more sense in the context. – P71
Almost all interviewees found issues with the quality of MT on SNS, especially while reading translations of slang or posts with connotative meaning. Seven participants provided examples of mistranslations of posts with a connotative meaning that they had encountered on SNS. P71 shared an example where an English loanword was used by Polish speakers, but the MT did not account for this and instead selected an unnatural wording:
[phrase in Polish] means “end of the week", which means weekend, but because Polish people don’t have a word for weekend, even on national TV, they say weekend. They’ve just adopted a word that everyone pretty much understands but it’s not necessarily a word of that native language. – P71
Similarly, two interviewees mentioned that they often saw slang or idioms translated incorrectly; they speculated that this may be because slang and idioms may need context to understand the meaning. For example, one said:
In Chinese, we have this some sort of pride idiom called “saving face" in Chinese and if you were to directly translate into English is called “Satan face". – P9
A few interviewees commented that a word can have more than one meaning. Therefore, if the MT chooses the wrong translation of a word, it may give the sentence a completely different meaning. Additionally, a few participants mentioned that incorrect translations can distort not the literal meaning but the intended tone of a post, e.g. making it sound harsher than in the original language.
“my wife completes my life" was translated as “my wife finishes my life" from Bangla to English – T22
I wrote [sentence in Polish] and the translation came as “My dad pissed me off yesterday!". “annoyed" was translated to “pissed" which made it sound more severe; definitely a heightened word that I wouldn’t post myself on social media so, it matters if the tone gets translated correctly or not. – P71

heading: 4.3 Desire for control
text: To understand participants’ preferences for the amount of control they would like to exercise over the translation of their posts, we first examined users’ perspectives about the types of posts that they felt are and are not acceptable to translate. Most of the interviewees mentioned that mundane status updates about things like travel, food, birthdays, weddings, gatherings, and events were acceptable for translation, because these kinds of posts would likely not offend people even if they were slightly mistranslated[1]. For example:
Posts about how your day has been like just general status updates are fine for machine translation because it’s not going to offend people. I had this for breakfast or this for lunch or this for dinner— phrases like those would be harmless posts to try to use for machine translation. Or just special occasions like birthdays, weddings or just gatherings in general. Post about those would be perfectly fine for machine translation. – P32
In contrast, most interviewees argued that posts about sensitive and private information, posts about controversial topics such as politics, religion, or relationships, and posts with negative comments should never be translated to another languages without the author’s permission. They also mentioned the sensitivity of such topics differs by culture, so they thought it was risky to translate such posts due to the potential for misinterpretation.
Posts about sensitive topics that other people might find controversial, that would be a little bit more difficult to handle. Like, the current state of political polarization in the US, [...] having a machine to translate it, I think, would not be in my best interest. – P32
I don’t want my political views to be translated or my religious views, because I have a different religious view. – T78
These comments were corroborated by our survey study results. First, Fig. 5 shows that participants who are more (less) familiar with MT had a higher (lower) preference for their posts to be translated every time, but the correlation was very weak (r = 0.21, p = 0.003). This weaker correlation suggests that people who are more familiar with MT may not want all of their posts to be translated each time they write. Additionally, we found that users who write posts in languages other than English are less likely to want their posts to be translated (mean = 0.80) (t(18.45) = −3.4, p = 0.003) than those who do not (mean = 0.41).
When asking interviewees why they do not want posts written in their native language to be translated, most mentioned that they often choose a certain language as a means to select a target audience: they write in the language of those they intend to read the post. When MT translates such a post, it makes it accessible to an unintended audience [13]:
More than topic, it depends on target audience. For my mom who doesn’t understand English, I usually post in Nepali. [...] if you’re targeting a certain audience and you just want them to understand that post and no other audience. – T78
Most of my friends are Bangladeshi, so they will read it in Bangla. I post it thinking that they will see it. – T22
Language of the post is more related to the audience of the post, so if I’m writ-
ing something in English and it’s more about people in the United States then it concerns those people. Because the culture over here is different than in India, or in other countries. So if the social media translates my post, it might not be perceived as I would want in the other countries, so that’s a concern. – T56
A few interviewees used their native language to post about culturally specific controversial content or negative events in their homeland. When outsiders are unable to read such posts, they avoid exposing their country’s problems worldwide:
There may be some issues regarding my country, maybe some bad things happen. I want to keep the good image of my country. In my country, everyone knows about it so, I am okay to share with them, but not to international people. – T66
Our survey findings corroborate the finding that users write posts in their native language to target particular audiences. Particularly, participants who write posts in other languages tend to more frequently write their posts for particular audiences (mean = 0.91) (t(155.42) = −11.8, p < 0.05) than those who do not (mean = 0.26), and participants who use SNS more frequently tend to more frequently change the language of their posts depending on the topic (r = 0.21, p = 0.003; Fig. 6).
As the above comments and analyses demonstrate, participants were found to have many concerns regarding their posts being misinterpreted due to lack of MT accuracy or a lack of transparency in terms of who can read the translation
of their posts. This suggests that there is an urgent need for users to have control over the translation of their posts. To confirm this, we asked interviewees about what types of control they would prefer to exercise regarding the translation of their posts. In this light, most interview participants mentioned that they would use the following features:
Edit the translation: Most interview participants mentioned that they would like to have the option to edit a translation if they feel it does not accurately conveys the meaning of their post:
It would be cool if I could edit the translation. – T91
It’s auto-generated by the machine and I think it doesn’t convey the information totally [...] users should have control over it, I believe in that. – T44
Filter audiences based on the post’s language: Interviewees also mentioned that they would like to have the option to select a target audiences based on the language of their post:
I think it is useful to have a filter of only audience who speak Spanish because I could only want to speak to those people about scenario. – P18
I can see some posts that are talking about sensitive issues, in this case, an author would probably want to specifically inform people of this language. So not only would they eliminate the risk of things possibly being lost in translation, but they’d also to avoid people that speak another language. – P32
Conversely, a few participants mentioned that it may not be worth translating a post into a language that only few audience members understand:
Depends on the audience: If there’s a higher number of audiences in that language it is worth translating, otherwise it is not worth it. – P9

heading: 5 Discussion
text: In this paper, we investigated authors’ awareness of, and attitudes towards, the automatic translation of their SNS posts. Our findings show that people often write posts in languages other than English,
and many of these people are familiar with and have experience with MT—likely because many of them interact with people from different language backgrounds, so they regularly see translations of posts from their friends which were originally written in different languages [11, 12, 4].
Users who are more (less) familiar with MT are more (less) likely to post on social networking sites (Sec. 4.1), which was opposite to our expectations. Interview participants who were aware of MT’s capabilities explained that it made them more cautious about posting sensitive content that could be mistranslated and cause personal or professional harm. This heightened awareness lead to more selective posting, rather than decreased posting overall. In particular, such users are confident about MT translations in terms of actual meanings (which makes them tolerant towards the translation of posts about non-sensitive topics) but less confident in terms of connotative meaning [17] (making them more cognizant of potential risks of the MT of posts containing figurative speech or slang).
Users want to avoid translating posts about sensitive topics—several participants wanted to share such information only to audiences who understand the language of the post. This may be because the information is specific to a particular culture and traditions that cannot be understood without context, or because another language may simply not have words for certain concepts [13, 12].
Since MT currently is fully automated, users are not able to see, let alone control the translation of their own posts. As a result, most participants were concerned about the possibility of their posts being mistranslated and consequently misinterpreted by their audience, and many wished for features that give them more control over the translation of their posts. In contrast, a few of the interviewees mentioned that they feel more concerned about the translation of their posts—especially those writing in low-resourced languages. But even interviewees who wrote in high-resourced languages such as Spanish and Chinese mentioned that slang often gets mistranslated. This is because the quality of automated MT, both for low- and high-resourced languages, is still not 100% accurate. If the automated MT becomes more capable of accurately conveying the intended meaning of posts, users may have less desire to control their translations.
That said, both HCI theory and empirical findings show that users desire control over automated
systems, even when the systems are so good that they rarely require intervention. Having control over their posts’ translation provides them a greater sense of security and ownership and helps individuals to present themselves more effectively online. Thus, existing research corroborates the finding that users want to exercise control over machine translation, but also suggests that an intermediate level of control would be optimal to reduce cognitive load [16]. An appropriate solution would be to allow users to view the MT of their posts and to correct potential translation errors. This will provide users a greater sense of control without requiring them to write the translation manually.
This solution is not without limitations. For authors, the editing of posts would only be possible for output languages that they can read and write in. Furthermore, authors’ ability to edit the translations of their posts could result in ethical concerns, as this feature may be used to create language-specific misinformation. A solution to this problem is to present the reader with both the original machine translation and the user-edited version.
Alternative solutions include allowing users to disallow the automatic translation of posts on a case-by-case basis, or to filter the audience of a post based on its original language. This aligns with the fact that users’ language choices are intentional, and depend heavily on the topic of the post and the target audience [13]: many purposely write posts in a certain language as a means to target those who speak that language.
Overall, we see a substantial benefit in making SNS users aware of how their posts are translated and shared with others, and in allowing them to remove the translation if the post was meant to be targeted to a specific audience or correcting the translation to preserve the original meaning of the post. Together, these design solutions would ensure users that their information is conveyed accurately and only to their intended audiences.

heading: 6 Limitations and future work
text: To recruit participants who are more likely to write posts in other languages, our recruitment was restricted to bi/multilingual individuals residing in the US. Due to these criteria, most participants in both studies were Spanish speakers and writers, which is the second most common language spoken in the US. Since Spanish is a rich-resourced language, findings about the quality or accuracy
of MT may be different if we consider a more linguistically diverse participant sample. Future work could recruit global bi/multi-lingual participants to understand broader perceptions of machine translation.
Our discussion section advocates for a number of control features that are currently not available in SNS. Our results suggest that these features would make users more comfortable with MT, but this suggestion would ideally be confirmed with a controlled experiment. Our future research will be focused on designing the proposed controls. In particular, we plan to conduct an experiment on prototypes of three different translation features: one allowing the user to read but not edit the MT, one allowing the user to add a translation manually, and one allowing the user to read and edit the MT if there are any errors. Each prototype will also have an option to include/not include the translation with each post. This experiment will measure which features most improves users’ perceived control, perceived satisfaction, ease of use, and intention to use MT.

heading: 7 Conclusion
text: In this paper we offer insights into users’ awareness of and concerns regarding the translation of their social media posts. We consider improvements to the user experience of MT in terms of providing authors with more control over post translation. Findings from our studies demonstrate that SNS users find MT of posts with connotative meaning to be very poor, perhaps because the MT may fail to properly account for contextual information to accurately translate the intended meaning; users also express concern that MT may get the tone of the original post wrong. As a result, users were found to be concerned about how their posts are translated and subsequently interpreted by their audiences. This caused many users to be selective in which of their posts they would want to be translated: indeed, some posts were considered too sensitive to be translated, while at other times users intentionally used language as a means to limit the audience of their post. These findings stand in stark contrast with the current practice on most SNS, where users’ posts are translated indiscriminately and without users’ explicit permission. Thus, we call upon existing SNS to give their users more control over the translation of their social media posts.

heading: Future Work
text: Future work could recruit global bi/multi-lingual participants to understand broader perceptions of machine translation