Human Raters Cannot Distinguish English Translations from Original English Texts
Authors: Shira Wein
Venue: EMNLP 2023 Main (2023)
Keywords: translationese
human evaluation
translation

Abstract: The term translationese describes the set of linguistic features unique to translated texts, which appear regardless of translation quality. Though automatic classifiers designed to distinguish translated texts achieve high accuracy and prior work has identified common hallmarks of translationese, human accuracy of identifying translated text is understudied. In this work, we perform a human evaluation of English original/translated texts in order to explore raters' ability to classify texts as being original or translated English and the features that lead a rater to judge text as being translated. Ultimately, we find that, regardless of the annotators' native language or the source language of the text, annotators are unable to distinguish translations from original English texts and also have low agreement. Our results provide critical insight into work in translation studies and context for assessments of translationese classifiers.

Human Raters Cannot Distinguish English Translations from Original English Texts
Shira Wein Georgetown University sw1 158@georgetown.ccm edu
Abstract
tendencies associated with translations emerge large scale, even natural-sounding translations 2019).
The term translationese describes the set of linguistic features unique translated texts, which appear regardless of translation quality. Though automatic classifiers designed to distin- guish translated texts achieve high accuracy and prior work has identified common hall- marks of translationese, human accuracy of identifying translated text is understudied. In this work, we perform a human evaluation of English original/translated texts in order to ex- plore (1) raters' ability classify texts being original or translated English and (2) the fea- tures that lead a rater to judge text as being translated. Ultimately, we find that, regardless the annotators' native language or the source language of the text, annotators are unable to distinguish translations from original English texts and also have low agreement. Our results provide critical insight into work in translation studies and context for assessments of transla- tionese classifiers.
(Kunilovskaya and Lapshinova-Koltun this raises the question whether individual trans- lated passages exhibit discernible features which distinguish them from text originally written in the language.
prior study which assessed Finnish speakers' ability identify translated texts (Tirkkonen- Condit, 2002), in this work we perform (to our knowledge, the first) comprehensive study for English texts assessing whether human raters are able to distinguish human-produced English translations from original English texts. Specifi- cally, we ask 8 human raters (both native and non- native speakers of English) to judge 120 English passages being translated or originals. Our lected data spans genres. We consider research questions:
Building
RQ1
Can human raters reliably (above chance) make a binary judgment as to whether a sen- tence is originally written in or translated into English?
1 Introduction
Translationese is the set of features that are unique to translated texts, even in high-quality translations
RQ2 Do raters agree on translationese judgments (regardless of accuracy)?
(Kunilovskaya and Bealhinovv-Koltunnss 2019). Prior work translationese has characterized hall- marks of translated texts (Baker et al., 1993; Toury, 1980), for example when the translated text forms more to the grammatical structure of the source language than the target language due to source language "shining through" (Teich, 2003). is clearly important to understand how trans- lations actually differ from original texts in per- ception. Though automatic classifiers can identify translated versus original English texts (Baroni and Bernardini, 2006; Ilisci et al., 2010; Koppel and Or- dan, 2011; Popescu, 201 1; Rabinovich and Wintner, 2015) by detecting statistical patterns associated with features translationese (section 2), work examining how humans distinguish translations is critically understudied. Because these statistical
RQ3 Are native speakers able make this judg- ment more accurately than nonnative speak- ers?
RQ4 Does source language affect rater accuracy? RQ5 Does genre affect rater accuracy?
We perform empirical analyses to examine the re- lationship between each variable (speakers' native language, source language of the text, and genre) and human raters' ability to distinguish English translations from original English texts, thus indi- cating how each variable affects translation percep- tion itself.
Where mean accuracy is denoted as Pa and M 0.5, we define our null hypothesis Ho as fo < M as we portend that human raters will not be able

---

distinguish English translations from original English texts at a rate above chance. The alternate hypothesis H1 is thus defined as 14a > M.
chance. Tirkkonen-Cowe (2002) also found that "unique" items such as collocations, idioms, and language-specifie features- are often an accurate indicator of an original Finnish text.
We expect that this work will provide much- needed foundational insight into translation studies and may also be useful for engineers building trans- lationese classifiers
In our work, we expand on Tirkkonen-Condit' (2002) study in four notable ways. First, we per- form the first human evaluation of translationese for English, shining light on whether humans' low accuracy distinguishing original/translated texts holds beyond Finnish. Second, while Tirkkonen- Condit (2002) seems to have only collected judg- ments from native speakers of Finnish, we collect annotations from both native and nonnative speak- ers of English. Third, we explore whether genre affects rater accuracy. Finally, we incorporate a statistical rigor into our study by addressing each research question with empirical analyses.
2 Related Work
The term translationese describes the set of fea- tures commonly found in translated texts that do not appear in text originally written in that language (Gellerstam, I 1986), including interference from the source language or over-normalizing to the tar- get language. Features of translationese include simplified language in the translated text (Blum and Levenston, 1978), overuse of cohesive mark- ers and overspecification of implied information in the source text per the explicitation hypothesis (Blum-Kulka 1986). Human and machine trans- lations exhibit different characteristics of transla- tionese (Bizzoni et al., 2020). Language-specific differences have been observed between original and translated texts in Chinese (Xiao, 2010) and Spanish (Ramón, 2015), among other languages. From an engineering perspective, the presence of translationese also has a negative effect on sys- tem performance and evaluation. Translationese in test sets can lead to inflated and inaccurate eval- uation scores (Graham et al., 2020) and when in- corporated into training data, the performance of the model may also be affected (Ni al., 2022; Yu et al., 2022). Recent work has set out to miti- gate the amount of translationese in embeddings (Dutta Chowdhury et al., 2022) and in text itself (Wein and Schneider, 2023).
3 Methodology & Experimental Design
In order to understand how translations differ from original texts and human perception of those differ- ences, we ask fluent speakers of English to judge passages as being either originally English or trans- lated into English.
Data. We sample data from Rabinovich et al. (2018) and Tolochinsky et al. (2018), which contain mix of human translated and original English sen- tences as well as metadata marking each sentence as translated/orrginal. The data consists of Euro- pean Parliament proceedings, Canadian Hansard (parliamentary proceedings), literature, political commentary, TED talks (Rabinovich et al., 2018), and United Nations proceedings (Tolochinsky et al., 2018). Before sampling from these datasets, we manually filtered out passages which clearly re- vealed the source language or nationality of the speaker (such as "I would therefore once more ask you ensure that get Dutch channel well," " and "the Government of the Russian Federation finds it necessary [...1")).
The most related work to ours is Tirkkonen- Condit (2002), translationese study which per- formed two pilots asking human raters to judge Finnish texts as being translated or original (a bi- nary judgment). The first pilot included 27 annota- tors and used 20 original Finnish passages and 20 translated Finnish passages, from various domains/ genres. Extra-linguistic identifiers such as named entities were filtered out as to not hint to whether the text translated not. In the second pilot, 74 teachers of Finnish foreign language judged 6 of the texts from the first pilot. The study found that the accuracy of humans to distinguish trans- lated or original Finnish texts was 61.5% for pilot one and 63.1 % for pilot two, only slightly above
From these datasets, we randomly select a uni- form sample of 60 passages originally in English and 10 translated passages originally written in each of the following 6 languages: Arabic, Chinese (Mandarin). French, German, Russian, and Span- ish. This results in an even split between original and translated English passages (120 passages total, 60 translated and 60 originally English). Following Brrkkonen-Condit (2002), which used passages of length 100 300 tokens, we collect passages be- tween 90 and 300 tokens (adapted slightly to fit

---

Group
Nonnative
Native
-0.1933 to 0.2236, which is very low agreement for even the most similar pair of annotators. There- fore, no, annotators do not agree on translationese judgments. Interestingly, even passages with high agreement tend to be inaccurate. Of the 6 passages which receive 6 or 7 judgments of T (none received all 8 Ts), only 3 were in fact translations. Of the 9 passages for which all raters gave judgment of O, only 3 of the passages were actually originally English. 11 passages received 7/8 O judgments and of those 11 were indeed originally English.
Mean
0.4875
0.5063
SD
0.0812
0.0463
SEM
0.0406
0.0232
Table I: Mean, standard deviation (SD), and standard error of mean (SEM) of accuracies for the native and nonnative speaker groups.
natural cutoff points off our data).
Human Evaluation. In order to investigate whether translations are more identifiable by native or nonnative speakers, we collect (binary) judg- ments from four native and four nonnative speakers graduate students
RQ3 Are native speakers able to make this judg- ment more accurately than nonnative speakers? No, native speakers are not able to more accurately distinguish translated/original English texts than nonnative speakers. Again using annotator accura- cies, we perform a two-sample t-test, which results in two-tailed p- p-value- - 0.7023, indicating that the difference is not statistically significant. The mean, standard deviation, and standard error of means for both groups of speakers can be seen in Table 1.
of English. All annotators fluent in English with background in linguistics. The annotators have not received special training in the specific task of translation detection prior to completing the task.
The instructions presented to the annotators can be seen in Figure 1. The annotators mark each passage being O (originally English) T (trans- lated into English) and are able to provide addi- tional comments justifying their annotations (dis- cussed in Section 5).
Does source language affect rater accu- racy? First, for each of the 7 original languages of the text (Arabic, German, English, Spanish, French, Russian, and Mandarin), we calculate accuracies for each annotator (seen in Table 2) and perform a one-way way ANOVA. The p-value for the ANOVA equals 0.0130, indicating that at least one treatment is statistically different. Examining the results of Tukey Honest Significant Difference (HSD) test (Gleason, 1999), we find that the English sample is statistically significantly different from the Arabic, German, and Russian samples.
RQ4
4 Results & Discussion
RQ1 Can human raters reliably (above chance) make binary judgment as to whether a sentence originally or translated English? We calculate ac- curacy for each annotator and with these accuracies we perform a one-sample t-test. In order to deter- mine whether annotators are able to perform above chance (> 0.5), we set the hypothetical mean to 0.5 and test whether the two-tailed p-value indicates a statistical significance between the hypothetical mean and the actual mean of accuracies. The two- tailed p-value= 0.8907, which is not a statistically significant p-value, so we fail to reject the null hy- pothesis Ho: Ha < M where M = 0.5. The actual mean of accuracies for the 8 annotators is 0.4969, with standard deviation of 0.0620. Thus, no, hu- man raters are not able to perform above chance when distinguishing original/taanllted English.
Moving beyond specific language pairs, now we consider whether annotator accuracy is statis- tically different between the non-English source languages, as well as whether there is statistical difference in annotator accuracy between English texts and all non- non-English texts (grouped together). When removing English and performing a one- way ANOVA on only the 6 non-English languages, the p-value= 0.8063, indicating that there is no statistical difference between the 6 non-English source languages. We then measure the statistical difference between English and all non- non-English lan- guages with two-sample t-test, and find that the two-tailed p-value- 0.0008, revealing that there is statistically significant difference in annotator accuracy for English and non-English. The mean English accuracy was 0.6458 (standard deviation
Do raters agree on translationese judg- ments (regardless of accuracy)? In order to assess annotators' agreement on translationese judgments. we calculate inter-annotator agreement via Cohen's Kappa. The inter-annotator agreement across all 28 calculations (every combination of the annota- tors) produces& a mean of 0.0706 and ranges from
RQ2

---

Thank you for taking part in this human evaluation. In the Survey tab, you will find 120 human--poddcce
English passages, some of which were originally uttered English, and some of which were originally uttered non-English language and then translated (by a human) into English. The question we would like you to answer is:
Does this sentence sound like it was originally written in English translated from another language into English?
There are two options: (1) Originally English, i.e. sentence was originally English, or (2) Translated into Englishh,
sentence was originally uttered non-English language and then human translated into English. For each sentence, pick either (1) or (2) by putting an O or T to indicate Originally English and for Translated into English.
For example, you think sentence example was produced originally in English. this is how you would mark the Survey tab:
O: Originally English
Translated into English
Sentence
This example sentence A.
If you have any additional comments thoughts sentence (e.g. there
is a specific feature of the sentence you would like to point out), you are able to optionally enter any comments into column of the Survey Tab, marked [Optional] Comments.
Figure I: The instructions presented to human annotators prior to classifying the passages as translated or originally English.
Language
AR
DE
EN
ES
FR
RU
ZH
Anno I
0.8000
0.7000
0.6833
0.4000
0.3000
0.5000
0,4000
Anno 2
0.3000
0.6000
0.5000
0.4000
0.6000
0.3000
0.1000
Anno
0.2000
0.300
0.7000
0.3000
0.3000
0.1000
Anno
0.2000
0,2000
0.5167
0.5000
0.5000
0.3000
0.6000
Anno 5
0.6000
0,2000
0.6333
0.7000
0.,000
0.5000
0.4000
Anno
0.1000
0,3000
0.5000
0,5000
0.4000
0,4000
0.3000
Anno 0.3000
0.1000
0.7167
0,5000
0.4000
0.2000
0,5000
Anno
0.9167
0,3000
Table 2: Mean accuracy for each annotator for English (EN) and for each of the 6 source languages translated into English: Arabic (AR), German (DE), Spanish (ES), French (FR), Russian (RU), and Chinese (ZH).
of 0.1455) while the mean non-English accuracy was 0.3354 (standard deviation of 0. 1508). This demonstrates that annotators are able to more ac- curately identify originally English passage as being such, though this is likely due the fact that the raters' judgments were imbalanced and marked more passages as originals than translations (an average of 78/ 120 passages across annotators were judged as originals). In particular, annotator 8 judged most of the passages to be originals, leading to high accuracy for the English passages, and very low accuracy for the non non-English passages (again seen in Table 2).
HSD, we find that there is a statistically significant difference between the following two treatments: Literature and Political Commentary, and Politi- cal Commentary and United Nations proceedings. This is indicative of annotators having more dif- ficulty judging literary texts and United Nations commentary than the political commentary (see ac- curacies in Table 3). The literary texts are mostly from the 1800s and are artistic in nature, making them more challenging to parse translations originally English. The literary texts are mostly originally English, source language (re: RQ4) likely does not serve as confounding variable here. Similarly, the United Nations proceedings are trans- lations speech and contain more complex topics. On the other hand, the political commentary genre is the most well-formed and structured in nature of the genres considered, which may be why it was
RQ5 Does genre affect rater accuracy? Yes. when performing one-way ANOVA on the mean accuracies for each annotator of each genre, the p-value= 0.0272, showing that there is a statistical difference in mean accuracy by genre. Per Tukey's

---

Genre
Europarl
Hansard
Literature
Politics
TED
UN
Anno
0.6154
0.7500
0.3750
0.7857
0.6667
0.5660
Anno
0.6154
0.2500
0.3750
0.5714
0.4167
0.4340
Anno
0.1000
0.6667
0,3750
0.7143
0.5833
0.3019
Anno 4
0.5385
0.4167
0,4375
0.5714
0.5833
0.4717
Anno
0.7692
0.2500
0,6250
0.7143
0.6667
0.4906
Anno 0,5385
0,5000
0,2500
0.6429
0,5000
0.3396
Anno 0.5385
0.5000
0,5625
0.6429
0.4167
0.5094
Anno
0,3846
0.7500
0,5000
0.6429
0,5833
0.3774
Table 3: Mean accuracy for each annotator for each of the 6 genres ("politics" refers to political commentary).
the casiest to distinguish translated or originally English.
that even passages with high agreement were often inaccurately annotated. With respect to genre, for- mal written texts are able to distinguished more accurately than speech or literary works.
5 Discussion of Annotator Comments
Though existing translationese classifiers are able to separate translated and original texts with high accuracy (>90%) (Pylypenko al., 2021), human raters are not able do so on the same genres. This is likely because classifiers are pick- ing up on more subtle cues that can measured statistically but are not easily visible to human an- notators; this is validated by Amponsah- Kaaaahr et al. (2022), which demonstrates that cand-crafted features of translationese are only small portion of what BERT-based translationese classifiers learn, and these hand-crafted features are even subsumed under what BERT learns without features. Our results further indicate that there are underlying properties texts that separate translations from originals which are not easily visible in the surface form to human annotators and that, in line with prior work, well-known features of translations do not fully encompass "translationese" on statistical scale.
Annotators generally described the task as challeng- ing and noted particular difficulty with passages containing technical or "bureaueratic" language. Annotators commented on "clues" which led them to their judgments, such as framing/content ("comparing modern geopolitics to ancient Greece seems like an American/Britiss thing to do") and date formatting. Linguistic features such as idioms (e.g. "its credibility is shor") and metaphor- ical language (c.g. "picture am painting") led annotators to mark passage as originally English. Appropriate use of infrequent prepositions, such as "amidstt, also caused annotators to believe the passage was originally composed in English. On the other hand, annotators cited surprising preposi- tion choice (e.g. "following the earlier attacks") and the use of excessive modifiers indicators of text being a translation. Unnatural sentence struc- ture, unnatural sounding language (e.g. "fund of information"), under/overspecification, and repe- tition were all also noted in comments as reasons to annotate a passage as a translation. Intriguingly, though the annotators achieved low accuracy, most of the reasons cited as being reasons to judge text translation are established hallmarks of translationese (Volansky al., 2013).
Limitations
While our empirical/staaisticaal and qualitative anal- yses provide novel insight into human perceptions' of English translations, engineering implications of our findings are largely left for future work, though do draw comparisons to translationese classi- fiers in Section 6. We are limited in our study by the availability of publicly available corpora which have both translated and original English passages in the same context, thus constraining the types of genres and source languages we are able to pull from for our study.
6 Conclusion
In this study we assessed human raters' ability to identify English translations as distinct from origi- nal English texts and found that raters are not able to accurately make this distinction agree with other annotators. Our statistical analyses revealed that, interestingly, human raters' (in)ability to dis- tinguish English translations from original English text persists regardless the speakers' native lan- guage or the source language of the text. We found
Acknowledgements
Thanks to anonymous reviewers for their use- feedback and to Rotem Dror for her guid-

---

ance on significance testing. Thanks also to Tat- suya Aoyama, Aryaman Arora, Shabnam Behzad, Michael Kranzlein, Lauren Levine, Jessica Lin, Yang Liu, and Wesley Scivetti. This work is sup- ported by a Clare Boothe Luce Scholarship.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72-81, Online. Association for Computational Linguisties.
Justina Ilisei, Diana Inkpen, Gloria Corpas Pastor, and Ruslan Mitkov. 2010. Identification of translationese: A machine learning approach. In Computational Lin- guisties and Intelligent Text Processing: 11th Inter- national Conference, CICLing 2010, layi, Romania, March 21-27, 2010. Proceedings 11, pages 503-511. Springer.
References
Kwahena Saaponsah-Kaaayiree Daria Pylypenko, Josef' Genabith, and Cristina España-Bonet, 2022. Explain- ing translationese: why are neural classifiers better and what do they learn? In Proceedings of the Fijhh BlackboxNLP Workshop on Analyzing and Interpret- ing Neural Networks for NLP, pages 281-296, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1318-1326, Portland, Oregon, USA. Association for Computational Linguistics.
Mona Baker, Gill Francis, and Elena Tognini-Bonelli 1993, Corpus linguistics and translation studies: Im- plications and applications, chapter 2.
Maria Kunilovskaya and Ekaterina Lapshinova- Koltunski. 2019. Translationese features as indi- cators of quality in English- -usiisnn human trans- lation. In Proceedings of the Irfore-Irformed Trans- lation and Interpreting Technology Workshop (HiT-IT 2019), pages 47-56, Varna, Bulgaria. Incoma Ltd., Shoumen, Bulgaria.
Marco Baroni and Silvia Bernardini. 2006. A new approach to the study of translationese: Machine- learning the difference between original and trans- lated text. Literary and Linguistic Computing. 21(3):259-277 274.
Yuri Bizzoni. Tom S Juzek, Cristina España- Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Te- ich. 2020. How human is machine translationese? comparing human and machine translations of text and speech. In Proceedings of the 17th International Conference Spoken Language Translation, pages 280-290, Online. Association for Computational Lin- guistics.
Jingwei Ni, Zhijing Jin, Markus Freitag, Mrinmaya Sachan, and Bernhard Schölkopf. 2022. Original or translated? a causal analysis of the impact of trans- lationese on machine translation performance. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5303-5320, Seattle, United States. Association for Computational Linguistics.
Shoshana Blum and Eddie A Levenston. 1978. Uni- versals of Texical simplification. Language learning, 28(2):399-415
Marius Popescu. 2011. Studying translationese at the character level. In Proceedings of the International Conference Recent Advances in Natural Language Processing 201 I, pages 634-639, Hissar, Bulgaria. Association for Computational Linguistics.
Shoshana Blum- Kulka, 1986. Shifts of cohesion and coherence in translation. in interlingual and intercul- tural communication: Discourse and cognition in translation and second language acquisition studies, edited by Juliane House and Shoshana Kul-Kulka, 17-36. Tübiingen: Narr.
Daria Pylypenko, Kwabena Amponsah-Kaakyire Koel Dutta Chowdhury, Josef van Genabith, and Cristina España-Boned" 2021. Comparing feature-engineering and feature-learning approaches for multilingual translationese classification. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 3596-8611, 1, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Koel Dutta Chowdhury, Rricha Jalota, Cristina España- Bonet, and Josef Genabith. 2022. Towards debias- ing translation artifacts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compudational Linguistics: Human Language Technologies. pages 3983-3991, Seattle, United States. Association for Computational Lin- guistics.
Ella Rabinovich and Shuly Wintner. 2015. Unsuper- vised identification of translationese, Transactions of the Association for Computational Linguistics, 3:419- 432.
Martin Gellerstam. 1986. Translationese in Swedish novels translated from English. Translation studies in Scandinavia, 95.
Ella Rabinovich, Shuly Wintner, and Ofek Luis Lewin- sohn. 2018. A parallel corpus of translationese. In Computational Linguistics and Intelligent Text Pro- cessing: 17th International Conference, CICLing
John R Gleason. 1999. An accurate, non-iterative ap- proximation for studentized range quantiles. Compu- lational statistics & data analysis, 3772):147-158.

---

2016, Konya, Turkey, April 3-9, 2016, Revised Se- lected Papers, Part 1I 17, pages 140-155. Springer.
Noelia Ramón. 2015. Comparing original and trans- lated spanish: A corpus-based analysis of adjective position. Babel, (42:55527
Elke Teich. 2003. Cross- Linga-Lin Variation Sys- tem and Text: A Methodology for the Investigation of Translations and Comparable Texts. De Gruyter Mouton, Berlin, Boston.
Sonja Pirkoneen-Cndife 2002. Translationese a myth or an empirical fact?: A study into the linguistic identifiability of translated language. Target. Interna- tional Journal of Translation Studies, 14(2):207-220.
Elad Tolochinsky, Ohad Mosafi, Ella Rabinovich, and Shuly Wintner. 2018. The UN parallel cor- pus annotated for translation direction. CoRR, abs/1805.07697.
Gideon Toury. 1980. In search of a theory of translation. Porter Institute for Poetics and Semiotics, Tel Aviv University.
Vered Volansky, Noam Ordan, and Shuly Wintner: 2013. On the features of translationese. Digital Scholarship in the Humanities, 30(1))98-118.
Shira Wein and Nathan Schneider. 2023. Translationese reduction using abstract meaning representation.
Richard Xiao. 2010. How different translated chinese from native chinese?: A corpus-hased study of trans- lation universals. International Journal of Corpus Linguistics, 15(1):5-35
Sicheng Yu, Qianru Sun, Hao Zhang, and Jing Jiang. 2022. Translate-train embracing translationese arti- facts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers), pages 362-370, Dublin, Ire- land. Association for Computational Linguistics.


Reviews:
[{"review_id": "ed5lhMxvX8", "reviewer": "Reviewer_p1Sh", "summary": "", "questions": "- I feel like somewhere earlier in the paper you need to make it clear that the translations are human-generated translation\n- For this study and dataset, exploring how a classifier baseline's performance compares to that of human raters could provide insightful information\n- Your discussion section ended with: “though the annotators achieved low accuracy, most of the reasons cited as being reasons to judge a text as a translation are established hallmarks of translationese.” \ndoes not hold the same significance? Or the genre of the text might play a role here; EuroParl and Hansard texts appear more protocol-based compared to open-domain texts.\n\n\n\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We will respond to your questions and comments.\n\nRegarding your comment “The paper lacks a discussion on the motivation behind the research and what would be some practical impact of the findings” as we mention in the paper, it is important to understand how translations actually differ from original texts in perception. If we want to make progress in translation studies, as well as in effective human evaluation of translation (such that machine translation models produce outputs most preferred by humans), understanding how humans perceive translations is critical and yet notably underexplored. In the final version, given more space to flesh out the motivation, we will do so. \n\nWe will consider comparing against a classifier and/or expanding to AI-generated text in future work. Existing literature has already demonstrated that, at the statistical level (across a whole training corpus), classifier models are able to distinguish translated and original texts at a high accuracy level. Instead, in this work, we investigate whether fluent English speakers are able to classify individual passages as translated or not, and empirically investigate five research questions related to this. Extending to AI-generated text is also an interesting idea for future work, again separate from our main goals with this work. If these suggestions are impacting your review, we ask you to reconsider as these are separate research questions.\n\nThank you for the suggestion to clarify earlier that the translations are human-generated; we will add this to the Data paragraph in Section 3 in the final version.\n\n“Your discussion section ended with: “though the annotators achieved low accuracy, most of the reasons cited as being reasons to judge a text as a translation are established hallmarks of translationese.” does not hold the same significance?” We are not sure what you mean here so unfortunately we are not able to respond to this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It focuses on the theoretical aspects of translation and provides valuable insights into the field of translation studies.\n\nThese insights can contribute to the establishment of more equitable and robust metrics for assessing both manual and automated translations.\n\n", "reasons_to_reject": "The paper lacks a discussion on the motivation behind the research and what would be some practical impact of the findings.  For example:\n- Why did we anticipate differences between two human-generated pieces of text (translation and original text) in the first place? \n- What aspects of these results were surprising? \n- How might these results reshape our perspective on machine translation research? \n- Following the same analogy, It would be insightful to see how this study could extend to AI-generated text. For instance, does text generated in response to Q/A or paraphrasing tasks differ from text produced in response to translation prompts? Can a classifier or human rater distinguish between them?\n", "questions_to_authors": "- I feel like somewhere earlier in the paper you need to make it clear that the translations are human-generated translation\n- For this study and dataset, exploring how a classifier baseline's performance compares to that of human raters could provide insightful information\n- Your discussion section ended with: “though the annotators achieved low accuracy, most of the reasons cited as being reasons to judge a text as a translation are established hallmarks of translationese.” \ndoes not hold the same significance? Or the genre of the text might play a role here; EuroParl and Hansard texts appear more protocol-based compared to open-domain texts.\n\n\n\n\n\n", "paper_topic_and_main_contributions": "This work studies humans' ability to differentiate between text originally written in English and translated text. The findings demonstrate that regardless of the raters' native language or the source language, they are unable to make such distinctions effectively.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xiNrqfQIsT", "reviewer": "Reviewer_xarF", "summary": "", "questions": "* Table 2 does seem to point to \"outlier\" annotators, i.e. people that seem to perform very well, e.g. Anno 1 in AR and DE, or Anno 8 in EN -- any explanations / findings around that which could be mentioned?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hello, thank you for your thoughtful review and comments. Regarding the note that \"corpora used could have been attempted to separate between both translated and original English sentences in the first place:” we agree, these distinctions certainly exist in a few corpora. We used two of these datasets for our annotation in this work. However, our research question is whether humans are able to distinguish between translated and original English sentences. \n\nRegarding “not much detail provided who the annotators are and how well they have (not) been trained for the task -- and whether that would make a difference at all?” Thank you for noting this, in the final version we will add more information on this. The annotators are all fluent English speakers and are all graduate students with some linguistic knowledge, making them even better suited to distinguish original and translated English than the average English speaker. As noted in the paper, of the 8 annotators, 4 are native and 4 are nonnative English speakers.\n\n“Table 2 does seem to point to \"outlier\" annotators […] any explanations / findings around that which could be mentioned?” As we mention in the paper, annotators are able to more accurately identify an originally English passage as being such, though this is likely due to the fact that the rater's judgments were imbalanced and marked more sentences as originals than translations (average of 78 / 120 passages judged as originals).  This is especially true for Annotator 8, as you note— we will add this to the paper. For other Annotators who more accurately mark the translated passages, they tend to mark a higher portion of the passages as being translated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* very interesting and highly debated research question\n* corroborating the results nicely, with thorough experimentation\n* presented very well and clearly written\n* extendable and could even inform future classifiers in the detection of translated material", "reasons_to_reject": "* corpora used could have been attempted to separate between both translated and original English sentences in the first place -- there are corpora available like that, e.g. Europarl which gives indications whether a passage is native or not\n* not much detail provided who the annotators are and how well they have (not) been trained for the task -- and whether that would make a difference at all?", "questions_to_authors": "* Table 2 does seem to point to \"outlier\" annotators, i.e. people that seem to perform very well, e.g. Anno 1 in AR and DE, or Anno 8 in EN -- any explanations / findings around that which could be mentioned?", "paper_topic_and_main_contributions": "This paper discusses highly interesting findings in terms of human raters' ability to identify whether an English sentence has been natively written as such or has been translated from another language. The source sentences span several languages among which Arabic, German, English, Spanish, French, Russian, and Mandarin. \nThe paper concludes with rather low inter-annotator agreements therefore pointing to the fact that humans seem not to be good at actually identifying whether a sentence has been translated or not.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9kuC9RXZVk", "reviewer": "Reviewer_J1E6", "summary": "", "questions": "RQ1 – did you apply any constraints on the sentence length for sentences subject to annotation? Extremely short sentences could be impossible for this task. I would even suggest to go beyond the sentence level – short paragraphs would constitute a more reliable setup.\n\nlines 140-146: was filtering done manually or automatically? If automatically, it would be useful to provide more details on the procedure.\n\nWhat were the guidelines provided to the annotators? Were they presented with any indicators or clues they should pay attention to? Were they familiar with the concept of translationese?\n\nRegarding the clues that helped the annotators conclude about a sentence being original or translated: I do not find framing or content (line 277), or date formatting (line 280) particularly interesting, since these are not markers of translationese but rather cultural and topical indicators.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. First we will answer your questions and then respond to your comments.\n\nRegarding RQ1: we do not perform annotations at the sentence level, and instead annotate short paragraphs. As we mention in the paper (line 156), following Tirkonnen-Condit (2002), which used passages of length 100 - 300 tokens, we collect passages between 90 and 300 tokens (adapted slightly to fit natural cutoff points off our data).\n\n“was filtering done manually or automatically?” Filtering was performed manually; we will clarify this in the final version.\nRegarding your question “What were the guidelines provided to the annotators?” As we note in Section 3 (line 161), we have provided the specific instructions and guidelines provided to annotators in Appendix A. \n\nFor your comment “Regarding the clues that helped the annotators conclude about a sentence being original or translated: I do not find framing or content (line 277), or date formatting (line 280) particularly interesting, since these are not markers of translationese but rather cultural and topical indicators.” We agree, these are less exciting markers of translationese, though still important to note, as they were reasons cited for annotators’ decisions and they do play a role in human perception of text being translated or original. Other cited reasons such as preposition (line 286) lead to more open questions.\n\nWe appreciate the suggestion for comparing human performance against automatic classifiers, though the question we are investigating here is solely in regard to human capability. Existing literature has already demonstrated that, at the statistical level (across a whole training corpus), classifier models are able to distinguish translated and original texts at a high accuracy level. Instead, in this work, we investigate whether fluent English speakers are able to classify individual passages as translated or not, and empirically investigate five research questions related to this. \n\nRegarding your comment “The authors say that “work examining how humans distinguish translation is critically understudied”, but it is still not clear why that’s an important question,” as we mention in the paper, it is important to understand how translations actually differ from original texts in perception. If we want to make progress in translation studies, as well as in effective human evaluation of translation, understanding how humans perceive translations is critical and yet notably underexplored. In the final version, given more space to flesh out the motivation, we will do so.\n\nThank you for your reference suggestions, we will incorporate those into the final version.\n\nWe hope this resolves anything you have missed or wondered about during your initial read of our paper and that this enables you to re-assess your review, in particular with regard to soundness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I find the work is quite interesting, generally clear and well-written.\n\nAll experiments are framed in a sound statistical manner, and supported by significance tests.\n", "reasons_to_reject": "The contribution, novelty, technical depth and potential impact of this work are somewhat lightweight, even for a short paper IMO. One possible way to enhance the work is to perform a range of experiments with automatic classifiers on the same set of datasets and examine the patterns highlighted by classifiers vs. those the annotators could capture.\n\nThe research questions presented in the paper are not sufficiently motivated. The authors say that “work examining how humans distinguish translation is critically understudied”, but it is still not clear why that’s an important question: can you tie it to some cognitive aspects of language perception?\n", "questions_to_authors": "RQ1 – did you apply any constraints on the sentence length for sentences subject to annotation? Extremely short sentences could be impossible for this task. I would even suggest to go beyond the sentence level – short paragraphs would constitute a more reliable setup.\n\nlines 140-146: was filtering done manually or automatically? If automatically, it would be useful to provide more details on the procedure.\n\nWhat were the guidelines provided to the annotators? Were they presented with any indicators or clues they should pay attention to? Were they familiar with the concept of translationese?\n\nRegarding the clues that helped the annotators conclude about a sentence being original or translated: I do not find framing or content (line 277), or date formatting (line 280) particularly interesting, since these are not markers of translationese but rather cultural and topical indicators.\n", "paper_topic_and_main_contributions": "This short paper studies the question whether human annotators can reliably distinguish between text originally written in some language (English) or translated from other language to English (translationese). The authors put forward four research questions and show that the task is very difficult for humans, despite the considerable degree of success on the same objective with automatic classifiers.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 52: originally  originally written in English?\n"}, "extra_text": {"Missing_References": "The first paragraph in introduction mentions a well-established evidence on translated texts, but the citation choice is somewhat weak; there are plenty of seminal studies on the differences between original and translated texts, to name a few: GideonToury, 1980. In Search of a Theory of Translation; William Frawley, 1984. Prolegomenon to a theory of translation; Mona Baker, 1993. Corpus linguistics and translation studies: Implications and applications; Martin Gellerstam, 1986. Translationese in Swedish novels translated from English.\n"}}]

Author rebuttal:

Meta-review: This work evaluates human abilities to identified original & translated texts.

All three reviewers agree that the soundness of the proposal is either good or strong, while there is more variability with respect to excitement (varying from 2-4).  The three reviewers have raised interesting points of strength and weakness during the review period, to which the authors have provided clear answers during the rebuttal period. Although there is some disagreement between the reviewers with respect to the issues of criticism, they are compatible and enriching. 

Should the manuscript be accepted, I recommend the authors to incorporate the suggestions made by reviewers, including all the details possible on the methodology (e.g. annotators, etc.), a detailed motivation and context/scope of the work, the missing references and typos. The suggestion to carry out additional experiments is interesting for future research but, in my opinion, should not preclude publication.
Decision: Accept-Main