Translating away Translationese without Parallel Data
Authors: Rricha Jalota
Koel Dutta Chowdhury
Cristina España-Bonet
Josef van Genabith
Venue: EMNLP 2023 Main (2023)
Keywords: Translationese Mitigation
Text Style Transfer
Unsupervised Training
Bias Mitigation
TL;DR: This work focuses on reducing translationese in translated texts, aiming to bring them closer in style to the originally-authored ones. We introduce a translation-based style transfer approach that does not rely on the availability of parallel data.

Abstract: Translated texts exhibit systematic linguistic differences compared to original texts in the same language, and these differences are referred to as translationese. Translationese has effects on various cross-lingual natural language processing tasks, potentially leading to biased results. In this paper, we explore a novel approach to reduce translationese in translated texts: translation-based style transfer. As there are no parallel human-translated and original data in the same language, we use a self-supervised approach that can learn from comparable (rather than parallel) mono-lingual original and translated data. However, even this self-supervised approach requires some parallel data for validation. We show how we can eliminate the need for parallel validation data by combining the self-supervised loss with an unsupervised loss. This unsupervised loss leverages the original language model loss over the style-transferred output and a semantic similarity loss between the input and style-transferred output. We evaluate our approach in terms of original vs. translationese binary classification in addition to measuring content preservation and target-style fluency. The results show that our approach is able to reduce translationese classifier accuracy to a level of a random classifier after style transfer while adequately preserving the content and fluency in the target original style.

Translating away Translationese without Parallel Data
Rricha Jalota Koel Dutta Chowdhury',
Cristina España-Bonet', and Josef van Genabith 1.2
Saarland University, Saarland Informatics Campus, Germany
Gerrann Research Center for Artificial Intelligence (DFKI)
AppTTek GmbH, Germany
rjalota@apptek, com, rr @ud koel. hetp..uuttachowuuuuuun----se- -atoowduurreuui-sraaaaddddd.. - de
de
{cristinae, Josef Van_Genabi thwedhhi.....
Abstract
direction of parallel training data matches the di- rection of the translation task, (i.e. when the source is original and the target is translated), MT systems perform better (Kurokawa et al., 2009; Lembersky et al., 2012). Similarly, Toral et al. (2018), Zhang and Toral (2019) and Graham et al. (2020) show that translating already translated texts results in increased BLEU scores. More recently, Artetxe et al. (2020a) observed that cross-lingual models when evaluated on translated test sets show false improvements simply due to induced translation artifacts. When investigating the effects of trans- lationese in cross-lingual summarization, Wang et al. (2021) found that models trained on trans- lated training data suffer in real-world scenarios. These examples show the importance of investigat- ing and mitigating translationese. Despite this, moving translationese signals already generated output of translations is an underexplored research topic. Dutta Chowdhury et al. (2022) remove trans- lationese implicitly encoded in vector embeddings, and demonstrate the impact of eliminating transla- tionese signals on natural language inference per- formance. Wein and Schneider (2023) leverage Abstract Meaning Representation (AMR) in- termediate representation to abstract away from surface-level features of the text, thereby reducing translationese. Neither of these works explicitly analyzes surface forms of the resulting "debiased text" and its resemblance to original texts.
Translated texts exhibit systematic linguistic differences compared to original texts in the same language, and these differences are re- ferred to as translationese. Translationese has effects on various ross-lingual natural lan- guage processing tasks, potentially leading to biased results. In this paper. we explore a novel approach to reduce translationese trans- lated texts: translation-bast based style transfer. As there are no parallel numan-translated and orig- inal data in the same language, use self- supervised approach that can learn from compa- rable (rather than parallel) mono-lingual orig- inal and translated data. However, even this self-supervised approach requires some paral- lel data for validation. We show how can eliminate the need for parallel validation data by combining the sufersuperi loss with an unsupervised loss. This unsupervised loss lever- ages the original language model loss over the style-transferred output and a semantic similar- ity loss between the imput and stle-transferree output. We evaluate our approach in terms of original vs. translationese binary classification in addition to measuring content preservation and styge fluency. The results show that our approach is able to reduce translationese classifier accuracy to level of random clas- sifier after style transfer while adequately pre- serving the content and fluency in the target original style.
1 Introduction
this work, we aim to reduce the presence of translationese in human-translated texts and make them closely resemble original texts, notably, with- out using any parallel data as parallel human orig- inal translated data in the same language do not exist. To this end, explore self-supervised neural machine translation (NMT) system (Ruiter et al., 2019) and its application style transfer (ST) (Ruiter al., 2022). In both works, validation is performed on a parallel dataset, either bilingual (MT data) or monolingual (ST data). However,
Translated texts often exhibit distinct linguistic features compared to original texts in the same language, resulting in what is known as transla- tionese. Translationese has a tangible impact on various cross cross-lingual and multilingual natural lan- guage processing (NLP) tasks, potentially leading to biased results. For instance in machine trans- lation (MT), during training, when the translation
Worr done while the author was student at Saarland University.

---

parallel human iiiginal-translationn data in the same language are unavailable. To overcome this challenge, we define an unsupervised validation criterion by combining a language model loss and a semantic similarity loss, inspired by Artetxe et al. (2020b). Our baseline is the self-supervised ap- proach (SSNMT) from Ruiter et al. (2019). How- ever, we go step further and propose a novel joint training objective that combines both self- supervised and unsupervised criteria, eliminating the need for parallel data during both training and validation.
Translation (Lample al., 2018a; Artetxe et al.. 2019). Common approaches involve disentangling the style and content aspects of the text. For con- tent extraction, approaches based on variational auto-encoders (VAE) (Shen et al., 2017; Fu et al., 2017), cycle consistency loss (Lample et al., 2019), or reinforcement learning (Xu et al., 2018) are com- monly employed. To induce the target style, often style discriminator is employed using a pretrained style classifier (Prabhumoye al., 2018; 8; dos San- tos al., 2018; Gong et al., 2019) or the decoder head is specialized to generate target-style outputs (Tokpo and Calders, 2022), or the content represen- tation is simply concatenated with the target-style representation (Fu et 2017).
The contributions of this work are as follows:
We are the first to formulate reduction of trans- lationese in human translations monolin- gual translation based style-transfer task, al- lowing for direct evaluation of the effects on the surface forms of the generated outputs.
Since unsupervised methods often perform poorly compared to their supervised counterparts (Kim et al., 2020; Artetxe et al., 2020b), recent ap- proaches have explored semi-supervised (Jin et al., 2019) and self-supervised learning (Ruiter et al., 2022; Liu et al., 2022). Liu et al. (2022) com- bine sentence embeddings with scene graphs mine parallel sentences on-the-fly for facilitating reinforcement learning-based style-transfer, while Ruiter et al. (2022) follow a simpler approach that exploits only the latent representations for online parallel sentence pair extraction from comparable data and leverage these pairs for self-supervised learning. Although their approach requires paral- lel validation set for model selection and hyperpa- rameter tuning, due its simplicity, we adopt it our starting point and baseline. We then present a novel version of this approach using unsupervised techniques, eliminating the need for a parallel vali- dation set.
We introduce joint self-supervised and un- supervised learning criterion that eliminates the need for a parallel (and non-existent) orig- inal translated dataset (in the same language) during training and validation.
We show that our method able to reduce the accuracy of translationse classifier to that of a random classifier, indicating that our ap- proach is able to successfully eliminate trans- lationese signals in its output.
We present an extensive evaluation that mea- sures (i) the extent which our methods mit- igate translationese as well as adequacy and fluency of the outputs (Quantitive Analysis), (ii) estimates the degree of translationese in the output using metrics derived from linguis- tic properties of translationese (Qualitative Analysis).
2.2 Unsupervised Model Selection
Several studies exploring unsupervised or semi/self- supervised settings either do not report their vali- dation scheme (Artetxe al., 2020b) or are only unsupervised (or semi/self-supperiieed during train- ing and rely on parallel in-domain validation sets for model tuning (Marie and Fujita, 2018: Marie et al., 2019; Dai et al., 2019; Ruiter et al., 2022). In contrast, some studies enforce strictly unsuper- vised settings in NMT by either using the vali- dation set from separate language pair (Artetxe et al., 2018a.b) not using validation scheme at all (Lample et al., 201 8c), risking sub-optimal models. To address this, Lample et al. (2018b,a); Artetxe et al. (2020b) proposed using an unsuper-
2
Related Work
2.1
Text Style Transfer
Text style transfer is the task of altering the stylistic characteristics of a sentence while preserving the original meaning (Toshevska and Gievska, 2022). The amount of parallel data available for this task is usually limited. Therefore, readily available mono-stylistic data together with a smaller amount of style-labeled data are often exploited using ap- proaches based on self- (Ruiter et al., 2019), semi- (Jin et al., 2019) or unsupervised Neural Machine

---

og-lio
Presrained LM
Cross Entropy Lose
II. (kk)
Semanto Similiaity
Sementic Similanty Loss
TR
Encodar
OG
Accepted pairs
SPE
Figure I: Model architecture. Here (tr, og) is a (Translated (source), Original (target)) style sentence pair, and og-like is the translationese-miiithe output. The dashed arrows correspond to on-the-fly parallel pair extraction that facilitates Supervised Training, while the red arrows in bold represent the path of approximated decoder outputs used in Unsupervised Training.
vised validation criterion over monolingual data that conforms with the evaluation criterion is guided by the target-distribution, similar Artetxe al. (2019) who combined cycle consistency loss with language model loss for unsupervised model selection.
ies specifically analyze the surface forms of the "debiased text".
To date, to the best of our knowledge, monolin- gual translation based style transfer on translation outputs to mitigate translationese has not been ex- plored. To some extent, this is expected, as, at least for monolingual translation-based style trans- fer, parallel original and translated texts in the same language do not exist. Below present novel ap- proach that builds translation-bau based style trans- fer but unlike previous work without parallel data for both training and validation sets.
2.3 Translationese Mitigation
Researchers have explored the effects and origins of translationese in previous studies. To miti- gate translationse effects in machine translation (MT) models, a prevalent approach is tagged train- ing (Caswell et al., 2019; Marie al., 2020; Ri- ley et al., 2020). This technique involves explic- itly marking the translated and original data using tags, enabling the models to recognize and account for these distinctions. Yu et al. (2022) introduced approach to mitigate translation artifacts in the wransate-traite cross-lingual setting. To re- duce translation artifacts in the target language, they learned an aogginal-to-translatiee function from the source language. They do this by projecting the original and translated texts in the source language to a common multilingual em- bedding space and then learning to minimize the distance between the mapped representations of the originals and translationese. Dutta Chowdhury et al. (2022) tackle the reduction of translationese from bias in trans- lated texts. They employ a debiasing approach to mitigate translationese by attenuating it the latent representations, while Wein and Schneider (2023) reduce translationese using AMR as intermediate representations. However, none of the above stud-
Translationese Mitigation via Style Transfer
Our goal is to climinate translationese signals from translated texts by transforming them into an original-like version. We define two style at- tributes, og and tr, representing original style and translated style, respectively, Given a text sample belonging to tr, our aim is to convert this Itr to Tag where Zog belongs style og but retains the same semantic content Zir- We denote the corpus with original sentences OG and the corpus with translated sentences T.R We illustrate the process Figure
mapping
3.1 Self-Supervised Architecture
different perspective, treating
In our work we build on a Transformer-based ENC- DEC self self-supervised system that jointly learns sen- tence pair extraction and translation in virtu- ous loop. Given two comparable mono-stylistic corpora (OG and TR), a sentence-pair extraction (SPE) module (Ruiter al., 2019) utilizes the inter- representations of the sentence pairs to extract sentences with similar meanings. This similarity matching module employs two types of latent rep-
'In this setting, the training data is translated from the source language into the target language and the translated texts are used for training.

---

Source ['Transtated]
Target [Original]
This is an urea in which we need to press on.
This is another aspect we have to work on. My group has discussed in detail the questions that you have posed. That not the case all. will
My group fully supports the substance of what you have said. That is not all the case.
shall endeavour be bricf.
brief.
Table I: Examples of accepted monolingual original-translationess pairs.
resentations: the sum of word embeddings w+() and the sum of encoder outputs e(*). The embed- ded pairs (f ((tr), ) and {e(tr), e(og))) are individually scored using a margin- baae measure (Artetxe and Schwenk, 2019), and the top candidate pairs selected for further filtering. Following Ruiter al. (2019), we apply two filtering criteria:
3.2 Joint Training Architecture
In the baseline system, all sentence pairs rejected by SPE are simply discarded, which is a major loss of useful mono-stylistic information. One way to utilize the discarded pairs is by combining the pervised training criterion with unsupervised train- ing. To this end, we introduce an unsupervised loss component to the final objective, which combines language model (LM) loss with semantic similarity loss. Both the losses, as shown in Figure 1, are computed over the decoder outputs when mono- stylistic ir is given as input.
Without Threshold [1]: A sentence pair is accepted for training if it is highest-ranked in both candidate-pair representations. This is used in Ruiter al. (2022).
As the input compute these losses is derived from a categorical distribution (i.e. after apply- ing argmax on the decoder output), this breaks the overall differentiability of the model during train- ing. Therefore, following Yang et al. (2018) and Unanue et al. (2021), during training, we use con- tinuous approximations of the decoder output to compute the two losses.
With Threshold [2]: A sentence pair is ac- cepted for training either it is highest-ranked in both candidate-pali representations if its encoder representation {e(Ir), e(og)} sur- passes a given threshold.
Examples of extracted accepted pairs are shown in Table 1.
Let Yoy be the style-transferred output when Ttr is given as input. Using greedy decoding, the pre- dictions from the decoder the j-th decoding step can expressed as follows:
Extracted parallel sentence pairs are used in an online fashion to train the ENC-DEC model in su- pervised manner by minimizing the cross-entropy loss (Isup):
Vog
arg max p Ftr, 1 0):5 1. 1...,. Ptr
N
V
Lsup
Σ
Y log H
(1)
(2)
To retain the end-to-end differentiability of the model, we replace the discrete argmax operation with a Gumbel-Softtax (Maddison et al., 2017; Jang et al., 2017) distribution and obtain contin- uous approximations of the decoder outputs. Let pi denote the probability of the i-th token in the V-sized vocabulary at the j-th decoding step in Equation 2 and Py represent the entire probability vector step j. Then the components of p, can be approximated using:
where N is the length of the target sequence, V is the shared ENC-DEC vocabulary size, Y repre- sents the i-th element of the one-hot encoded true distribution at j-th position and H² the i-th ele- ment of the predicted distribution (hypothesis) H, The joint SPE-translation learning loop continues until convergence. We use BART-style denoising autoencoding (DAE) (Lewis et al., 2020) for model initialization (see details in Section 4.2).
exp((logp)) + g D exp( pj + 9 //)/))
(3)
'The threshold was determined empirically by inspecting values in the range of [0.95, 1.03), and validated through a manual assessment of the quality of accepted pairs. We ob- serve that with higher threshold, fewer yet better quality (tr,og og) parallel pairs were extracted. We, therefore, set the value to 1.01 or 1.02 in our experiments (see A.5) as it pro- vided a good trade-off between the quality and quantity of accepted pairs.
where is sample drawn from the Gumbel (0, 1) distribution and 7 is temperature parameter' that controls the sparsity of the resulting vectors.
set to 0.1 (Jauregi Unanue et al., 2021)

---

The continuous probability vectors denoted as T at each decoding step j. represent probabilities over tokens in the shared ENC-DEC vocabulary. During the training phase, we use these vectors to compute the language model loss and the semantic similarity loss. We define our language model loss and semantic similarity loss below.
Duckworth al., 2019). This approach works par- ticularly well as we only feed mono-stylistic input to the Transformer (Vaswani et al., 2017) for unsu- pervised training. During training, the Transformer decoder run once without accumulating gradi- ents, and the (shifted) predicted sequence together with the encoder output are then fed into the Trans- former decoder again to compute the unsupervised losses described above.
Language Model Loss: To ensure target-style fluency, we follow the mantinous-approxxmaiio approach from Yang et al. (2018). In this ap- proach, we use the language model as a discrimi- nator. The language model is initially pretrained on the larget-style (i.e. the originals) to capture the target-style distribution and is denoted as L.Mog- We feed the approximated decoder output y (from the j-th decoding step) to the pretrained language model LMog by computing the expected embed- dings J, where Fi represents the embedding matrix of LMog- The output received from L.Mog is a probability distribution over the vocabulary of the next word, 9 Then the loss j-th step is defined as the entsspy loss as shown below:
During the validation phase, the output from the first-pass decoding is used to compute the semantic similarity in terms of BERTScore between the input translation and the style-transferred output. Addi- tionally, mean per-word entropy is computed over the trans-rraneferred output. Note that, to measure semantic similarity between input Zir and output Yogg during training, the Style Transfer Encoder is used compute cosine similarity while during val- idation, a pretrained BERT (Devlin al., 2018) model is employed for measuring the BERTScore. The reason is that as the Style Transfer Encoder is continually learning to represent the two styles in every iteration, the pretrained BERT model pro- vides a more stable and reliable similarity measure at every validation step.
Lm j+1 log 9,+1
(4)
When the output distribution from the decoder т i+ matches the language model output distribu- tion 4j'n 1. the loss achieves its minimum.
The unsupervised objective is a linear combina- tion of the language model loss and the semantic similarity loss, weighted by hyper-parameters 8 and y:
og
To enforce content preservation, the encoder representations of the input translation e(tr) and the expected token em- beddings from the decoder el FT are used to compute cosine similarity. Here, Eenc represents the embedding matrix of the style transfer trans- former encoder and e(*) refers the contextual- ized encoder representation. We define the loss as the mean-squared error of the cosine similarity Ioss:4
Semantic Similarity Loss:
Lunsup =iLim + 7Los
(6)
The final loss is the sum of the supervised and unsupervised components:
L = a a1)
The hyperparameter determines the balance be- tween the two objectives and can either be fine- tuned trained'. Note that while validation hap- pens only over the unsupervised loss, in the final joint training objective, both the supervised Lsup 1Fq.1) and the unsupervised loss Tunsup (Eq.6) are considered. Furthermore, in joint training, an ini- tial self-supervised training phase is carried out for 300 optimization steps for English (EN) and 600 optimization steps for German (DE) to facilitate style-transfer learning in a guided manner and only then it is combined with unsupervised training.
Las 1 E0- cos cos(r( (eliee))e(FFocee (Feen
(5)
where M refers to the number of input sentences in batch.
Training and Validation To achieve a continu- ous approximation of the decoder output at each time-step and ensure end-to-end differentiabilityy we employ a two-pass decoding approach (Mi- haylova and Martins, 2019; Zhang et al., 2019;
'In our experiments, both / and 7 are set to 1. "We set it to 0.7 in our experiments to regulate the effect of weaker unsupervisory signals us monostylistic TR data is much larger than the accepted pairs used for supervised learning.
'Wo also experimented with directly minimizing the co- sine embedding loss at a lower learning rate and observed no differences.

---

4 Experimental Settings
sequence length of 512 sub-word units. For online parallel-pair extraction, SSNMT requires index- ing the mono-stylistic corpora for fast access. We use FAISS (Johnson et al., 2019) for this purpose. Sentence vectors are clustered into buckets using IVF 00 (wherein 100 equals k in k-means ()) and stored without compression (i.e. with Flat index- ing). At search time, the top 20 buckets matching query are examined. All other parameters set to the values reported in (Ruiter al., 2022).
4.1 Data
Training Data: We use subset of the EuroParl corpus annotated with translationese information from HaapponsahKKaakyiin et al., 2021) (referred to as MPDE). Our focus is on two language scenarios: (i) EN-ALL: English originals (EN), and German and Spanish (ES+DE=ALL) translated into English and, (ii) DE-ALL: German originals (DE), and English and Spanish (ES+EN=ALL) translated into German.
Joint Training, we use the Fairseq imple- mentation of a Transformer Decoder (Vaswani et al., 2017) Language Model to compute Language Model Loss during training and validation. The same model is used to measure perplexities dur- ing testing. Further information regarding the hy- perparameters used for training the Style-Transfer models under different scenarios (i.e. with with- out threshold across different data settings) can be found in Appendix A.5.
Validation and Test data: The Self-Supervised baseline (SSNMT) relies on a parallel validation set for hyperparameter tuning. However, as this kind of data does not exist naturally, we generate (mTR) validation and test data by round-trip -uid-trip translating the original sentences (og) in the target language. Similarly, we denote the human-transsltionnsee as hTR. For our baseline SSNMT-based monolingual style transfer model, instead of using unaligned (hTR, og) pairs for vali- dation, we utilise aligned (mTR, og) pairs.
4.2.1 Classifier
The style transfer models are evaluated ing a BERT-based (Devlin et al., 2018) bi- nary classifier trained to distinguish human- translated data from originals. For English, we finetune bert-base-cased for the trans- lationese classification task, and for German, The binary classifier is trained on the MPDE data with equal amounts of Human-translater and original data (hTR, og). For EN-ALL, the training, validation and test splits for binary classification consist of 96536, 20654 and 20608 sentences, respectively while for DE-ALL, they consist of 87121, 18941 and 18976 sentences. respectively.
For EN-ALL, we translate the original sentences from the EN-DE validation and test splits using Facebook's s pre-trained 1M EN->DE and DE *EN models (Ng et al., 2019) and for DE-ALL, we use 00 (Fan et al., 2021) with EN as the pivot langauge for round-trip translation. Refer to Appendix A.1 for the dataset statistics of the Style Transfer model.
For DAE and Language Model pretraining (for Joint Training), English and German monolin- gual data are collected from the EuroParl Cor- pus (Koehn, 2005) from the OPUS website? (Tiede- mann, 2012). The English corpus contains over 1.5M train sentences and 5k dev and test sentences, while the German one has 2. 5k. 5k train, test and dev sentences, respectively. Noisy input data for BART-style pretraining generated with the values of parameters reported in Ruiter al. (2022). For LM finetuning, we use the Original training split of the Comparable Dataset used for Style Transfer (see Appendix A.1).
5 Evaluation
We perform evaluation on the outputs (og) of the style transfer models, given the human- translationese (hTR) (without an original refer- ence) half of the test data as input.
5.1 Quantitative Analysis
We compute three metrics: Acc. (Translationese Classification Accuracy), BERTScore (BERT-FI) and Perplexity (PPL)
4.2 Model Specifications
We implement our baseline system and the pro- posed Loint-Training system in Fairseq (Ott et al., 2019), using transformer-base with maximum
Acc.: This metric measures the extent to which the models mitigate translationese, in terms of the
hhtt::/Kopu.nipp..
1'This value set based on the size our corpus and the recommendation FAISS wiki. "Thhe higher the value, the longer the search time.

---

Setup
Quantitative
Qualitative
EN-ALL
Acc.[1]
Bert-F1
PPLoy
TTR
LD
#Identical
Original Test OG
0.902
33,70
Human-Translation Test
79.62
108.75
0.896
31.86
Self-Supervised [1]
71.00
0.94
37.24
0.841
27.13
1186
Self Self-Supervisel [2]
66.67
0.94
33.35
0.865
27.60
1186
Joint Training [1]-bTR
55.97
0.90
38.95
0.861
29.10
280
Joint Training [2]-bTR
57.32
0.90
50.98
0.887
32.33
390
Joint Training [I]-
52.89
0.89
42.29
0.872
29,43
215
Joint Training [2]-mTR
57.26
0.90
50.91
0,887
32.32
390
(a) Results on the English MPDE test data from different training sctups.
Setup
Quantitative
Qualitative
DE-ALL
Acc.(1]
Bert-F1
PPLog
TTR
LD
#Identical
Original Test OG
0.919
33.49
Test
79.30
142,66
0.913
35.66
Self-Supervised [1]
76.03
0.95
60,73
0.861
33.21
1913
Sule-Supervieed [2]
77.89
0.97
104.07
0.904
34.79
3481
Joint Training [1]-hTR
61.26
0.87
25.36
0.876
34.73
148
Joint Training [2]-bTR
63.50
0.87
82.47
0,843
39.75
181
Training [mIRR
59.48
0.87
77.99
0.881
35.51
146
Joint Training [2]-mTR
59.92
0.84
100.41
0.796
54.05
141
(b) Results on the German MPDE test data from different training setups.
Notation: No 11reshold:111, With Threshold:21; Validation set: human TannlationthTRRmmaelle traaslaaionnnnnltiio (mTR); Acc. classification accuracy on the entire test set (hTR, og) and on the syle-transferree outputs (óg. og); #ldentical: Houtputs that are same as the input. Bold numbers highlight the best overall result under each metric while the second best results are underlined.
accuracy of classifying the styles (translated vs. original). We report classification results the en- tire test set (hTR, og) and on the test sets generated from the outputs of our style transfer models (óg, og). Note that, a near-random accuarcy indicates better style transfer.
when compared to the hTR texts and 10.5% reduc- tion when compared to Self-Supervised [1] outputs. This indicates, for the baseline self-supervised ap- proach, additionally relying on threshold to ex- tract more parallel pairs helps improve style trans- fer for EN-ALL MPDE
BERT-F1 is computed between the input trans- lations and the style-transferred outputs to measure the degree of content preservation. For EN-ALL, we use pretrained rober ta-base and for DE-ALL xlm-roberta-base
Next, we examine if the same holds true for the Joint Training architecture and if joint training fur- ther improves the style transfer. In Table 2a, the consistent close-to- to-edoo accuracy across all the four variants of the Joint Training setup confirm the efficacy of this approach. The results show that in the Joint Training setup, regardless of the validation distribution (i.e., hTR or mTR), the style- transferred outputs from the models with no thresh- olds (i.e. Joint Training [1]-hTR and Joint Training (1]-mTR) achieve 2%. 8% reductions in accuracy w.r.t. their counterparts with thresholds.
PPL is calculated for the style-transferred outputs using language model (LMog) that has been fine- tuned on original text evaluate the fluency and coherence of the target-style generated text.
For EN-ALL, Table 2a shows that the binary classifier achieves an accuracy of 79.62 on the (hTR, og) test set. This is our reference point. First, we study both self-supervised approaches - with- out [1] and with a threshold [2]. Results show that both the baseline approaches [1.2] were able to reduce translationese in while preserving the semantics (as evidenced by the F1-BERTScore of 0.94). More precisely, Self- Supervised [2] reduces the classification accuracy by 16% when compared to the reference and by 11% when compared Self-Supervised [1]. Fur- thermore, the style-tran tral--rranferred outputs from Self- Supervised [2] show a reduction in LM,g perplexity
This observation suggests that when the model is trained with a larger amount of data through un- supervised training, it may be adequate to utilize only high-quality SPE pairs, without the need for including additional sub-optimal pairs based on threshold. When comparing these models against each other (i.e. Joint Training [1]-hTR vs. Joint Training [1] [m]-MTR] ), the generations from the model validated on hTR exhibit similar semantic
texts
12See Appendix A.2.1 for the statisties on the accepted pairs.

---

similarity but with approximately 8% lower per- plexity, which suggests a potential advantage in employing the same distribution (hTR) for valida- tion as during training and testing.
lines preserve the content better. Earlier studies have shown that BERTScore is sensitive both meaning and surface changes (Hanna and Bojar, 2021; Zhou et al., 2022), and therefore, one needs to manually examine the outputs.
We replicate the same analysis for the DE-ALL setup, as demonstrated in Table 2b. Note that, this dataset is even smaller than the EN-ALL MPDE dataset (see Table 4). Here, the reference accuracy is 79.30 on (hTR, og) test set. When using the baseline Self-Supervised methods [1,2], the classi- fication accuracy reduces only marginally by 2% to 4%, with no significant distinction between the two variants. Self-Superrised [1], however, unlike in EN-ALL, benefits from 12% lower LMag per- plexity than Self-Supervised [2]. Nonetheless, sim- ilar to the EN-ALL setup, with Joint Training, the degradation is more pronounced. In contrast EN- ALL, however, the style- transfrraneeeree outputs from Joint Training without threshold (i.e. Joint Train- ing [1]-hTR and Joint Training [1]-mTR) exhibit only a marginal decrease in accuracy compared to their counterparts with threshold. Assessing con- tent preservation and fluency in the target style within the Joint Training setup, all the models, ex- cept Joint Training [2]-mTR,, yield similar results. Notably, the outputs from Joint Training [1]-hTR demonstrate the lowest perplexity. In case of Joint Training [2]-mTR, the oddly high perplexity when evaluating with LM could be attributed to either the sub-optimal parallel pairs accepted by addition- ally applying a threshold or performing validation on mTR or both.
5.2 Qualitative Analysis
Prior research (Baker et al., 1993) has shown that translated texts are often simpler than original texts, In order to measure the level of translationese in translation in terms of lexical simplicity, Toral (2019) introduced two metrics: Type-Token Ratio (TTR) and Lexical Diversity (LD). Following Ri- ley et al. (2020), we conduct a qualitative analysis using these metrics.
TTR: Measures lexical variety by dividing the num- ber of unique word types in a text by the total number of words (tokens) in the same text. The lower the ratio, the more reduced the vocabulary of the style-transferre ouiput, indicating weak re- semblance to the target original style.
LD: Calculated by dividing the number of content words (words that earry significant meaning ad- jectives, adverbs, nouns and verbs) by the total number of words in the text. Higher content word density implics a text conveys more information, aligning it more closely with the original style.
Table 2a shows that for EN-ALL MPDE, the TTR and LD scores for the outputs of the base- line Supervisdd approach [2] are higher than that of Self-Supervised [1]. This is in line with the quantitative results, which clearly indicate that ad- ditionally applying a threshold to retrieve a higher number of parallel pairs helps improve style trans- fer. However, this does not hold for Joint Training. The TTR and LD scores for Joint Training [2]- (hTR/mTR) models are higher than for the models without threshold, although they achieve higher LMog perplexities. Interestingly, the LD scores from Joint Training [2]-hTR/mTTR even surpass the reference LD score on hTR.
Finally, we compare the Joint Training setup with the baseline Self-Supervised setup across both the MPDE datasets. Overall, Joint Training reduces the classification accuracy to a near-random accu- racy, with more pronounced drop in accuracy for EN-ALL MPDE 13 However, we also observe a degradation in the F1 -1-BERTSccre we move to the Joint Training setup. This discrepancy can be attributed to the fact that Self-Supervised training yields higher number of outputs that are identical to the given input translations 14, Therefore, a higher FI-BERTScomm does not necessarily mean that the non-identical outputs from Self-Supervised base-
In case of DE-ALL MPDE (Table 2b), which has an even smaller size of training data, lack of correlation between the quantitative and qualitative results is even more evident within the baseline self-supervised and the joint training setups. This suggests the need for an extrinsic evaluation on a downstream task, similar to the one performed by Dutta Chowdhury et al. (2022) on the Natural Language Inference (NLI) task.
Ioo closely inspect the impact of style transfer, Ap- pendix A.2, we report the classification accuracies on only the translationese half the test set. This provides an insight into the number sentences htR or og that are considered by the classifier as original-like (see #0G-like in Table 6c).
14 "Recall that the target reference is unavailable, and BFRTScore is computed between the translated input and the style-transferred output.
In Table 3, we present some examples from the different variants of our Style Transfer system and

---

In this work, we reduce the presence of transla- tionese in Huiman-translatee texts and make them more closely resemble originally authored texts. monolingual Source (bTR) huppily leave to you to examine this matter.
Self-Supervised21 leave you
Joint Training( [
Joint Training/I]-hTTR
I will leave to you reflect on this matter.
will therefore leave to you consider this matter.
examine this matter.
Unfortunately, this hope has not become reality.
Unfortunately, has not
Despite that. it has
The reality is, however, rather different.
become reality.
become reality.
Please could he just
Could just explain that?
Could he just explain that?
Could he please explain that?
explain that?
Please could you lake
Could you suitable
you please do something about
Could you please
suitablc action here.
action?
some action?
am in Tuurkyy's admittance EU,, but Copenhagen criteria
favour of Turkey's admittance
am very much in
very much favour Turkey's accession the European Union, but the Copenhagen criteria must be met.
of Turkey's admittance the EU, but these must taken into account.
the EU, but the Copenhagen criteria must
must be
Table Qualitative analysis of the outputs from different systems.
broadly analyse the generated outputs (og) with respect to some well-known linguistic indicators of translationese (Volansky et al., 2015)15 In the first and second example, the intended meaning and sur- face form is retained in the outputs from all systems while the outputs obtained from Joint Training[ hTR introduces a higher level of formality, altering both the connective and the phrase. In the second example, Self-Supervised[2) retains key elements from the source text, while Joint Training brings about more significant changes in terms of connec- tives, lexical choices, and sentence structure. In the third example, while Self-Supprrised[2) and Joint Training[ | IR change the formulation of the question while removing politeness ("please"), Joint [raining[[]-hTT] keeps it intact. Similarly, in the fourth example, all three versions use differ- ent connectives, with varying politeness and for- mality. In last example, supe-vupee retains a substantial portion of the source text, while Joint Training[ ( J-TTR eliminates the specific mention of the "Copenhagen criteria" and replaces it with "these," which a more general reference and Joint Trainingl --TT changes the phrase by using "acces- sion to the European Union" instead of "admittance to the EU."
translation baaed style transfer task. Due to the absence of parallel translated and original data in the same language, we employ self-supervised style transfer approach that leverages comparable monolingual data from both original and translated sources. However, even this self-supervised ap- proach necessitates the use of parallel data for the validation set to facilitate the learning of style trans- fer. Therefore, we develop a novel joint training objective that combines both self-supervised and unsupervised criteria, eliminating the need for par- allel data during both training and validation. Our unsupervised criterion is defined by combining two components: the original language model loss over the style-transferred output and the semantic simi- larity loss between the input and style- styne-rranferrred output. With this, we show that the generated out- puts not only resemble the style of the original texts but also maintain semantic content. We evaluate our approach on the binary classification task be- tween original and translations. Our findings show that training with joint loss significantly reduces the original and translation elassification accuracy to a level comparable to that of a random classifier, in- dicating the efficacy of our approach in mitigating translation-induus innduced--decd artifacts from translated data, As future work, we intend to explore more sophis- ticated approaches to improve content preservation and evaluation. It would be also interesting to ap- ply version of our style transfer approach to the output of machine translation to alleviate transla- tionese in cross-lingual scenarios.
Overall, we observe that our proposed approach is able to mitigate translationese in its style- transferred outputs while preserving the semantics reasonably well and achieving greater fluency in target original style.
6 Conclusion
Limitations
The availability of original and professionally trans- lated data in the same language, domain and genre is limited, both in terms of quantity and language coverage. While our system does not rely on par-
We approach the challenge
See Appendix A.3 for the analysis of German outputs.

---

allel data, for our approach to work it is important ensure that the data in both modalities (original and translationese) are comparable.
Do not rely on relay translations: Multilingual par- allel direct Europarl. In Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age, pages 1-7, online, Association for Computational Linguistics.
Manually evaluating the quality of style transfer and the reduction of translationese is inherently sub- jective. While we conduct preliminary analysis to evaluate the outputs, more nuanced línguistic exper- tise is required for in-depth analysis. Though we propose evaluation metries, there is no universally accepted gold standard for measuring the effective- ness of translationese reduction. These factors may introduce biases and challenges in comparing the performance of different approaches. At the indi- vidual text level, even human experts struggle to distinguish between translationese and originals. Detecting translationese reliably involves analyz- ing large quantities of data or training classifiers on original and translated text. While Amponsah- Kaakyire et al. (2022) show evidence that high- performance translationese classifiers may in some cases rely on spurious data correlations like topic information, recent work by Borah et al. (2023) indicates that the effect arising from spurious topic information accounts only for small part of the strong classification results. A decrease in classifier accuracy strongly suggests reduced translationese signals. That said, further research on addressing spurious correlations in translationese classifica- tion is an important research not explored in our paper.
Kwahena Amponsah-Kaakyire,, Daria Pylypenko, Josef Genabith, and Cristina España España-Bonet, 2022. Explain- ing translationese: why are neural classifiers better and what do they Icarn? Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpret- ing Neural Networks for NLP, pages 281-296, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre 2018a. Unsupervised statistical machine translation. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3632- 3642, Brussels, Belgium. Association for Computa- tional Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2019. An effective approach to unsupervised machine trans- lation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 194-203, Florence, Italy. Association for Computa- tional Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a. Translation artifacts in cross-lingual transfer learning, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7684, Online. Association for Computa- tional Linguistics.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b. Unsupervised neural ma- chine translation. In International Conference on Learning Representations.
Finally, it is worth noting that the proposed sys- tems may attempt to improve translations already of high quality that should not really be touched. Our results include potential quality degradation due to system overcorrections. Future work aims to address this phenomenon using an oracle-based style transfer approach.
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre 2020b. A call for more rigor in unsupervised cross-lingual learning, In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7375- 7388, Online. Association for Computational Lin- guistics.
7 Acknowledgments
Mikel Artetxe and Holger Schwenk. 2019, Margin- based parallel corpus mining with multilingual sen- tence embeddings. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics. pages 3197-3203, Florence, Italy, Asso- ciation for Computational Linguistics.
We would like to thank Etienne Hahn for help- ing with the manual analysis of German outputs. We also thank the anonymous reviewers for their invaluable feedback. This work was funded by the Deutsche Forschungsgemeinschaff (DFG, Ger- man Research Foundation) - SFB 1 1102 Informa- tion Density and Linguistic Encoding.
Mona Baker, Gill Francis, and Elena Tognini-Bool Bonelli. 1993. 'corpus linguistics and translation studies: Im- plications and applications'. In Text and Technology: In Honour of John Sinclair. John Benjamins Publish- ing Company.
References
Steven Bird, Ewan Klein, and Edward Loper. 2009. Nat- ural language processing with Python: analyzing text with the natural language toolkit. " "O O'Reilly Media, Inc.". "
Kwabena Amponsah-Kaake Daria Pylypenko,
Cristina Kispaña-Bonet, and Josef van Genabith. 2021.

---

Angana Borah, Daria Pylypenko. Cristina Espana- Bonet, and Josef van Genabith. 2023, Measuring spurious correlation in classification: "clever hans' translationese. In Recent Advances Natural Language Processing RANLP 2023, Varna, Bulgaria.
Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 72-81, Online. Association for Computational Linguisties.
Isaac Caswell, Ciprian Chelba, and David Grangier. 2019. Tagged back-translation. In Proceedings of the Fourth Conference on Machine Translation ( (Volume 1: Research Papers), pages 53 23-63, Florence. Italy. Association for Computational I Linguistics.
Michael Hanna and Ondiej Bojar. 2021. A fine-grained analysis of BERTScore. In Proceedings of the Sixth Conference on Machine Translation, pages 507-517, Online. Association for Computational Linguistics.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the As- sociattin for Computational Linguistics, pages 5997- 6007. Florence, Italy, Association for Computational Linguisties.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Cate- gorical reparameterizatioo with gumbel-softmax, In International Conference on Learning Representa- tions.
Inigo Jauregi Unanue, Jacob Parnell, and Massimo Pic- cardi. 2021. BERTTune: Fine-tuning neural machine translation with BERTScore, In Proceedings of the 59th Annual Meeting of the Association for Compu- rational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Popers), pages 915-924, Online. Asso- ciation for Computational Linguistics.
Jacob Devlin, Ming-Wei, Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/1810.04805.
Cícero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In An- nual Meeting of the Association for Computational Linguistics.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. IMaT: Unsupervised text attribute transfer via iterative matching and trans- lation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3097-3109 109, Hong Kong, China. Association for Com- putational Linguistics.
Daniel Duckworth, Arvind Neelakantan, Ben Goodrich, Lukasz Kaiser, and Samy Bengio. 2019. Parallel scheduled sampling-
Koel Dutta Chowdhury, Rricha Jalota, Cristina España- Bonet, and Josef Genabith. 2022. Towards debias- ing translation artifacts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3983-3991, Seattle, United States. Association for Computational Lin- guistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-seab scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-544 547.
Yunsu Kim, Miguel Graça, and Hermann Ney. 2020. When and why is unsupervised neural machine trans- lation useless? In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 35-44, Lisboa, Portugal. Euro- pean Association for Machine Translation.
Angela Fan, Shruti Bhosale, Holger Sehwenk, Zhiyi Ma. Ahmed Ei-Kish Siddharth Goyal, Man- deep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2021. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1).
Philipp Koehn. 2005, Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79-86, Phuket, Thailand.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2017. Style transfer in text: Exploration and evaluation. In AAAI Conference on Artificial Intelligence.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Bullis,n-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondfej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the As- sociation for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Hongyu Gong. Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training cor- pus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putationaa Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Popers), pages 3168- 3180, Minneapolis, Minnesota, Association for Com- putational Linguistics.

---

David Kurokawa, Cyril Goutte, and Pierre Isabelle. 2009. Automatic detection of translated text and its impact machine translation. In Proceedings of Machine Translation Summit XII: Papers.
Benjamin Marie, Haipeng Sun, Rui Wang, Kehai Chen, Atsushi Fujita, Masao Utiyama, and Elichiro Sumita. 2019. NICT's s unsupervised neural and statistical machine translation systems for the WMT19 news translation task. In Proceedings of the Fourth Con- ference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 294-301, Florence, Italy. Association for Computational Linguistics.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representa- tions.
Tsvetomila Mihaylova and André F. T. Martins. 2019. Scheduled sampling for transformers. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties: Student Research Work- shop, pages 351-356, Florence, Italy. Association for Computational Linguistics.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In Interna- tional Conference on Learning Representations.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc Aurelio Ranzato, 2018c. Phrase- based & neural unsupervised machine translation. Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 5039-5049, Brussels, Belgium. Association for Com- putational Linguisties.
Nathan Ng. Kyra Yee, Alexei Baevski, Myle Ott. Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1). pages 314-319, Florence, Italy, Association for Computational Linguistics.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In International Conference on Learning Representa- tions.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Adapting translation models to translationese improves smt. In Proceedings of the 13th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 255-265.
Shrimai Prabhumoye. Yulia Tsvetkov, Ruslan Salakhut- dinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
Parker Riley, Isaac Caswell, Markus Freitag, and David Grangier. 2020. Translationese as a language in "mul- tilingual" NMT. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7737-7746, Online. Association for Computational Linguistics.
BART: Denoising sequence-to-sequee for natural language generation, translation. and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Dana Ruiter, Cristina España- Bonet, and Josef van Gen- abith. 2019. Self-supervised neural machine transla- tion. In Proceedings of the 57th Annual Meeting of the A Association for Computational Linguistics, pages 1828-1834 Florence, Italy. Association for Compu- tational Linguistics.
Ruibo Liu, Chongyang Gao, Chenyan Jia, Guangxuan Xu, and Soroush Vosoughi. 2022. Non-parallel text style transfer with self-parallel supervision. In Inter- national Conference on Learning Representations.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relax- ation of discrete random variables. In International Conference on Learning Representations.
Dana Ruiter, Thomas Kleinbauer, Cristina España- Bonet, Josef van Genabith, and Dietrich Klakow. 2022. Exploiting social media content for self- supervised style transfer. In Proceedings of the Tenth International Workshop on Natural Language Pro- cessing for Social Media, pages 11 11-34, Seattle, Wash- ington. Association for Computational Linguistics.
Benjamin Marie and Atsushi Fujita. 2018 Unsuper- vised neural machine translation initialized by unsu- pervised statistical machine translation.
Benjamin Marie, Raphael Rubino, and Atsushi Fujita. 2020. Tagged back-translatie revisited: Why does it really work? In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5990-5997, Online. Association for Computational Linguistics.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Informa- tion Processing Systems, volume 30. Curran Asso- ciates, Inc

---

Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in opus. In Proceedings of the Eight Inter- national Conference on Language Resources and Evaluation (LREC"1)) 12), Istanbul, Turkey. European Language Resources Association (ELRA).
pages 979-988, Melbourne, Australia, Association for Computational Linguistics.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P. Xing, and Taylor Berg- Kirkpatrick. 2018. Unsupervised text style transfer using language models as discrimina- tors. In Neural Information Processing Systems.
Ewoenam Kwaku Tokpo and Toon Calders. 2022. Text style transfer for bias mitigation using masked lan- guage modeling. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciution for Computational Linguisties: Human Lan- guage Technologies: Student Research Workshop, pages 163-171, Hybrid: Seattle, Washington On- line. Association for Computational Linguistics.
Sicheng Yu, Qianru Sun, Hao Zhang, and Jing Jiang- 2022. Translate-train embracing translationese arti- facts. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Vol- ume 2: Short Papers), pages 362-370, Dublin, Ire- land. Association for Computational Linguistics.
Antonio Toral. 2019. Post-editese: an exacerbated translationese. In Proceedings of Machine Trans- lation Summit XVII: Research Truck, pages 273-281, Dublin, Ireland. European Association for Machine Translation.
Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation ( (Volume Research Papers), pages 73- 81, Florence, Italy. Association for Computational Linguisties.
Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? reassessing claims of human parity in neural machine translation. In Pro- ceedings of the Third Conference Machine Trans- lation: Research Papers, pages 113-123, Brussels, Belgium. Association for Computational Linguistics.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019. Bridging the gap between training and inference for neural machine translation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4334- 4343, Florence, Italy. Association for Computational Linguistics.
Martina Toshevska and Sonja Gievska. 2022. A re-
view of text style transfer using deep learning, IEEE Transactions on Artificial Intelligence. 3(5):669 684.
Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky. 2022. Problems with cosine as a measure of embedding similarity for high frequency words. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 401 401-423, Dublin, Ireland. Association for Computational Linguisties.
Inigo Jauregi Unanue, Jacob Parnell, and Massimo Pic- cardi. 2021. Berttune: Fine-tuning neural machine translation with bertscore. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence Natural Language Processing (Volume 2:
Short Papers), pages 915-924.
A
Appendix
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Mia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
A.1
Dataset Statistics for Style Transfer System
Data Preprocessing: into sentences using NLTK (Bird et al., 2009) and then tokenized and truecased using standard Moses scripts (Koehn et al., 2007). In addition, we remove all duplicates from the train/test/dev splits. After this, byte-pair encoding of 10k merge operations is applied on the concatenated monolingual-EN EuroParl and MPDE's EN-ALL training split and similarly, 1 1k merge operations are applied on the concatenated monolingual-Do EuroParl and the DE-ALL training split.
First, the paragraphs are split
Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30( 1)998-118.
Shuo Wang, Zhaopeng Tu, Zhixing Tan, Shuming Shi, Maosong Sun, and Yang Liu. 2021. On the language coverage bias for neural machine translation. In Find- ings of the Association for Computational Linguis- tics: ACI-IJCNLP 2021, pages 4778-4790, Online. Association for Computational Linguistics.
Shira Wein and Nathan Schneider. 2023. Translationese reduction using abstract meaning representation.
A.2 Supplementary Results
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xu- ancheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-senin cled reinforcement learning approach. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers),
A.2.1 Parallel Pairs for Joint Training
translation: A cy-
Table 5 provides an insight on how the accepted parallel pairs influence style transfer training. The models corresponding to the underlined numbers

---

Data EN-ALL
Split
0G
TR
EN-ALL
Acc./2]
#OG-like
train
96,290
96,206
Human- Translation Test
82.83
dev
10,327
10,327
test
10,304
10,304
Self-supervised [1]
66.72
3430
DE-ALL
truin 64,917
43,560
Self-xupervieed [2]
58.87
4239
dev 9,470
9,470
Joint Training [I]-TTR
35.70
6625
lest 9,488
9,488
Joint Training [2]-hTR
38:40
6347
Joint Truining [I]-mTR
29.54
7260
Table 4: Number of sentences in each split of the Original [OG] and Transiationese(TR) halves the MPDE dataset.
Joint Training [2]-mTR
38.28
6360
(a) Size of the test set: 10304.
DE-ALL
Acc.[2]
#OG-like
attain the best checkpoint earlier than their counter- parts and hence, we report the number of parallel pairs from this epoch. As seen in the quantita- tive analysis for EN-ALL, the large number of ac- cepted pairs due to the application of threshold in the self- supervidee approach, helps improve style transfer. However, for the Joint Training, due to the weaker unsupervised signals, the threshold does not have the same impact. Interestingly. for DE-ALL, the additional use of threshold in both the Self- Supervised and Joint Training approaches does not increase the number of parallel pairs. Furthermore, across both the data setups, Joint Training with no threshold achieves greater number of parallel pairs than its counterpart with threshold. This ex- plains why the quantitative metrics perform slightly better for this model variant.
Test
82.14
1695
Self-supervised [1]
76.15
2263
Self-supervised [2]
79.87
1908
Joint Training []]-TTR
19,90
7600
Joint Training [2]-hTR
24.58
7156
Joint Training [1]-mTR
16.18
7953
Joint Training [2]-mTTR
17.10
7866
(b) Size of the test set: 9488.
(c) Supplementary results on MOTRA EN-ALL and DE-ALLL test sets. Notation: No Threshold:[1]. Validation set: numan-translationte Tmyccchin translaatannniiionnn (mTR); Acc. classification accuracy on the translationese half the test set (hTR,) and on the stylettransfere outputs (ogg);: #OG-like: #sentences in (1TR.) and (oig,) that are classi- fied as original by the classifier.
Threshold:2]):
the first two short sentences, although Joint Training[1]-hTe makes a different lexical choice with the use of "Klarstellung" and "allerdings", it does not alter the meaning of the two sentences. In the third example, with the use of "vertrauen darauf", Joint Training J-hTR heightens the inten- sity of the sentence while preserving the intended meaning. In the final example, the use of the pronoun "Dies" (this) by Joint Training[ results in a loss of specificity regarding the city of Straßburg. However, both Seff-Supervised[2) and Joint Training(2J-hTR manage to maintain the source sentence's 's meaning, albeit with a slight shift in the intensity of the word "schöne" (beautiful).
Data EN-ALL
Setup Self-Supervised Joint Training-hTR
Epoch
No Thresh.
Thresh.
317
2.978
492
405
Joint Training-mIW
492
405
DEALLL
Sele-Supervisel
12
416
362
Joint Training-hTT -/TR
48
673
407
Joint Training-mTR
48
673
398
Table 5: Statistics of the accepted parallel pairs for style trans- fer training. The number of parallel pairs are reported from the earliest epoch that gives the best model checkpoint between the two model variants: with and without threshold, In each row, the epoch number ussociated to the underlined model.
A.4 Experimental Setup
A.2.2 Translationese Classification on the Style- TeanseTranserreed Outputs
We run experiments on SLURM cluster using GPU instances of V 1100-32GB or RTXA6000. Both the baseline and the joint training approach are run on a single GPU while for DAE pretraining and LM training (pretraining and finetuning), we use GPUs with 16 CPUs with each CPU containing 12 GB memory. One training on our datasets takes approximately 4 hours with the Baseline Approach and 7-9 hours for the Joint Training Approach. Additionally, obtaining results for all the setups in the quantitative measures simultancously takes approximately 30 minutes for each setup. How-
In order to directly witness the impact of style trans- fer, we measure the accuracy just on the transla- tionese half of the test set. This provides an insight into the number of sentences in the translated half that are considered by the classifier as original-like (see #0G-like in Table 6c). Note that, the lower the accuracy, the better the style transfer.
A.3 Manual Inspection of German outputs
Table 7, we provide examples from the style transfer models trained on MPDE DE-ALL. For

---

Source (hTR)
Self-Superviee[[]]
Joint Training[[]-mTT
Joint Training[
Das bedarf einer
Das bedarf einer
Das bedarf einer
Dos bedarf ciner
Erklärung.
Erklärung.
Erklärung
Klarstellung
Es besteht jedoch cin
Es bestcht jedoch cin Problem
Es besteht allerdings cin Problem
Es bestcht allerdings ein
Problem
Problem
botlen, Sie schaffen.
daB
vertrauen daBi Sie schaffen.
schaflen.
schaffen.
Straßburg durchaus cine schöne Stadt.
Straßburg ist eine schöne
Dies eine schüne Stadi.
Straßbuug ist eine sehr schöne Stadt.
Stadt.
Table 7: Qualitative analysis of the German outputs from different systems.
ever, if we run them independently, it takes be- tween 2 10 minutes to obtain results using these measures. The qualitative measures, the other hand, take longer process due the utilization of large spaCy models (en_core_web) wew_wwebrfrff for EN and de_ se__epp_eewe ptp__ews_trr trf for DE), requiring approxi- mately 15-20 minutes.
A.5 Hyperparameters
Experiment
MPDE
Experimental Setting
SSNMT requires indexing the mono-stylistic cor- pora for fast access. We use FAISS (Johnson et al., 2019) for this purpose. For indexing with FAISS, since each half of the baseline corpus (details in Section 4.1) contains less than 1M sentences, to facilitate fast and accurate search, the sentence vectors are clustered into buckets using IVF100 (wherein 100 equals in k-means) and stored without compression (i.e. with Flat indexing). At search time, the top 20 buckets matching query are examined.
Self-Supervied
Learning Rate 0:0003
Validation Ratch size 500
Warm-up updates 300
Batch size 80
Save interval updates
Threshold 1.01 (when
Epochs 100
Training
Learning
Validation Batch size 500
Warm-up updates 300
Batch-size 160
Suve-interval-uppe
Threshold 1.01 (when set) Start unsupervised training 300
Supervised-low -osss coeflicieat
Other hyperparameter settings for the Style Transfer models are shown in Table 8.
Unsupervised loss cient
coetli-
Epochs 30
The BERT-based Translationese binary classi- fiers are fine-tuned for at most 10 epochs with learning rate 2e-5, 1000 warm-up updates and a batch size of 16. The Transformer-basee LM is trained for at most 50k updates with a learning rate of 5e-3 and finc-tuned for at most 20k up- dates at lower learning rate of 5e-4 with the eos sample-breek-mooee
Patience
Joint Training
DB-ALL
Learning Validation Batch size 500 Warm-up updates 600
Ratch-size 40
Save-interval-apphnn
Threshold 1.02 (when set)
Start unsupervised training
Supervised-loss coeflicieat
Unsupervised-loss coeffi.
The BERT-based Translationese binary classi- fiers are finetuned for most 10 epochs with learning rate of 2e-5 and 1000 warm-up updates. LM trained for at most 50k updates with a lcarning rate of 5e-3 and finc- tuned for at most 20k updates at a lower learning rate of 5c-4.
Epochs 30 Patience
Table Hyperparameters used the Style Transfer experi- ments.
The Transformer-batt
16This value is set based on the recommendation in FAISS wiki.


Reviews:
[{"review_id": "twzM3K0d7g", "reviewer": "Reviewer_PG8x", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1 for the thoughtful feedback. We address  translationese mitigation through translation-based style transfer. Due to the impossibility of obtaining parallel mono-lingual translationese-original data, our approach centers on a self-supervised NMT approach that simultaneously learns to perform two tasks in an online fashion: 1) extract (near) parallel pairs from a comparable original-translationese corpus, and 2) perform style transfer using the parallel pairs (a task made complex by the absence of parallel translated and original data in the same language). To eliminate the need for parallel validation data and to effectively utilize the monostylistic translationese data, we extend the baseline self-supervised system with unsupervised learning by combining a language model loss and a semantic similarity loss. Our experimental results indicate our approach is able to reduce translationese classifier accuracy to a level of a random classifier after style transfer while adequately preserving the content and fluency in the target original style.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, and experiments are described well. \nThe authors not only explain the challenges and provide solutions to the challenges but also provide a suitable justification for the solution.\nThe method combines self-supervised and unsupervised learning, eliminating the need for parallel corpora, which doesn't exist for this particular task. Moreover, the method extended to low-resource settings as well.\n", "reasons_to_reject": "None", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores a novel approach to reduce translationese in translated texts which is a translation-based style transfer. As there is no parallel translated and original data in the same language, the author uses a self-supervised approach that can learn from comparable mono-lingual original and translated data. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line number 533 \"translation. to rThis\""}, "extra_text": {}}, {"review_id": "lA4q5Sk2Ym", "reviewer": "Reviewer_k8r3", "summary": "", "questions": "A: If only one half of the test set changes between the different models being evaluated, why didn’t you report only results on that half? It would be clearer if you reported how many of the sentences of that half are considered by the classifier as “og”.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2 for the thorough and insightful feedback.\n\n> The use of a classifier for determining how well the approach converts the translationese into regular sentences undermines the reliability of the conclusions, as changes in its accuracy can be attributed to the classifier’s own biases, in addition to the properties of the sentences generated by the system.\n\n**(Soundness of using a classifier)** Previous research has shown that human experts (professional translators) are usually not able to reliably distinguish between originally authored and professionally translated texts in the same language, genre, topic etc. beyond chance ([Tirkkonen-Condit et. al (2002)](https://api.semanticscholar.org/CorpusID:144823252), [De Sutter et. al (2020)](https://www.tandfonline.com/doi/abs/10.1080/0907676X.2019.1611891)).  Reliably detecting translationese requires either (i) careful corpus analysis of sufficiently large volumes of original and translated data ([Gellerstam et. al (1986)](https://api.semanticscholar.org/CorpusID:59685951), [Baker et. al(1993)](https://api.semanticscholar.org/CorpusID:57174748), [Laviosa et. al (1998)](https://api.semanticscholar.org/CorpusID:678873)\n, [Koppel et. al (2011)](https://aclanthology.org/P11-1132)) to show differences in the distribution of e.g. discourse connectives and/or other translationese manifestations, or (ii) the use of classifiers trained on original and translationese data ([Baroni et. al (2006](ttps://api.semanticscholar.org/CorpusID:11006810), [Kurokawa et. al (2009)](https://aclanthology.org/2009.mtsummit-papers.9), [Rabinovich et. al (2015)](https://aclanthology.org/Q15-1030), [Volansky et. al (2015)](https://api.semanticscholar.org/CorpusID:5133943), [Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676)]). Given enough training data and given enough data at prediction time, translationese classifiers can be astonishingly accurate, with up to > 90% accuracy ([Rabinovich et. al (2015)](https://aclanthology.org/Q15-1030),  [Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676), [Amponsah et al. (2022)](https://aclanthology.org/2022.blackboxnlp-1.23)]). In our work, we use SOTA BERT-based approaches as e.g. by [Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676). Given that the MPDE EuroParl data ([Amponsah et. al (2021)](https://aclanthology.org/2021.motra-1.1)) we use is fairly uniform in terms of genre and domain, we (i) expect the classifiers to be reliable and (ii) reduced classifier accuracy to (near) random (per classification datum/input) to be a strong indication that (any previous) translationese signals have been reduced in the data under classification. In this respect, the methodology and approach we follow is as good and as bad (as sound) as the current SOTA in translationese research at large ([Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676), [Dutta Chowdhury et al. (2022)](https://aclanthology.org/2022.naacl-main.292)). That said, we do take the point raised by the reviewer: [Amponsah et al. (2022)](https://aclanthology.org/2022.blackboxnlp-1.23) show some evidence that high-performance translationese classifiers may use spurious correlations in the data (such as topic) that are not translationese signals. We are aware of previous research e.g. [Dutta Chowdhury et al.(2020)](https://aclanthology.org/2020.coling-main.532) that uses masking (e.g. PoS) to abstract from text data to mitigate possible topic influences in translationese data. However, they work at the level of word embedding spaces (hidden representations), while we are focusing on generating surface text (converting translationese into original style text). Quantifying (and perhaps even reducing) spurious correlations in (translationese) classification remains an important topic not addressed in our paper. If the paper is accepted, we will use the extra page to discuss the point raised by the reviewer following the outline above.\n\n> A downstream evaluation of the output of the system, for instance, by training an NLI model as did by Dutta Chowdhury et al. (2022), would assess in a more precise way the performance of the proposed approach.\n\nWe fully agree: extrinsic evaluation, such as NLI as in [Dutta Chowdhury et al. (2022)](https://aclanthology.org/2022.naacl-main.292) would be very useful to assess the performance of our approach. We looked at the data. Unfortunately, the substantial topic differences between the NLI data and our MPDE EuroParl data ([Amponsah et. al (2021)](https://aclanthology.org/2021.motra-1.1)) require retraining our translating away translationese system to fit it to the NLI data. Without retraining, our approach would import unwanted domain differences over and above translationese mitigation in the conversion. This is related to question W1 of the reviewer above. We will explore our approach on the NLI data as part of our future work. \n\n**Q1: If only one half of the test set changes between the different models being evaluated, why didn’t you report only results on that half? It would be clearer if you reported how many of the sentences of that half are considered by the classifier as “og”.**\n\n**(Soundness):** Classification accuracies on only style-transferred human-translation (tr) outputs are shown below (in Table A) and are in line with the results on all inputs (tr + og) reported in the paper (compare Table 2). Lower accuracy on style-transferred tr, i.e. “mis-classifying” them as og, indicates that style transfer successfully mitigates translationese.\n\n*Table A: Classification results on 10,305 human translation (tr) test sentences of EN-ALL MPDE*\n|   EN-ALL    | Accuracy | # og-classified sentences  |\n|         :---:        |    :----:   |          :---: |\n| Human-Translation (tr) half     | 82.38       | 1816   |\n| Self-Supervised[1]   | 66.72        | 3430     |\n| Self-Supervised[2]   | 58.87        | 4239     |\n| JointTraining[1]-hTr   | 32.00        | 7008    |\n| JointTraining[2]-hTr  | 29.13         | 7303    |\n| JointTraining[1]-mTr   | 32.74        | 6931    |\n| JointTraining[2]-mTr  | 30.24         | 7189    |\n\n> The proposed approach is better than a simpler baseline without the auxiliary losses according to the classifier, but content preservation and perplexity show a mixed picture.\n\n**(Soundness)** Regarding the mixed results in content preservation, it is important to note that we compute BERTScore between the input-translated sentences and the style-transferred outputs. Therefore, if more sentences in the output are identical to the input, BERTScore would be higher. The scores in Tables 2 and 3 indicate that the self-supervised approach yields a higher number of identical sentences compared to the Joint Training method, and therefore, achieves a higher BERTScore indicating less Translationese to Original style-transfer. In a sense, BERTScore is sensitive to both meaning and surface changes ([Hanna et. al. (2021)](https://aclanthology.org/2021.wmt-1.59), [Zhou et. al. (2022)](https://aclanthology.org/2022.acl-short.45)).\nRegarding perplexity, the main goal is to assess whether the perplexities from both approaches are lower than those on the Human Translated test set when using a language model trained on originals. Perplexities in Tables 2 and 3 in the paper do not show a consistent trend across the languages in the two setups and we agree that this prompts at least the need for further manual inspection of the style-transferred outputs. We provide a short comparison in Table  B and will extend this in the revised version of the paper. \nAdditionally, we think our response to Reviewer 3 about the manual analysis of examples (Table C) can also be of interest in the context of the concern raised here. \n\n*Table B: Comparison of the outputs from the Self-Supervised and Joint Training system, using the same translated input sentences.* \n\n| Source (tr) | Hypothesis1: Self-Supervised[2] | Hypothesis2: JointTraining[1]-hTr | \n| :---: | :---: | :---: |\n| That is the first issue | That is the first issue | That is the first issue |\n| That is why I am asking for your support. | That is why I am asking for your support. | That is why I am asking for your assistance. | \n| Please could you take suitable action here . | Could you take suitable action ? | Could you please take some action ? |\n| They have presented their opinions very well , and I thank them for this . | They have done so very well , and I thank them for that . | They have expressed their views on this subject and I thank them for that . |\n\nTable C presents the style-transferred outputs resulting from both the baseline Self-Supervised approach and the proposed Joint Training approach. These outputs are generated using the same translated source sentence (tr) as input. Depending on the source sentence, both models either yield consistent outputs (the first row), or they exhibit differences in the lexical choice and sentence structure while still effectively capturing the meaning of the input (rows 2-3). In the last example, we observe that the baseline approach opts for a more general description (“done so”), potentially leading to a somewhat less specific representation. On the other hand, the Joint Training approach is more descriptive with its use of “expressed their views on this subject”, enhancing the overall clarity and concreteness of the conveyed source sentence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper advances state of the art when converting translationese into regular sentences (as if they had been written from scratch instead of being translated)\n- The paper is clear and well-written", "reasons_to_reject": "The use of a classifier for determining how well the approach converts the translationese into regular sentences undermines the reliability of the conclusions, as changes in its accuracy can be attributed to the classifier’s own biases, in addition to the properties of the sentences generated by the system. A downstream evaluation of the output of the system, for instance, by training an NLI model as did by Dutta Chowdhury et al. (2022), would assess in a more precise way the performance of the proposed approach.", "questions_to_authors": "A: If only one half of the test set changes between the different models being evaluated, why didn’t you report only results on that half? It would be clearer if you reported how many of the sentences of that half are considered by the classifier as “og”.", "paper_topic_and_main_contributions": "The paper describes a method to modify ‘translationese’ so that they look like regular sentences originally written in the target language. This has important implications in the training and evaluation of different natural language processing models. The proposed method involves training a Transformer encoder-decoder model that performs the conversion. It is trained in an iterative way from a comparable corpus of translationese/non-translationese sentences in the same language with two auxiliary losses: one that enforces that the generated sentence has the style of a sentence originally written in the target language, and another one that enforces semantics preservation. The proposed approach is evaluated according to the following factors: whether an automatic classifier is able to distinguish between the outputs and ”original” text, content preservation, and perplexity according to a language model trained on “original” text. The proposed approach is better than a simpler baseline without the auxiliary losses according to the classifier, but content preservation and perplexity show a mixed picture.\n\nThe main contributions of the paper are:\n- Presenting a new algorithm for changing the style of ‘translationese’ to “regular sentences”\n- Being the first algorithm for this purpose that actually generates surface forms and does not require an Abstract Meaning Representation parser", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "420: \"Refer to Section 4.1 for the dataset statistics\", but we are already in Section 4.1."}, "extra_text": {}}, {"review_id": "ngu418p7qF", "reviewer": "Reviewer_whSa", "summary": "", "questions": "- A) For the self-supervised part (Section 3.1), you use the SPE module to pick matching sentences for tr-og pairs that will be used to train the encoder-decoder. Each sentence is tested against every other sentence? What is this SPE module, more concretely? Is it a BERT-like encoder? Did you use the one from (Ruiter et al., 2019) or did you train your own? On what corpora was SPE trained on?\n- B) The SPE with threshold uses a threshold of 1.01. Can you say more on how this value was determined?\n- C) When calcutating the probability vectors, a temperature value of 0.1 is used. Can you say more on how this value was picked?\n\n[all questions were addressed in the rebuttal]", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3 for the insightful feedback and comments. \n\n> Is the approach valid? Fluency is assessed by a language model, content preservation is assessed by an encoder, and accuracy is assessed by whether a translationese classifier can be \"fooled\" into random classification. This is fine for an automatic evaluation, but these are only proxies for the \"real thing\", which is whether the fixed translationese is natively fluent, preserves content, and is enough to \"fool\" a human. This has to be evaluated manually. The task proposed in this paper is begging for some sort of A/B testing where the translationese text and the result of fixing it are put side-by-side and presented to human translators, which then have to pick the best. Without this, the approach isn't sufficiently validated. This issue is partly alluded to in the Limitations section at the end of the paper, but it nevertheless remains as a weakness.\n\n**(Evaluation-Soundness/Validity)**  We agree that in an ideal scenario, A/B testing using human-expert evaluation should be conducted. However, translationese effects for professional human translation (as we are considering in the MPDE Europarl data ([Amponsah et. al (2021)](https://aclanthology.org/2021.motra-1.1)) are subtle, and even human experts are usually not able to reliably distinguish between translationese and originals reliably above chance ([Tirkkonen-Condit et. al (2002)](https://api.semanticscholar.org/CorpusID:144823252), [De Sutter et. al (2020)](https://www.tandfonline.com/doi/abs/10.1080/0907676X.2019.1611891)). In fact, to reliably detect translationese one either (i) has to collect sufficiently large quantities of original and translation data and do careful corpus analysis to detect reliable differences in the distributions of translationese relevant linguistic phenomena such as for example the distributions of discourse connectives suggesting “explicitation” in translationese or (ii) has to train classifiers on original and translated data ([Baroni et. al (2006)](ttps://api.semanticscholar.org/CorpusID:11006810), [Kurokawa et. al (2009)](https://aclanthology.org/2009.mtsummit-papers.9), [Rabinovich et. al (2015)](https://aclanthology.org/Q15-1030), [Volansky et. al (2015)](https://api.semanticscholar.org/CorpusID:5133943), [Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676)). Given enough training data and given enough data at prediction time, translationese classifiers can be astonishingly accurate, with up to > 90% accuracy ([Rabinovich et. al (2015)](https://aclanthology.org/Q15-1030), [Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676),[Amponsah et al. (2022)](https://aclanthology.org/2022.blackboxnlp-1.23)). In our work, we use SOTA BERT-based approaches as e.g. by [Pylypenko et al. (2021)]. Given that our data, is fairly uniform in terms of genre and domain (EU MPDE data), we (i) expect the classifiers to be reliable and (ii) reduce classifier accuracy to (near) random (per classification datum/input) to be a strong indication that (any previous) translationese signals have been strongly reduced in the data under classification. See also Table A for Reviewer 2. In this regard, the methodology and approach we follow are as good and as bad (as sound) as the current SOTA in the translationese research field at large ([Pylypenko et al. (2021)](https://aclanthology.org/2021.emnlp-main.676), [Dutta Chowdhury et al. (2022)](https://aclanthology.org/2022.naacl-main.292)). \n\nIn addition to classification, we also use BERTScore as a proxy for content preservation and perplexity given a language model trained on original data as a proxy for closeness (or otherwise) to original data. Just as the classification results, these are proxies, reflecting current practice and SOTA in the field. \n\nIn the revised version of the paper, we intend to compile a more extensive list of transformation examples (see Table C below) for an extended manual inspection and error analysis. \n\n> Is there the danger of this approach ruining translations that are already good? The proposed approach appears to be a great technique to improve automatic translations, so much so that it seems that it would be something that one would always want to run over the output of an automatic translation system. However, this raises the question of whether doing so can actually do more harm than good when the translated text is already of good quality and has little to no translationese. However, we don't have an oracle that, for a given translated text, can tell us if we're in this situation, and thus if we should or shouldn't attempt fix the translation. The paper makes no mention of this scenario.\n\n**(Soundness)** Yes, overcorrection happens, which is also an important point in research on Automatic Post-Editing (APE) in Machine Translation where the output of neural machine translation systems in high-resource training scenarios is already very good and changes required to make outputs even better are often subtle and risk corrupting already good translations. At the moment, all the results in the paper are inclusive of any deteriorations brought about by overcorrections on the part of our system. We will highlight this explicitly in the paper. \n\nInspired by the reviewer’s question and their mentioning of an “oracle”, in the future, we may be able to extend the system architecture to incorporate oracle-guided style transfer. In this scenario, the oracle would be approximated by a pre-trained high-accuracy translationese classifier. During the training phase of the extended system, the decoder would receive two inputs: the extent of translationese in the input translated text (TR) (as inferred by the classifier) and the TR-encoder representations. This may enable the overall system to make a better-informed decision about style transfer based on the extent of translationese predicted in the translated input text. We will include this in the further work section.\n\n\n> The changes caused by this approach can go beyond simply fixing translationese. In Section 5.2, Table 4, the authors show some examples. In the fourth example, \"unless radical changes are made\" is transformed into \"if this House goes along with\". The authors show this as being a \"sentence structure\" change and state that it \"[provides] a different perspective on the conditional statement\". This transformation goes far beyond a structural/syntactic change or merely providing a differente perspective. The paper quickly glosses over this case, but a more detailed qualitative analysis of such results is in order. \n\nWe agree: the example in Table 4 goes well beyond structural or syntactic change and we will discuss this better in the paper. The system transformed \"unless radical changes are made\" to \"if this House goes along with\", both articulating perhaps a negative stance towards the Buitenweg report, capturing the context of the source sentence as part of its broader implication, while abstracting away from the specific condition mentioned in the source sentence. We will adjust the discussion of this example accordingly. Below in Table C we provide further qualitative examples (hypotheses) across different versions of our approach: \n\n*Table C: Qualitative analysis of outputs from different systems.* \n\n| Source (tr) | Hypothesis1: Self-Supervised[2] | Hypothesis2: JointTraining[1]-mTr | Hypothesis3: JointTraining[1]-hTr |\n| ----------- | ------------------------------ | --------------------------------- | --------------------------------- |\n| I happily leave it to you to examine this matter. | I leave it to you to examine this matter. | I will leave it to you to reflect on this matter. | I will therefore leave it to you to consider this matter. |\n| Unfortunately, this hope has not become reality. | Unfortunately, this has not become reality. | Despite that, it has not become reality. | The reality is, however, rather different. |\n| Please could you take suitable action here. | Could you take suitable action? | Can you please do something about it? | Could you please take some action? |\n| There is no doubt that the directive will have to be revised if its objectives for the future are to be achieved. | There is no doubt that the directive will have to be revised if its objectives are to be met. | There is no room for fudge or doubt that the directive will need to be revised if it is to be implemented. | There is no doubt that the directive will have to be revised if its objectives are to be met. |\n| I am in favour of Turkey’s admittance to the EU, but the Copenhagen criteria must be met. | I am in favour of Turkey’s’s admittance to the EU, but the Copenhagen criteria must be met. | I am very much in favour of Turkey’s admittance to the EU, but these must be taken into account. | I am very much in favour of Turkey’s accession to the European Union, but the Copenhagen criteria must be met. |\n\nOverall, we observe that the baseline Self-Supervised approach tends to be more conservative in its transformation of the source input (tr) compared to the Joint Training approach. With Joint Training, not only do we witness changes in lexical choices, but also see alterations in sentence structure that retain the original meaning. For instance, Joint Training[1]-hTr exhibits a  more liberal transformation of the sentence \"Unfortunately , this hope has not become reality .\" to \"The reality is, however , rather different .\", capturing and conveying the meaning of the source input from a distinct perspective. Similarly, the last example highlights the added fluency from Joint Training[1]-hTr in the source input due to the expansion of the abbreviation EU and use of a more refined word choice for \"admittance\".\n\n**Q1. For the self-supervised part (Section 3.1), you use the SPE module to pick matching sentences for tr-og pairs that will be used to train the encoder-decoder. Each sentence is tested against every other sentence? What is this SPE module, more concretely?  Is it a BERT-like encoder? Did you use the one from (Ruiter et al., 2019) or did you train your own? On what corpora was SPE trained on?**\n\nYes, the SPE module is used to extract (near-)parallel pairs from the comparable corpus to train the Encoder-Decoder.  Each sentence from Original (OG) is tested against all sentences in Translated (TR). We use FAISS index ([Johnson, et. al (2019)](https://arxiv.org/abs/1702.08734)) for this purpose, to make the search more efficient. Following [Ruiter et. al (2022)](https://aclanthology.org/2022.socialnlp-1.2), we build separate FAISS indexes for both TR and OG sentence representations and perform a bidirectional search to retrieve only high-quality pairs. This implies that during search time, the TR index is queried with an OG sentence vector to retrieve its nearest neighbors among the TR sentences, and vice-versa. This procedure results in two kinds of mappings: TR_to_OG and OG_to_TR. Only if a sentence-pair attains the highest rank in both directions, it is considered a candidate pair for further filtering, as described in Section 3.1 and in [Ruiter et. al (2019)](https://aclanthology.org/P19-1178).\n\nSPE is not quite a BERT encoder but a similarity matching module that receives translated and original latent representations from the Style Encoder to find and extract sentences with similar meanings ([Ruiter et. al (2019)](https://aclanthology.org/P19-1178)). Specifically, the module employs two types of representations, a) the sum of static word embeddings w(*) and b) the sum of contextualized encoder outputs e(*).  The embedded pairs {w(tr), w(og)} and {e(tr), e(og)} are individually scored using FAISS and a margin-based measure ([Artetxe  et. al (2018)]), and the top candidate pairs are selected for further filtering (as explained in Section 3.1).\nSPE is trained on the same comparable data (MPDE)  that is used for Style Transfer, and the two (SPE and Style Transfer) feed and improve each other in a virtuous loop. In SPE, parallel pairs are extracted in an online fashion. In every iteration, the Style Encoder learns to distinguish and represent the two styles (TR and OG) better. Consequently, the SPE module receives updated representations for TR and OG sentences in each iteration to be able to extract better quality pairs as the training progresses. \n\n**Q2. The SPE with threshold uses a threshold of 1.01. Can you say more on how this value was determined?**  \n\nThe value was determined empirically by inspecting values in the range of [0.95, 1.03], and validated through manual assessment of the quality of accepted pairs. We observed that with a higher threshold, fewer yet better quality original-translated parallel pairs are extracted. A threshold of 1.01 provided a good trade-off between quality and quantity. \n\n**Q3. When calculating the probability vectors, a temperature value of 0.1 is used. Can you say more on how this value was picked?**\n\nWe are glad that you asked because we missed a citation. The value was taken following the work ([Unanue et. al (2021)](https://aclanthology.org/2021.acl-short.115)) to enforce sparsity.\n\n> Unsupervised approaches seem almost magical, getting results from apparently nothing.\n\n**(Soundness/Excitement)** Previous approaches use un- ([Lample et al. 2018](https://arxiv.org/abs/1711.00043), [Artetxe et al. 2019](https://aclanthology.org/P19-1309)) or self-supervised ([Ruiter et al. 2019](https://aclanthology.org/P19-1178), [Liu et. al. 2022](https://arxiv.org/pdf/2204.08123.pdf)) methods to address the lack of parallel training data. Here we use them to address the logical impossibility of mono-lingual translated - original parallel training and validation data. This may well be a first.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Unsupervised approaches seem almost magical, getting results from apparently nothing. This paper goes a bit along those lines, being able to fix translationese without any explicit examples (parallel corpora) of how to do it.\n\nThe following contributions that the authors chose to highlight are all good reasons to accept the paper:\n- Tackling the task using a style-transfer approach is novel.\n- The method combines self-supervised and unsupervised learning, eliminating the need for parallel corpora, corpora that doesn't even exist for this particular task.\n- Translationese is successfully eliminated (according to an automatic classifier).", "reasons_to_reject": "In my view, the main weaknesses of the paper concern the evaluation. I'll address these issues next.\n\nIs the approach valid? Fluency is assessed by a language model, content preservation is assessed by an encoder, and accuracy is assessed by whether a translationese classifier can be \"fooled\" into random classification. This is fine for an automatic evaluation, but these are only proxies for the \"real thing\", which is whether the fixed translationese is natively fluent, preserves content, and is enough to \"fool\" a human. This has to be evaluated manually. The task proposed in this paper is begging for some sort of A/B testing where the translationese text and the result of fixing it are put side-by-side and presented to human translators, which then have to pick the best. Without this, the approach isn't sufficiently validated. This issue is partly alluded to in the Limitations section at the end of the paper, but it nevertheless remains as a weakness.\n\nIs there the danger of this approach ruining translations that are already good? The proposed approach appears to be a great technique to improve automatic translations, so much so that it seems that it would be something that one would always want to run over the output of an automatic translation system. However, this raises the question of whether doing so can actually do more harm than good when the translated text is already of good quality and has little to no translationese. However, we don't have an oracle that, for a given translated text, can tell us if we're in this situation, and thus if we should or shouldn't attempt fix the translation. The paper makes no mention of this scenario.\n\nThe changes caused by this approach can go beyond simply fixing translationese. In Section 5.2, Table 4, the authors show some examples. In the fourth example, \"unless radical changes are made\" is transformed into \"if this House goes along with\". The authors show this as being a \"sentence structure\" change and state that it \"[provides] a different perspective on the conditional statement\". This transformation goes far beyond a structural/syntactic change or merely providing a differente perspective. The paper quickly glosses over this case, but a more detailed qualitative analysis of such results is in order.\n\n[all issues were addressed in the rebuttal]", "questions_to_authors": "- A) For the self-supervised part (Section 3.1), you use the SPE module to pick matching sentences for tr-og pairs that will be used to train the encoder-decoder. Each sentence is tested against every other sentence? What is this SPE module, more concretely? Is it a BERT-like encoder? Did you use the one from (Ruiter et al., 2019) or did you train your own? On what corpora was SPE trained on?\n- B) The SPE with threshold uses a threshold of 1.01. Can you say more on how this value was determined?\n- C) When calcutating the probability vectors, a temperature value of 0.1 is used. Can you say more on how this value was picked?\n\n[all questions were addressed in the rebuttal]", "paper_topic_and_main_contributions": "The paper addresses the issue of \"translationese\", the common problem found in translated text whereby the text exhibits linguistic features distinct from those of natively-written texts, often being closer to the features of the source language.\n\nThe authors tackle the task of correcting, or at least mitigating, the translationese present in translated text. The proposed approach is based on machine translation and style transfer techniques and works directly over the translationese text, changing it into text closer to one written natively in the target language.\n\nMachine translation and style transfer techniques are typically trained and validated on parallel corpora, but there is no original-translationese parallel corpus that can be used for the task at hand. As such, the authors propose an unsupervised validation criterion based on the language model loss and the semantic similarity loss. The language model loss measures the fluency of the fixed translationese, as assessed by a language model for the target language, while the semantic similarity loss measures whether the semantic content has been preserved by comparing the similarity between the encoder representations of the translationese and the encoder representations of the fixed translationese.\n\nEvaluation addresses multiple aspects:\n- Fluency: The perplexity of the fixed translationese according to a language model for the target language.\n- Content preservation: The BERTScore that measures the semantic similarity between the translationese and the fixed translationese.\n- A sort of \"adversarial\" task: The accuracy of a translationese detector, which is expected to perform at about 50% (random choice) if the translationese has been successfully fixed.\n- Finally, the type-token ratio and lexical diversity of the produced text are also measured.\n\nThe approach is novel, the method avoids the need for parallel corpora, and the evaluation shows that the goal of reducing the accuracy of a traslationese detector to random chance is achieved.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not a missing reference, but the opposite: (Vaswani et al., 2017a) and (Vaswani et al., 2017b) are the same."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 077: \"bilingual (MT) or monolingual (ST)\": This is a bit confusing. Are the acronyms meant to refer to the techniques mentioned earlier? If so, it would be clearer to write something like \"bilingual (using NMT) or monolingual (using ST)\".\n- p.3: The caption of Figure 1 is missing the closing parenthesis. In addition, it would be best if in Figure 1 and its caption you use curly brackets to denote pairs, as you've done in the body of the paper. \n- 449: \"Transfomer\"\n- 533: \"translation. to rThis\"\n- 538: \"Bert-score-based\": Change to \"BERTScore-based\", to be consistent with the way you've written it elsewhere.\n- p.5, Equation 5: It seems to me that the semantic similarity loss you show here is the cosine one, and not the MSE one.\n- There are a few places (306, 438, 441) where you have a footnote reference right after a number, which is needlessly confusing as it makes it look like an exponent. I'd move the footnote reference to a nearby word.\n- Section 5.2 concerns the qualitative analysis, but I'd consider the TTR and LS metrics that you use to still be a quantitative analysis, not qualitative. The closest thing to a qualitative analysis is the manual analysis of the output.\n- Throughout the paper, I feel a term like \"native\" would be clearer and more explicit than \"original\". For instance, \"Our goal is to eliminate translationese signals from translated texts by transforming them into an native-like version\" (Section 3)."}}]

Author rebuttal:

Meta-review: The reviewers have praised this papers' technical contribution to the field and originality. They agree that the paper advances the state-of-the-art in machine translation particularly concerning the well-studied phenomenon of translationese. 

Finally, the reviewers also praised the paper style, writing, and clarity.
Decision: Accept-Main