A Call for Standardization and Validation of Text Style Transfer Evaluation
Year: 2023

Abstract:
Text Style Transfer (TST) evaluation is, in practice, inconsistent. Therefore, we conduct a meta-analysis on human and automated TST evaluation and experimentation that thoroughly examines existing literature in the field. The meta-analysis reveals a substantial standardization gap in human and automated evaluation. In addition, we also find a validation gap: only few automated metrics have been validated using human experiments. To this end, we thoroughly scrutinize both the standardization and validation gap and reveal the resulting pitfalls. This work also paves the way to close the standardization and validation gap in TST evaluation by calling out requirements to be met by future research.

heading: 1 Introduction
text: Text style transfer (TST) is the task of transferring text from one style to another. Examples include sentiment transfer (making a negative text more positive) (Shen et al., 2017), politeness transfer (Niu and Bansal, 2018), formality transfer (Rao and Tetreault, 2018), and many more. Anyone working and communicating with texts has most likely manually performed many of these transfer tasks countless times. Whether generalized (Reif et al., 2022) or task-specific models, evaluating their performance on TST is crucial to measure progress in task-specific text generation. In the last few years, there has been a surge in research on TST (see Figure 1, Appendix), the majority of which was in the last two years. When a field develops that quickly, the development of evaluation procedures often does not keep up.
TST research is now at a point where reflection on evaluation practices is becoming increasingly urgent, as progress in this field is becoming difficult if the performance of new methods cannot be reliably compared to existing methods. It can even become hard to determine the state of the art
one should aim to improve. However, recent TST surveys (Toshevska and Gievska, 2022; Jin et al., 2022; Hu et al., 2022) focus on summarizing TST methods and their evaluation but do not point out evaluation and experimentation issues. Works on TST evaluation either focus on getting more reliable metrics for a particular type of TST (Mir et al., 2019), focus on a specific evaluation aspect for validation (Yamshchikov et al., 2021), or emphasize only shortcomings of automated evaluation in a multilingual setting of one particular type of TST (Briakou et al., 2021a). As a first step, it is, therefore, necessary to examine evaluation practices and experimental setups to point out the need for standardization of both, which is what we set out to do in this paper.
Early TST works (Shen et al., 2017; Li et al., 2018) focus on human evaluation, which is still considered the most reliable (Briakou et al., 2021b). Many later publications rely only on automated metrics, which can be validated by correlating them with human evaluations. So far, no comprehensive overview exists showing which metrics are validated and how leading to researchers using popular metrics rather than those with the highest correlations. In our meta-analysis, we counter-act this by reporting which metrics are validated and analyzing cases where the same metric received different validation results. To sum up, our contributions are:
1. Our examination of TST methods and TST evaluation practices highlights the need for standardization (Section 2).
2. Our overview of automated TST evaluation reveals an unmet need for validation (Section 3).
We conclude with a list of requirements to help close these two gaps.
10791

heading: 2 A Meta-Analysis of Text Style Transfer
text: In this meta-analysis, we focus on TST publications in top-tier NLP and AI venues (see Appendix A.2 for selection details), resulting in 89 considered papers summarized in Table 7, 8, and 9. The dimensions of TST evaluation are fluency, content preservation, and style transfer strength, as well as other aspects. Therefore, we divide the automated and human evaluation subsections according to these aspects.

heading: 2.1 Automated Evaluation
text: For automated evaluation, we aggregate the findings of our meta-analysis in Table 1 (with a detailed description of conventions in Appendix A.4.1). Overall, 21/89 papers use only automated evaluation, and 33/89 use at least one metric that has not been validated. Statistically significant results are reported for only 3/89 papers.
Fluency We find a total of 34 different setups to measure fluency automatically. 45/89 papers rely on automated metrics for measuring fluency. The by far most common metric (see Table 4) is the Perplexity (PPL) of a Language Model (LM). However, the pre-training procedure of the LM varies. It is either trained on all styles of a dataset (dos Santos et al., 2018; Dai et al., 2019; John et al., 2019; Cheng et al., 2020), trained on each style separately (Yang et al., 2018; Gong et al., 2019), pre-trained on external datasets (Logeswaran et al., 2018; Jain
et al., 2019), or trained on the dataset and the TST output (backward/forward) (Zhao et al., 2018; Huang et al., 2020). 19/89 papers do not mention how the LM is trained. These numbers show the need for consistency.
Content Preservation We find 66 papers using a total of 35 different automated metrics to measure content preservation. Only 15 of these metrics have been validated.
The most common metrics (see Table 5) to measure content preservation are Source-BLEU and Ref-BLEU (applying BLEU (Papineni et al., 2002) to the input/output, reference/output respectively). However, as pointed out by Mir et al. (2019), Source-BLEU only measures n-gram overlaps and does not consider that a change of the sentence is necessary to change the style. This results in contradictory validation results (see Section 3). Post (2018) has shown that the reported BLEU scores heavily depend on preprocessing and several parameters (e.g., number of references, length penalty, maximum n-gram length, smoothing applied to 0-count n-grams). Works using BLEU for evaluation need this specification to make the results consistent and reproducible. In our meta-analysis, we could not find a single TST paper that specifies BLEU sufficiently (including above mentioned details). Ref-BLEU is the second most popular method to measure content preservation. In addition to the general problems with BLEU, we see two more shortcomings. On the one hand, the needed reference sentences are not available for all datasets. On the other hand, calculating BLEU scores between the output and multiple human references can improve its reliability and, thankfully, for the arguably most popular TST dataset—Yelp— Jin et al. (2019) and Luo et al. (2019) introduced additional reference sentences. These, however, are only used by 6/30 papers applying Ref-BLEU on Yelp.
Style Transfer Accuracy For automated evaluation of style transfer accuracy, we find 23 evaluation metrics, of which 14 have been validated. Overall, 77 papers use automated style transfer accuracy metrics. Table 6 provides an overview of the most common automated text style transfer strength metrics. The top two are TextCNN (Kim, 2014) and fastText (Joulin et al., 2017).
Other 34/89 papers measure a fourth aspect. For 29/89 it is an overall metric, and 8/89 measure
another aspect of TST. We find eight validated metrics to measure overall performance. None of the metrics for other aspects have been validated.

heading: 2.2 Human Evaluation
text: For human evaluation, we aggregate the findings of our meta-analysis in Table 2 (with a detailed description of our conventions in Appendix A.3). Overall, 68/89 papers conduct a human evaluation. However, only a tiny fraction of 2/68 papers detect a statistically significant difference in model performance. Most papers fail to include a statistical test, be it a statistical power analysis before conducting the human evaluations or a statistical significance test afterward. This is a common problem in the NLP community (Card et al., 2020).
Releasing human evaluations is also relatively uncommon. Only 5/68 papers publish their human experimentation results. However, releasing them would facilitate reproducibility and the validation of new automated metrics. Reproducibility of human TST evaluation is a challenge, as is reproducibility of human evaluation in general (Belz et al., 2020). 67/68 papers conducting human evaluations have no adequate description. We consider the description of human evaluation to be adequate if the following is specified: annotators’ background, the number of annotators, the number of annotators per sample, incentivization, data selection, questionnaire design, and rating scale (Briakou et al., 2021b). With published human evaluations, one could also easily estimate the parameters of a simulation-based power analysis as done by Card et al. (2020).
Overall, there is still room for improvement despite previous calls for standardization (Briakou et al., 2021b). For human evaluation, we find a total of 24 different evaluation schemes (viewed on a high level whether each aspect is present and
evaluated relatively or absolutely, not considering different rating scales and questionnaire designs).

heading: 2.3 Experimentation
text: Aspect Count Multiple runs 5/89 Reproducibility
Code provided 56/89 Evaluation code provided 42/89
Preprocessing specified 38/89

heading: 3.1 Fluency

heading: 3 Automated Metrics and Their (Missing) Validation
text: between sentence-level Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) LM PPL and human fluency evaluations and conclude that LM PPL is an inappropriate metric to measure fluency in the TST setting. On the other hand, Pang and Gimpel (2019) show a high correlation but also note that PPL is not equal to fluency. Briakou et al. (2021a) report a relatively low correlation for the PPL of a 5-gram KenLM (Heafield, 2011) with human evaluations and slightly higher correlations for Pseudo Log-Likelihoods (PLL) of BERT (Devlin et al., 2019), and XLM (Conneau et al., 2020). Overall, LM PPL for TST evaluation has only been validated on a fraction of the deployed TST datasets, namely Yelp[s] and GYAFC. Previous work is divided as to whether and to what extent this metric correlates with human evaluations. As reported in Section 2, many different architectures, training methods, and application methods of LMs for TST fluency evaluation exist. However, as a simplification, we assume the LM PPL to be a validated fluency metric.
Other approaches to evaluate fluency are based on BLEU. In contrast to Luo et al. (2019), Li et al. (2018) show no significant correlation for Ref-BLEU. Pryzant et al. (2020) report a low correlation of the Source-BLEU score with human evaluations. Wu et al. (2020) and Rao and Tetreault (2018) report moderate correlation for their metric (Heilman et al., 2014). Pryzant et al. (2020) show that their style classifier correlates more with human evaluations of fluency than the style transfer strength. None of these metrics is deployed by more than two papers.

heading: 3.2 Content Preservation
text: There are two extensive studies for automated content preservation metrics by Mir et al. (2019) and Yamshchikov et al. (2021). However, both have limited scope: Mir et al. (2019) only report scores where style words have been masked or removed (not done by any other paper). Yamshchikov et al. (2021) report correlations only on the datasets themselves and not on actual model outputs. Both report Word Mover’s Distance (WMD) (Kusner et al., 2015) having the highest correlation, outperforming Source-BLEU, other embedding-based metrics (such as also investigated by Fu et al. (2018) and Pang and Gimpel (2019)), and chrF (Popović, 2015).
Cao et al. (2020) find high correlation for SourceBLEU, whereas Pryzant et al. (2020) find low correlation. For Ref-BLEU, Li et al. (2018), Luo et al. (2019), and Cao et al. (2020) show a high, and Xu et al. (2012) a low correlation, whereas Briakou et al. (2021a) investigate Ref-BLEU and SourceBLEU among others, but show chrF having the highest correlation. The suitability of BLEU for TST evaluation remains questionable as Mir et al. (2019) point out that BLEU cannot capture whether words were changed on purpose.

heading: 3.3 Style Transfer Strength
text: Style transfer strength is usually evaluated by applying a sentence classifier trained to classify the output sentences by style. Early work (Xu et al., 2012) compares several metrics showing the highest correlation for Logistic Regression (LR). The two most popular methods nowadays are TextCNN (Kim, 2014) and fastText (Joulin et al., 2017). Luo et al. (2019) show high correlation for TextCNN, Pang and Gimpel (2019) for fastText, whereas Mir et al. (2019) validate both, showing slightly better correlations for TextCNN. Li et al. (2018) validate a Bi-directional LSTM for style classification and note that the correlation to human evaluations highly depends on the dataset. Rao and Tetreault (2018) report a moderate correlation to their style transfer strength metric. Also, Mir et al. (2019) show that the Earth Mover’s Distance (EMD) in combination with TextCNN or fastText has a higher correlation with human evaluations than only TextCNN or fastText, but no other paper uses it.

heading: 3.4 Other
text: Niu and Bansal (2018) report a high correlation of Source-BLEU with overall dialogue quality. Rao and Tetreault (2018) report a relatively high correlation for Ref-BLEU compared to TERp and PINC with the overall human evaluations. This is in agreement with Wang et al. (2020). Wu et al. (2020) perform dataset-dependent studies and found no significant correlation between automated metrics and human scores.

heading: 3.5 Metrics Validated for Multiple Aspects
text: Some metrics, such as Ref-BLEU, Source-BLEU, PINC, and embedding-based metrics, are validated for multiple aspects. However, Ref-BLEU shows the highest correlation as an overall metric only for Rao and Tetreault (2018) (also outperforming
PINC), otherwise (Xu et al., 2012; Li et al., 2018; Luo et al., 2019; Cao et al., 2020; Wang et al., 2020; Briakou et al., 2021a), there is no clear picture. For Source-BLEU (Niu and Bansal, 2018; Mir et al., 2019; Pryzant et al., 2020; Cao et al., 2020; Yamshchikov et al., 2021; Briakou et al., 2021a) and also for embedding-based metrics (Xu et al., 2012; Fu et al., 2018; Mir et al., 2019; Pang and Gimpel, 2019; Wu et al., 2020; Yamshchikov et al., 2021; Briakou et al., 2021a), we find mixed results across the different aspects.

heading: 4 Conclusion & Future Work
text: Our research emphasizes the pressing need for standardization and validation in TST. While human evaluation is still considered to be the most reliable, it is expensive and time-consuming, hindering reproducibility. Many publications use automated metrics as surrogates, but only a fraction of them is validated. Furthermore, human and automated evaluations and experimental setups lack standardization, making it difficult to compare the performance of different models. Summarizing our results, we pose the following six requirements to be met by future TST research:
1. It needs to use validated metrics (see Table 10), focusing on those showing the highest correlations with human evaluations
2. In experiments, multiple runs need to be performed on different random seeds, reporting mean and standard deviation.
3. A statistical significance test needs to be performed on the results of automated metrics.
4. If a human evaluation is done, a statistical power analysis is necessary in advance, and all human evaluation details need to be published as suggested by Briakou et al. (2021b).
5. To improve reproducibility, researchers should always specify the preprocessing pipeline and publish their code (including evaluation code).
6. A comparison with state-of-the-art methods on the validated metrics is called for.
To help with these requirements, we plan a largescale experimental comparison to rank existing methods according to validated metrics in the future and examine how automated metrics generalize from one TST task to the other. We also plan
to (re-)validate existing automated metrics to help meet the first requirement.

heading: Limitations
text: The present work only points out problems of existing research and presents no final solutions. We also simplify the assumption that an automated metric validated for a specific TST task generalizes to other tasks. However, this is problematic since there is, to our knowledge, no investigation of whether a validation on one task generalizes. This concern is motivated by the fundamental differences in how different TST tasks are defined. There are several different definitions, such as datadriven TST (e.g., sentiment transfer) and linguistically motivated TST (e.g., formality transfer) (Jin et al., 2022). Also, we consider only TST papers (no text simplification) and focus on top-tier NLP and AI venues (non-workshop).

heading: Acknowledgements
text: The authors acknowledge support by the CarlZeiss Foundation, the BMBF awards 01|S20048, 03|B0770E, and 01|S21010C, and the DFG awards BU 4042/2-1, BU 4042/1-1, KL 2698/2-1, KL 2698/5-1, KL 2698/6-1, and KL 2698/7-1.

heading: A Meta-Analysis
text: This section describes the details of the conducted meta-analysis on TST evaluation.

heading: A.1 Popularity of TST
text: TST has become a popular topic in research. When Searching for "text style transfer" on dblp. uni-trier.de (accessed Jan 18, 2023), we can find a steep increase in the number of publications per year (see Figure 1).

heading: A.2 Paper Selection
text: We consider the papers listed in a recent TST survey (Jin et al., 2022) 1 with the addition of more recent publications. We consider only TST papers (no text simplification) and focus on top-tier NLP and AI venues (non-workshop) with the following statistics: ACL:25, EMNLP:17, NAACL:9, COLING:8, AAAI:7, NeurIPS:5, ICML:4, ICLR:3, IJCAI:3, ECIR:2, EACL:1, CVPR:1, NLPCC:1, USENIX:1, ICAART:1, Royal Society Open Science (journal):1 resulting in 89 considered papers where 60 were already surveyed by Jin et al. (2022).

heading: A.3 Human Evaluation
text: For human evaluation, we approach the metaanalysis with the following conventions used in Tables 7, 8, and 9.
• In the Statistical Analysis, Evaluations Released, and Adequately Specified columns, n/a refers to publications that did not conduct a human evaluation
• Statistical Analysis: whether either a statistical significance test on the obtained results or a power analysis in advance to estimate a sufficient number of evaluations to detect the hypothesized effect size is done
• Evaluations Released: whether the human evaluations have been released and are publicly available
• Adequately Specified: If the authors specify annotators’ background, number of annotators, number of annotators per sample, incentivization, data selection, questionnaire design, rating scale
1https://github.com/fuzhenxin/ Style-Transfer-in-Text
• Abbreviations (categorization according to Briakou et al. (2021b)) for fluency, content, style, and other: A=Absolute rating of the model output (independent from the output from other models, also includes ratings where the output sentence was rated relative to the input sentence), R=Relative to other TST models’ output (e.g., ranking)
• Other: separate question; we do not consider combinations (e.g., geometric means) of other metrics as other

heading: A.4 Automated Evaluation

heading: A.4.1 Conventions
text: For automated evaluation, we approach the metaanalysis with the following conventions used in Tables 7, 8, 9, and 10.
• We add the automated metric to the evaluation dimension that the authors claim it measures
• Statistical Significance: Yes, if authors conduct a statistical significance test for their claimed results
• Fluency
– LM based on LSTM (Hochreiter and Schmidhuber, 1997) (LSTM-LM)
– LM based on Gated Recurrent Units (Cho et al., 2014) (GRU-LM)
– LM based on Recurrent Neural Networks (RNN-LM) (Zhao et al., 2018)
– GPT-2(Radford et al., 2019) – KenLM (Heafield, 2011) – RoBERTa (Liu et al., 2019) – Statistical Grammaticality Predictor
(SGP) (Heilman et al., 2014) – Kneser LM (Kneser and Ney, 1995) – Transformer (Vaswani et al., 2017)
• Content
– Cosine similarity of sentence embeddings (Cosine Sim)
– Cosine distance of sentence embeddings (Cosine Dist)
– Logistic Regression (LR) – BLEU (Papineni et al., 2002)
* Source-BLEU: Comparing output to source sentences
* Ref-BLEU: Comparing output to reference sentences, number of references (if more than 1) in brackets
– chrF (Popović, 2015) – BERT (Devlin et al., 2019) – METEOR (Banerjee and Lavie, 2005) – BLEURT (Sellam et al., 2020) – COMET (Rei et al., 2020) – ROUGE (Lin, 2004) – CIDEr (Vedantam et al., 2015) – BERTScore (Zhang et al., 2020a) – CNN Similarity Measure (CNN-SM)
(He et al., 2015) – Hu Sentiment Classifier (HSC) (Hu et al.,
2016) – Davidson Classifier (DC) (Davidson
et al., 2017) – Manning Classifier (MC) (Manning
et al., 2014)
• Style
– All models are classifiers if not differently specified
– TextCNN (Kim, 2014) – fastText (Joulin et al., 2017) – Word Mover’s Distance (WMD) based
on Earth Mover’s Distance (EMD) (Kusner et al., 2015)
– Pretrained Transformer PT (Wolf et al., 2020)
– Formality Classifier (FC) (Pavlick and Tetreault, 2016)
• Other
– We do not consider combinations (e.g., geometric means) of other metrics as other
– GLEU (Napoles et al., 2015) – PINC (Chen and Dolan, 2011) – Sentence Sim (Wieting et al., 2019) – Flesch-Kincaid readability index (Kin-
caid et al., 1975) – Li Diversity (LD) (Li et al., 2016a)

heading: A.4.2 Ranking
text: We provide an overview of the most common automated metric setups for fluency in Table 4, for content preservation in Table 5, and for style transfer strength in Table 6.

heading: A.5 Experimentation
text: For experimentation, we approach the metaanalysis with the following conventions used in Tables 7, 8, 9, and 10.
• Datasets
– Amazon (He and McAuley, 2016) – Beer reviews (McAuley et al., 2012) – Bible (Carlson et al., 2018) – Blogs (Schler et al., 2006) – Caption (Li et al., 2018) – DIAL (Blodgett et al., 2016) – Europarl (Koehn, 2005) – FlickrStyle10K (Gan et al., 2017) – Gender (Reddy and Knight, 2016) – Gigaword (Napoles et al., 2012)
Style Count TextCNN 24 fastText 13 Classifier 10 BERT 6 LSTM 5 Bi-LSTM 3 CNN 3 GRU 3 LR 2 HSC 2 RoBERTa 2

heading: A.6 Summary Tables
text: 4https://www.yelp.com/dataset/challenge
H um
an E
va lu
at io
n A
ut om
at ed
E va
lu at
io n
E xp
er im
en ts
So ur
ce St
at is
tic al
A na
ly si
s E
va lu
at io ns R el ea se d
A de
qu at
el y
Sp ec
ifi ed
Fl ue
nc y
C on
te nt
St yl
e O
th er
St at
is tic al Si gn ifi ca nc
e Fl
ue nc
y C
on te
nt St
yl e
O th
er D
at as
et s
>1 R
un C
od e Pr ov
id ed
E va
l. C
od e
Pr ov
id ed
Pr ep r. Sp ec ifi
ed
X u
et al
.( 20
12 )
N o
N o
N o
A A
A O
ve ra
ll A
N o
- -
C os
in e
si m
,L M
,L R
Se m
an tic
A de
qu ac
y: R
ef -
B LE
U ;L
ex ic
al D
is si
m ila rity :P IN C
Sh ak
es pe
ar e
N o
N o
N o
N o
L ie
ta l.
(2 01
6b )
N o
N o
N o
- -
- C
on si
st en cy R
N o
- -
- O
ve ra
ll: R
ef -B
L E
U Tw
itt er
Pe rs
on a,
Tw itt
er So
rd on
i, T V Se ri es Tr an sc ri pt s
N o
Y es
N o
N o
M ue
lle re
ta l.
(2 01
7) n/
a n/
a n/
a -
- -
- Y
es -
- -
O ve
ra ll:
L M
L og
L ik
el i-
ho od
B ee
rr ev
ie w
s, Sh
ak es
pe ar
e Y
es N
o N
o N
o
H u
et al
.( 20
17 )
n/ a
n/ a
n/ a
- -
- -
N o
- -
H SC
- IM
D b,
SS T
N o
Y es
Y es
N o
G an
et al
.( 20
17 )
N o
N o
N o
- -
R -
N o
- -
- O
ve ra
ll: So
ur ce
-B L
E U , M E T E O R , R O U G E , C ID E r
Fl ic
kr St
yl e1
0K ,Y
ou tu
be 2t
ex t
N o
N o
N o
Y es
Sh en
et al
.( 20
17 )
N o
N o
N o
A -
A O
ve ra
ll R
N o
- -
Te xt
C N
N -
Y el
p[ s]
N o
Y es
Y es
N o
H an
et al
.( 20
17 )
n/ a
n/ a
n/ a
- -
- -
N o
- -
- O
ve ra
ll: R
ef -B
L E
U Sh
ak es
pe ar
e N
o N
o N
o N o Fu et al .( 20 18 ) N o N o N o - A - - N o - C os in e D is t L ST M - A m az on ,P ap er -N ew s Y es Y es Y es
Y es
L ie
ta l.
(2 01
8) N
o Y
es N
o A
A A
- N
o -
R ef
-B L
E U
B i-
L ST
M -
Y el
p[ s]
,A m
az on
,C ap
tio ns
N o
Y es
Y es
N o
X u
et al
.( 20
18 )
N o
Y es
N o
- A
A -
N o
- So
ur ce
-B L
E U
Te xt
C N
N -
Y el
p[ l]
,A m
az on
N o
Y es
Y es
N o
Pr ab
hu m
oy e
et al
.( 20
18 )
N o
N o
N o
A A
- -
N o
- -
C N
N -
G en
de r,
Po lit
ic al
Sl an
t, Y
el p[
s] N
o Y
es Y
es N o do s Sa nt os et al .( 20 18 ) n/ a n/ a n/ a - - - - N o LS TM -L M PP L on da ta se t C os in e D is t D C - Tw itt er ,R ed di t N o N o N o
Y es
L ia
o et
al .(
20 18
) N
o N
o N
o A
A -
- N
o -
- Te
xt C
N N
,M C
- Y
el p[
s] N
o Y
es N
o N o Z ha ng et al .( 20 18 b) N o N o N o A A A - N o - So ur ce -B L E U Te xt C N N - Y el p[ l] N o Y es Y es
Y es
Z ha
o et
al .(
20 18
) N
o N
o N
o A
A A
- N
o R
N N
-L M
PP L
fo r-
w ar
d/ ba
ck w
ar d
So ur
ce -B
L E
U fa
st Te
xt -
Y el
p[ s]
,Y ah
oo N
o Y
es Pa
rt ia
lly N
o
R ao
an d
Te tr
ea ul
t( 20
18 )
N o
N o
N o
A A
A O
ve ra
ll R
N o
SG P
C N
N -S
M FC
O ve
ra ll:
R ef
-B L
E U , PI N C ,T E R p
G YA
FC N
o N
o N
o Y
es
Z ha
ng et
al .(
20 18
a) n/
a n/
a n/
a -
- -
- Y
es -
- C
la ss
ifi er
O ve
ra ll:
R O
U G
E G
ig aw
or d
Y es
N o
N o
Y es
L og
es w
ar an
et al
.( 20
18 )
N o
N o
N o
A A
A -
N o
L M
PP L
on ex
te rn al da ta se t O
w n
ba se
d on
B LE
U ,R
ef -
B L
E U
H SC
- Y
el p[
s] ,I
M D
b, Sh
ak es
pe ar
e N
o N
o N
o N
o
C he
n et
al .(
20 18
) N
o N
o N
o A
A A
- N
o -
- Te
xt C
N N
O ve
ra ll:
R ef
-B L
E U
; D ive rs ity :S ou rc eB L E U
Y el
p[ s]
N o
Y es
Y es
Y es
Y an
g et
al .(
20 18
) n/
a n/
a n/
a -
- -
- N
o L
M PP
L on
da ta
se t
pe r
st yl
e So
ur ce
-B L
E U
Te xt
C N
N O
ve ra
ll: R
ef -B
L E
U Y
el p[
s] N
o N
o N
o Y
es
C ar
ls on
et al
.( 20
18 )
n/ a
n/ a
n/ a
- -
- -
N o
- -
- O
ve ra
ll: R
ef -B
L E
U ;D iff fr om so ur ce :P IN
C B
ib le
N o
Y es
Y es
Y es
N iu
an d
B an
sa l(
20 18
) N
o N
o N
o -
- A
O ve
ra ll
A N
o -
So ur
ce -B
L E
U SV
M ,C
N N
,L ST
M -C
N N
- St
an fo
rd Po
lit en
es s
C or
pu s,
M ov ieT ri pl es N
o Y
es Y
es Y
es
G uu
et al
.( 20
18 )
N o
N o
N o
A A
- Pl
au si
bi lit y A
N o
- -
- -
Y el
p[ l]
,B ill
io nw
or d
N o
Y es
N o
N o
Sh et
ty et
al .(
20 18
) N
o N
o N
o -
A -
- N
o -
M E
T E
O R
L ST
M -
B lo
gs ,O
w n
N o
N o
N o
N o
Ja in
et al
.( 20
19 )
N o
N o
N o
- -
- R
ea da
bi lit y R
N o
4- gr
am K
en L
M PP
L on
ex te
rn al
da ta
se t
C os
in e
Si m
- O
ve ra
ll: R
ef -B
L E
U ;
R ea
db ili
ty :
Fl es
ch -
K in
ca id
re ad
ab ili
ty in
de x
O w
n N
o Y
es Y
es N
o
D ai
et al
.( 20
19 )
N o
N o
N o
R R
R -
N o
5- gr
am K
en L
M PP
L on
da ta
se t
So ur
ce -B
L E
U fa
st Te
xt O
ve ra
ll: R
ef -B
L E
U Y
el p[
s] ,I
M D
b N
o Y
es Y
es N
o
Jo hn
et al
.( 20
19 )
N o
N o
N o
A A
A -
N o
3- gr
am L
M PP
L on
da ta
se t
C os
in e
Si m
,w or
d ov
er la
p Te
xt C
N N
- Y
el p[
s] ,A
m az
on N
o Y
es Y
es N
o
K aj
iw ar
a (2
01 9)
n/ a
n/ a
n/ a
- -
- -
N o
- -
- O
ve ra
ll: R
ef -B
L E
U , F1 on ad de d, ke pt an d de le te d w or ds
G YA
FC N
o N
o N
o Y
es
W u
et al
.( 20
19 a)
N o
N o
N o
A A
A -
N o
- R
ef -B
L E
U Te
xt C
N N
- Y
el p[
s] ,A
m az
on N
o Y
es Y
es N o Ji n et al .( 20 19 ) N o N o N o A A A - N o L ST M -L M PP L R ef -B L E U [5 ,4 ] Te xt C N N - Y el p[ s] ,G YA FC N o N o N o N o L ie ta l. (2 01 9) N o N o N o R R R O ve ra ll R N o - R ef -B LE U ,S ou rc eB LE U Te xt C N N - IM D b, G Y A FC ,A m az on ,Y el p[ s] ,Y aho o N o Y es Y es N o
Ta bl
e 7:
Sh ow
n ar
e th
e pa
pe rs
co ns
id er
ed fo
rt he
m et
aan
al ys
is .
H um
an E
va lu
at io
n A
ut om
at ed
E va
lu at
io n
E xp
er im
en ts
So ur
ce St
at is
tic al
A na
ly si
s E
va lu
at io ns R el ea se d
A de
qu at
el y
Sp ec
ifi ed
Fl ue
nc y
C on
te nt
St yl
e O
th er
St at
is tic al Si gn ifi ca nc
e Fl
ue nc
y C
on te
nt St
yl e
O th
er D
at as
et s
>1 R
un C
od e Pr ov
id ed
E va
l. C
od e
Pr ov
id ed
Pr ep r. Sp ec ifi
ed
Sh an
g et
al .(
20 19
) N
o N
o N
o A
A A
- N
o -
R ef
-B L
E U
Te xt
C N
N O
ve ra
ll: G
L E
U G
YA FC
,O w
n N
o N
o N
o Y es Su dh ak ar et al .( 20 19 ) N o N o N o R R - - N o G PT -2 PP L on da ta se tp er st yl e So ur ce -B L E U fa st Te xt O ve ra ll: G L E U Y el p[ s] , A m az on , C ap tio ns , Po lit ic al Sl an t, G en de r N o Y es Y es N o W an g et al .( 20 19 b) n/ a n/ a n/ a - - - - N o - B E R T fin etu ne d w ith ST S L ST M O ve ra ll: R ef -B L E U , PI N C G YA FC N o Y es Pa rt ia lly Y es Ti kh on ov et al .( 20 19 ) n/ a n/ a n/ a - - - - N o - So ur ce -B L E U C la ss ifi er O ve ra ll: R ef -B L E U Y el p[ s] Y es Y es Y es N o L ee ft in k an d Sp an ak is (2 01 9) N o N o N o - - A O ve ra ll A N o - - A tte nt io nR N N - IM D b2 ,R ot te n To m at oe s re vi ew s N o N o N o Y es L am pl e et al .( 20 18 ) N o N o N o A A A O ve ra ll R N o 5- gr am K ne se rL M PP L R ef -B LE U ,S ou rc eB LE U fa st Te xt - Y el p[ s] ,Y el p[ l] ,A m az on N o Y es Y es Y es L uo et al .( 20 19 ) N o N o N o A A A - N o - R ef -B L E U [4 ,4 ] Te xt C N N - Y el p[ s] ,G YA FC N o Y es Y es N o W u et al .( 20 19 b) N o N o N o A A A - N o - R ef -B L E U B iL ST M - Y el p[ s] ,A m az on N o N o N o N o Fu et al .( 20 19 ) n/ a n/ a n/ a - - - - N o - - L R ,T ex tC N N ,p iv ot cl as - si fie r - Y el p[ s] , A m az on , C ap tio ns , Po lit ic al Sl an t, G en de r, Pa pe rN ew s N o Y es Y es N o (M ir et al ., 20 19 ) N o Y es N o A A A - N o L ST M -L M PP L ,u ni gr am an d ne ur al lo gi st ic re gr es - si on cl as si fie rs as ad ve rs ar - ia ls St yl e re m ov al ,s ty le m as kin g w ith So ur ce -B L E U , M E T E O R ,e m be dd in g av - er ag e, gr ee dy m at ch in g, ve ct or ex tr em a, W M D Te xt C N N ,f as tT ex t - Y el p[ s] N o Y es Y es N o R om an ov et al .( 20 19 ) n/ a n/ a n/ a - - - - N o L M PP L on da ta se t C os in e Si m G R U - Pa pe rN ew s, Sh ak es pe ar e N o Y es N o N o G on g et al .( 20 19 ) N o N o N o A A A - N o G R U -L M PP L on da ta se t pe rs ty le C os in e D is t L ST M - Y el p[ s] ,G YA FC N o N o N o N o W an g et al .( 20 19 a) N o N o N o A A A - N o L M PP L on da ta se t R ef -B L E U fa st Te xt - Y el p[ s] ,A m az on ,C ap tio ns N o Y es Pa rt ia lly N o L ie ta l. (2 02 0c ) n/ a n/ a n/ a - - - - N o - So ur ce -B L E U C la ss ifi er - Pe rs on al ity ca pt io ni ng ,F lic kr St yl e1 0K , Y ah oo ,Y el p[ s] N o N o N o N o L iu et al .( 20 20 ) N o N o N o A A A - N o 3- gr am K ne se r L M PP L on da ta se t w or d ov er la p, no un ov er - la p B iL ST M O ve ra ll: R ef -B L E U -2 Y el p[ s] ,A m az on N o Y es Y es Y es Pr yz an te ta l. (2 02 0) Y es N o N o A A A - Y es - - - O ve ra ll: So ur ce -B L E U , C la ss ifi er O w n, IB C , N ew s he ad lin es , Tr um p sp ee ch es N o Y es Y es N o Sy ed et al .( 20 20 ) n/ a n/ a n/ a - - - - N o - So ur ce -B L E U , R O U G E - 1, R O U G E -2 ,R O U G E -3 , R O U G E -L St yl is tic A lig nm en t( O w n) - G ut en be rg , O pi no si s, W ik ip ed ia 2, Sh ak es pe ar e Y es N o N o N o W u et al .( 20 20 ) N o N o N o A - A O ve ra ll R N o SG P - G R U O ve ra ll: So ur ce -B L E U , B LE U -2 ,E m be dd in g A vg , E m be dd in g E xt re m a, E m - be dd in g G re ed y M T FC ,T C FC N o Y es Y es Y es C ao et al .( 20 20 ) N o N o N o - A - - N o B E R T on da ta se tp er st yl e So ur ce -B LE U ,R ef -B LE U fa st Te xt - Si m pW ik i, M SD N o N o N o Y es M ad aa n et al .( 20 20 ) N o N o N o A A A - N o - So ur ce -B L E U L ST M O ve ra ll: R ef -B L E U , R O U G E ,M E T E O R Y el p[ s] , A m az on , G en de r, Po lit ic al Sl an t N o Y es Pa rt ia lly Y es Z ha ng et al .( 20 20 b) N o N o N o A A A - N o - - - O ve ra ll: R ef -B L E U G YA FC ,O w n N o N o N o N o Z ho u et al .( 20 20 ) N o N o N o A A A - N o - R ef -B L E U [4 ] Te xt C N N - Y el p[ s] ,G YA FC N o Y es Y es N o Ji n et al .( 20 20 ) N o N o N o A A R A ttr ac tiv e. A N o G PT -2 PP L - - O ve ra ll: So ur ce -B L E U , M E T E O R , R O U G E , C ID E r O w n N o Y es Pa rt ia lly N o D ua n et al .( 20 20 ) N o N o N o - A A - N o - - Te xt C N N D iv er si ty :L D Y el p[ s] ,P ap er -N ew s N o Y es Y es Y es Tr an et al .( 20 20 ) N o N o N o A A - - N o G PT -2 PP L So ur ce -B L E U ,M E T E O R , R O U G E C la ss ifi er - O w n N o N o N o Y es H ua ng et al .( 20 20 ) N o N o N o A A A - N o R N N -L M PP L fo rw ar d/ ba ck w ar d So ur ce -B L E U fa st Te xt - Y el p[ s] ,Y ah oo N o N o N o N o W an g et al .( 20 20 ) Y es N o N o - - - O ve ra ll A N o - B E R T fin etu ne d w ith ST S G R U O ve ra ll: R ef -B L E U [4 ], PI N C G YA FC N o Y es Pa rt ia lly N o K im an d So hn (2 02 0) n/ a n/ a n/ a - - - - N o 3- gr am L M PP L on da ta se t So ur ce -B L E U fa st Te xt - Y el p[ s] ,A m az on N o Y es Y es N o Ja fa ri ta ze hj an ie ta l. (2 02 0) N o N o N o A R A - N o G R U -L M PP L on da ta se t C os in e Si m Te xt C N N - Y el p[ s] N o N o N o Y es Sa nc he ti et al .( 20 20 ) N o N o N o - A A - N o - So ur ce -B L E U Te xt C N N - G YA FC ,Y el p[ l] ,S ha ke sp ea re N o N o N o N o Pa nt et al .( 20 20 ) n/ a n/ a n/ a - - - - N o - So ur ce -B L E U C la ss ifi er - Y el p[ s] ,Y el p[ l] N o N o N o N o C ha kr ab ar ty et al .( 20 20 ) N o N o N o - A A O ve ra ll A N o - R ef -B L E U ,B E R T Sc or e - N ov el ty :O w n O w n N o Y es Y es Y es
Ta bl
e 8:
Sh ow
n ar
e th
e pa
pe rs
co ns
id er
ed fo
rt he
m et
aan
al ys
is .
H um
an E
va lu
at io
n A
ut om
at ed
E va
lu at
io n
E xp
er im
en ts
So ur
ce St
at is
tic al
A na
ly si
s E
va lu
at io ns R el ea se d
A de
qu at
el y
Sp ec
ifi ed
Fl ue
nc y
C on
te nt
St yl
e O
th er
St at
is tic al Si gn ifi ca nc
e Fl
ue nc
y C
on te
nt St
yl e
O th
er D
at as
et s
>1 R
un C
od e Pr ov
id ed
E va
l. C
od e
Pr ov
id ed
Pr ep r. Sp ec ifi
ed
K ri
sh na
et al
.( 20
20 )
N o
N o
N o
A A
- -
N o
R oB
E R
Ta on
ex te
rn al
da ta
se t
So ur
ce -B
L E
U R
oB E
R Ta
- G
YA FC
,S ha
ke sp
ea re
N o
Y es
Y es
Y es
M a
et al
.( 20
20 )
N o
N o
N o
A R
R -
N o
Pr et
ra in
ed G
PT PP
L B
E R
T Sc
or e
C la
ss ifi
er R
ep et
iti ve
ne ss
: O
w n;
D i-
ve rs
ity :O
w n
R O
C ,P
ar ap
hr as
e co
rp us
N o
N o
N o
Y es
M al
m ie
ta l.
(2 02
0) n/
a n/
a n/
a -
- -
- N
o -
So ur
ce -B
L E
U B
E R
T -
Y el
p[ s]
N o
N o
N o
Y es
C he
ng et
al .(
20 20
) N
o N
o N
o -
R R
C on
te xt
C on
si s-
te nc
y R
N o
LS TM
-L M
PP L
on da
ta se
t R
ef -B
L E
U Te
xt C
N N
O ve
ra ll:
G L
E U
G YA
FC ,Y
ah oo
,O w
n N
o N
o N
o Y
es
L ie
ta l.
(2 02
0b )
n/ a
n/ a
n/ a
- -
- -
N o
- R
ef -B
LE U
,S ou
rc e-
B LE
U fa
st Te
xt -
Y el
p[ s]
,I M
D b
N o
N o
N o
Y es
D at
ha th
ri et
al .(
20 20
) N
o N
o N
o A
- R
- N
o G
PT PP
L -
C la
ss ifi
er D
iv er
si ty
:L D
n/ a
N o
Y es
Y es
n/ a
H e
et al
.( 20
20 )
n/ a
n/ a
n/ a
- -
- -
N o
LS TM
-L M
PP L
on da
ta se t pe rs ty le R
ef -B
LE U
,S ou
rc e-
B LE
U Te
xt C
N N
- Y
el p[
s] ,G
YA FC
N o
Y es
Y es
Y es
X u
et al
.( 20
20 )
n/ a
n/ a
n/ a
- -
- -
N o
G PT
-2 PP
L So
ur ce
-B L
E U
C N
N O
ve ra
ll: G
L E
U Y
el p[
s] ,A
m az
on N
o Y
es Pa
rt ia
lly Y es Y ie ta l. (2 02 0) N o N o N o A A A - N o 5- gr am K en L M PP L R ef -B L E U ,C os in e Si m B E R T - Y el p[ s] ,G YA FC ,O w n N o Y es Pa rt ia lly Y es L ee (2 02 0) N o N o N o A A A - N o G PT -1 PP L ,G PT -2 PP L R ef -B L E U [2 ], So ur ce - B L E U ,B E R T Sc or e Te xt C N N - Y el p[ s] ,A m az on N o Y es Y es Y es L ie ta l. (2 02 0a ) N o N o N o A A - - N o G PT -2 PP L So ur ce -B L E U R oB E R Ta - G YA FC N o Y es Y es Y es Ly u et al .( 20 21 ) N o Y es N o A A A - N o - - - O ve ra ll: So ur ce -B L E U , M E T E O R , R O U G E , C ID E r O w n N o Y es Pa rt ia lly Y es L iu et al .( 20 21 ) N o N o N o A A A - N o G PT -2 PP L So ur ce -B LE U ,R ef -B LE U fa st Te xt - Y el p[ s] ,A m az on ,I M D b N o Y es Y es Y es G oy al et al .( 20 21 ) N o N o N o A A A O ve ra ll A N o Tr an sf or m er PP L on da ta se t So ur ce -B LE U ,R ef -B LE U fa st Te xt - G YA FC ,I M D b N o N o N o N o B ri ak ou et al .( 20 21 c) N o Y es Y es A A A O ve ra ll R N o 5- gr am K ne se rL M PP L So ur ce -B L E U B E R T O ve ra ll: R ef -B L E U G YA FC ,O w n N o Y es N o Y es M a an d L i( 20 21 ) N o N o N o A A A - N o G R U -L M PP L on da ta se t R ef -B L E U [4 ] Te xt C N N - Y el p[ s] ,G YA FC N o Y es Y es N o L ai et al .( 20 21 ) n/ a n/ a n/ a - - - - N o - B L E U R T, C O M E T, R ef - B L E U [4 ] C la ss ifi er - Y el p[ s] ,G YA FC N o Y es Y es N o M ir es hg ha lla h an d B er gK ir kp at ri ck (2 02 1) N o N o N o R R - - N o G PT -2 PP L ,G L E U ,l ex ica ld iv er si ty So ur ce -B L E U C la ss ifi er - Y el p[ s] ,T w itt er 2, D IA L N o Y es Y es Y es X ia o et al .( 20 21 ) N o N o N o A A A C on si st en cy A N o 5- gr am K en L M PP L R ef -B LE U ,S ou rc eB LE U B E R T - Y el p[ s] ,G YA FC N o Y es Y es N o M a et al .( 20 21 ) N o N o N o A A A - N o G R U -L M PP L on da ta se t R ef -B L E U [4 ] Te xt C N N - Y el p[ s] ,G YA FC N o Y es Y es Y es L au gi er et al .( 20 21 ) N o N o N o A A A O ve ra ll A N o G PT -2 PP L O w n B E R T - Y el p[ s] ,I M D b, To xi ci ty N o Y es Y es N o Y u et al .( 20 21 ) N o N o N o A A A - N o G ra m m ar ly ,L M PP L ,a dve rs ar ia l un ig ra m cl as si - fie r A ttr ib ut e H it (O w n) ,R ef - B L E U , So ur ce -B L E U , E M D C la ss ifi er - Y el p[ s] ,I M D b N o N o N o Y es B ri ak ou et al .( 20 21 a) N o N o N o A A A O ve ra ll R N o L L ,5 -g ra m K en L M PP L , B E R T PL L ,X L M PL L R ef -B L E U , So ur ce - B L E U , M E T E O R , ch rF , W M D ,C os in e Si m ,B ER T, B E R T Sc or e, X L M B E R T, X L M - G YA FC N o Y es Y es N o K as hy ap et al .( 20 22 ) N o N o N o A A A - N o R oB E R Ta fin etu ne d on ex te rn al da ta se t Se nt en ce Si m fa st Te xt - Y el p[ s] ,I M D b, Po lit ic al Sl an t N o N o N o N o L iu et al .( 20 22 ) N o N o N o A A A - N o - - Te xt C N N O ve ra ll: R ef -B L E U G YA FC N o Y es Y es N o R ei fe ta l. (2 02 2) N o N o N o A A A - N o G PT -2 PP L R ef -B L E U PT - Y el p[ s] ,G YA FC N o N o N o N o
Ta bl
e 9:
Sh ow
n ar
e th
e pa
pe rs
co ns
id er
ed fo
rt he
m et
aan
al ys
is .

heading: B Human Validation Details
text: We summarize the validation of automated metrics for TST evaluation in Table 10. In general, we consider an automated metric to be validated if it was validated for the mentioned aspect. We observe the three aspects of fluency, content preservation, and style transfer strength being validated. In addition, we also find validations for metrics evaluating TST as a whole. For fluency, we consider the PPL of an LM for measuring fluency to be validated. For style transfer strength, we consider any classifier architecture to be validated. Validation by Yamshchikov et al. (2021) is not considered because of the different purpose (validated on datasets for TST and paraphrasing and not actual model outputs).
So ur
ce Fl
ue nc
y C
on te
nt St
yl e
O ve
ra ll
D at
as et
s
X u
et al
.( 20
12 )
- R
ef -B
L E
U R
ef -B
L E
U ,P
IN C , L M ,L R ,C os in e Si m -
Sh ak
es pe
ar e
N iu
an d
B an
sa l(
20 18
) -
- -
So ur
ce -B
L E
U St
an fo
rd Po
lit en
es s
C or
pu s
L ie
ta l.
(2 01
8) R
ef -B
L E
U R
ef -B
L E
U B
iL
ST M
- Y
el p[
s] ,C
ap tio
ns ,
A m
az on
Fu et
al .(
20 18
) -
C os
in e
D is
t -
- A
m az
on ,P
ap er
-N ew s R ao an d Te tr ea ul t( 20 18 ) SG P C N N -S M FC R ef -B L E U ,P IN C ,T E R p G YA FC L uo et al .( 20 19 ) R ef -B L E U R ef -B L E U Te xt C N N - Y el p[ s] ,G YA FC
M ir
et al
.( 20
19 )
A dv
.C la
ss ifi
er ,
L ST
M -L
M PP
L
So ur
ce -B
L E
U ,M
E T
E O
R ,
E m
be d
A ve
ra ge
,E m
be d
G re
ed y,
E m
be d
E xt
re m
a, W
M D
Te xt
C N
N ,f
as tT
ex t
- Y
el p[
s]
Pa ng
an d
G im
pe l(
20 19
) L
M PP
L C
os in
e Si
m Te
xt C
N N
- Y
el p[ s] W an g et al .( 20 20 ) - - - R ef -B L E U G YA FC
Pr yz
an te
ta l.
(2 02
0) So
ur ce
-B L
E U , C la ss ifi er
So ur
ce -B
L E
U ,
C la
ss ifi
er So
ur ce
-B L
E U , C la ss ifi er
- W
N C
W u
et al
.( 20
20 )
SG P
- G
R U
E m
be d
A ve
ra ge
,E m
be d
E xt
re m
a, E
m be
d G
re ed y, B L E U -2
M T
FC ,T
C FC
C ao
et al
.( 20
20 )
- R
ef -B
L E
U ,S
ou rc
eB
L E
U -
- Si
m pW
ik i,
M SD
Y am
sh ch
ik ov
et al
.( 20
21 )
- M
ul tip
le -
- 7
T ST
& Pa
ra ph
ra se
D at
as et
s
B ri
ak ou
et al
.( 20
21 a)
5- gr
am K
en L
M PP
L ,
B E
R T
PL L
,X L
M PL
L
R ef
-B L
E U
,S ou
rc e-
B L
E U , M E T E O R ,c hr F, W M D , C os in e Si m ,B E R T Sc or e, B E R T, X L M
B E
R T,
X L
M -
G YA
FC
Ta bl
e 10
:S ho
w n
ar e
pu bl
ic at
io ns
va lid
at in
g au
to m
at ed
m et
ri cs
;u nd
er lin
ed ar
e th
e on
es sh
ow in
g th
e hi
gh es
tc or
re la
tio n
(i fm
ul tip
le co
m pa
re d)
.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations
A2. Did you discuss any potential risks of your work? Not applicable. We review existing work in terms of text style transfer evaluation and try to point out existing problems.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 0, 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We conducted an extensive meta-analysis of text style transfer. We surveyed 89 works summarized in
Sections 2, 3 and the Appendix

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 1, 2, 3, 4, Limitations, Appendix
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. We just reviewed the existing works.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 1,2,3,4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We reviewed existing papers, we did not collect data ourselves.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 7 Did you run computational experiments?

heading: Left blank.
text: C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, we reviewed text style transfer papers and pointed out the problems of existing human evalutions in
this field. Section 2.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Summarizing our results, we pose the following six requirements to be met by future TST research:
1. It needs to use validated metrics (see Table 10), focusing on those showing the highest correlations with human evaluations
2. In experiments, multiple runs need to be performed on different random seeds, reporting mean and standard deviation. 3. A statistical significance test needs to be performed on the results of automated metrics. 4. If a human evaluation is done, a statistical power analysis is necessary in advance, and all human evaluation details need to be published as suggested by Briakou et al. (2021b). 5. To improve reproducibility, researchers should always specify the preprocessing pipeline and publish their code (including evaluation code). 6. A comparison with state-of-the-art methods on the validated metrics is called for. To help with these requirements, we plan a largescale experimental comparison to rank existing methods according to validated metrics in the future and examine how automated metrics generalize from one TST task to the other. We also plan
to (re-)validate existing automated metrics to help meet the first requirement