$\textit{``Don't Take This Out of Context!''}$ On the Need for Contextual Models and Evaluations for Stylistic Rewriting
Authors: Akhila Yerukola
Xuhui Zhou
Elizabeth Clark
Maarten Sap
Venue: EMNLP 2023 Main (2023)
Keywords: stylistic rewriting
contextual evaluation
contextual generation
TL;DR: Preceding textual context needs to be included in the generation and especially the evaluation stages of stylistic text rewriting.

Abstract: Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambiguous, and incoherent rewrites. In this paper, we investigate integrating the preceding textual context into both the $\textit{rewriting}$ and $\textit{evaluation}$ stages of stylistic text rewriting, and introduce a new composite contextual evaluation metric $\texttt{CtxSimFit}$  that combines similarity to the original sentence with contextual cohesiveness.
We comparatively evaluate non-contextual and contextual rewrites in formality, toxicity, and sentiment transfer tasks. Our experiments show that humans significantly prefer contextual rewrites as more fitting and natural over non-contextual ones, yet existing sentence-level automatic metrics (e.g., ROUGE, SBERT) correlate poorly with human preferences ($\rho$=0--0.3). In contrast, human preferences are much better reflected by both our novel  $\texttt{CtxSimFit}$  ($\rho$=0.7--0.9) as well as proposed context-infused versions of common metrics ($\rho$=0.4--0.7). Overall, our findings highlight the importance of integrating context into the generation and especially the evaluation stages of stylistic text rewriting.

"Don't 't Take This Out of Context!"
On the Need for Contextual Models and Evaluations for Stylistic Rewriting
Akhila Yerukola
Xuhui Zhou
Elizabeth Clark
Maarten Sap
Language Technologies Institute, Carnegie Mellon University
Google DeepMind & Allen Institute for Al ayerukol @andrew. cmu www.cccc..lllo.o.loe edu
Abstract
Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambigu- ous, and incoherent rewrites. In this paper, we investigate integrating the preceding textual context into both the rewriting and evaluation stages of stylistic text rewriting, and introduce a new composite contextual evaluation metric CtxSimFi t that combines similarity to the orig- inal sentence with contextual cohesiveness. We comparatively evaluate non non-contextual and con- textual rewrites formality, toxicity, and senti- ment transfer tasks. Our experiments show that humans significantly prefer contextual rewrites more fitting natural over non-contextual ones, yet existing sentence-level automatic met- rics (e.g.. ROUGE, SBERT) correlate poorly with human preferences (p=0-0.3). contrast, human preferences are much better reflected by both our novel CtxSimFit (p=0.7-0.9) as well proposed context-infused versions of mon metrics (p=0.4-0.7) Overall, our findings highlight the importance of integrating context into the generation and especially the evalua- tion stages of stylistic text rewriting.
Preceding Dialog: '1 can't believe how much work have do."
Informal response: know, right? I'm drowning them."
fiewiee 1Les formal
Or make less toxic change to positive centimen
O O
Contextual rewrite worklead quitc everwnelmiin
Non-contextual Rewrite Indeed. inur
Figure I: Example using the preceding dialog utter- ance to help with stylistic rewriting: here, transform an informal response into formal language. Incorporat- ing "workload" and "overwhelming" enhances the con- textual cohesiveness of the rewritten text, while solely using "inundated" results in a more generic rewrite.
Similarly, preceding textual context has largely been overlooked in automatic evaluations for stylis- tic rewriting, with most work focusing on sentence- level metrics (c.g., Li et al., 2018; Reif et al., 2022). This lack of context at the modeling and evaluation stages hinders the creation of effective Al-assisted rewriting tools for users (e.g., for assistive writing tools; MacArthur, 2009; Clark et al., 2018).
1 Introduction
Existing methods for stylistic text rewriling, i.e.. adapting the text particular style while preserv- ing its originally intended meaning. often fail to ac- count for statement's 's context (e g., Hu et al., 2017; Shen et al., 2017; Fu et al., 2018: Li et al., 2018; Lample et al., 2019; Madaan et al., 2020; Hallinan et al., 2023). As result, these systems may change the speakers' original communicative intents and generate contextually irrelevant and generic out- puts. For example, in Figure non-contextual model rewriting an informal response to formal one simply replaces words with more formal syn- onyms, whereas contextual rewriting model can use the broader conversational context to produce a more specific and natural formal rewrite.
this paper, we present a comprehensive analy- sis of the need for context in stylistic rewriting and its evaluation, on three different rewriting tasks (formality transfer, sentiment change, and text detoxification) and two types of textual con- texts (preceding turns in a conversation, preceding sentences in document). To study these effects, design a contextual human evaluation frame- work ($5) to comparatively evaluate non-contextual and contextual rewriting methods built on few-shot prompted large language models ($4).

---

We show that human evaluators prefer contex- tual rewrites in terms of naturalness, style strength, and intended meaning preservation, across three tasks. However, non-contextual automatic met- rics for lexical or semantic meaning preservation correlate poorly with these preferences (p=0-0.3; $6), despite being commonly used in previous style transfer work to measure meaning preservation (Mir al., 2019; Madaan et al., 2020).
lack of datasets that include contextual informa- tion. While new models have emerged that do not require parallel data for training (Hu et al., 2017; Li et al., 2018; Ma et al., 2020; Hallinan et al., 2023), they also operate without contex- tual information. Building on some preliminary research that explored context in small custom- trained seq2seq rewriting models (Cheng et al., 2020; Atwell et al., 2022) and large language mod- els for exemplar- based conversation rewriting (Roy al., 2023), we extend the investigation large language models with defined style attributes like formality, toxicity, and sentiment. Importantly, we also explore the need for context in evaluations in addition to modeling, and propose new suite of contextualized metrics for automatic evaluation.
To address the need for context in automatic eval- uations, we introduce CtxSimFi it, new composite metric that combines original sentence similarity and contextual cohesiveness to evaluate the quality of rewrites, taking into account the preceding con- text (S7). Additionally, we propose context-infusee versions of commonly used automatic metries for meaning preservation. Our results show that hu- man preferences are significantly correlated with these contextual metrics- estrics--speciallyy CtxSimFit (0=9.,-0.9), much more than non contecttttt ones.
Evaluation of Stylistic Text Rewriting Evaluat- ing whether sentence rewriting preserves meaning while achieving the desired target style has proved challenging. Existing metrics and approaches can disentangle meaning and style (Mukherjee et al., 2022: Yu et al., 2021). However, determining what constitutes "meaning preservation" remains incon- sistent. Some works (Li et al.. 2018; Sudhakar et al.. 2019; Mir et al., 2019; Reif et al., 2022; Madaan et al., 2020) use metrics such as BLEU, ROUGE, and METEOR, which measure n-gram overlaps and lex- ical similarity as indicators of meaning preserva- tion respectively, while other studies (Wang et al., 2019; Reid and Zhong, 2021; Roy et al., 2023) adopt metrics like SBERT and BERTScore measur- ing semantic similarity of embeddings as proxies for meaning preservation. Further, the majority of work (Hu et al., 2022: Madaan et al., 2020; Li et al., 2018) does not provide annotators with any preced- ing context during human evaluations. Thus, more standardized and context-aware evaluation metrics are needed for text rewriting approaches.
Our contributions are summarized as follows: (1) We investigate the need for context in text rewriting, showing that incorporating it, whether at the docu- ment or conversational level, leads to contextually coherent and relevant rewrites preferred by human annotators across style transfer tasks. (2) We con- duct a comprehensive analysis on the need for con- text in automatic evaluation, revealing that existing metrics don't align with human preferences. (3) We propose a custom metric, CtxSimFit, along with context-infused versions of common auto- matic metrics, to bridge the gap between contextual understanding and automated metrics. Overall, our contributions provide a more nuanced understand- ing of the importance of context, which is criti- cal for development of more effective and reliable stylistic text rewriting techniques.
2 Background & Related Work
this section, we discuss the increasing interest in incorporating context into NLP tasks and motivate the significance of context during the rephrasing and evaluation phases of stylistic text rewriting.
Task and Datasets
To measure the importance of context in rewriting. we scope our investigations around three specific at- tribute controls: formality, sentiment, and toxicity, chosen because they necessitate varying degrees of meaning preservation and style intensity. We present statistics for each of the datasets used in our rewriting tasks in Table 1.
Stylistic Text Rewriting Despite being intro- duced over ten years ago (Xu et al., 2012), cur- rent methods for stylistic rewriting (c.g. Shen et al., 2017; Xu al., 2018b; Fu al., 2018; Lample al., 2019; Jin al., 2022: Chawla and Yang. 2020; Yerukola et al., 2021; Dale al., 2021; Lo- gacheva et al., 2022, etc.) still rely solely on paral- lel source-to-target sentence pairs, primarily due to
3.1 Tasks & Datasets
Changing Formality Formality transfer (Rao and Tetreault, 2018) aims to transform sentences

---

Task
Context Type
Datasets
# Instances
Conversation
Reddit
Preceding Dialogue
Formality
Document
CNN Daily
1000
Dialogue
+ Blog Authorship
Conversation
DailyDialog
1000
Original Sentence/ Utterance
Original Sentence/ Utterance
Original Sontence/ Utterance
Sentiment
Document
Yelp Reviews
1500
Conversation
CCC
1000
Toxicity
Conversation
MDMD
900
Open sourced
Conversation
ProsocialDialog
Table I: Statistics the collected datasets, presented by task and context type, considering both preceding sentences in a document and turns in conversation.
GPT 3.5
GPT-NeeX
Contentus! rewrite
Non coninetunt rewrite
Rondom comesiat rewrite
from informal or casual language into formal lan- guage, and vice versa. This requires making stylis- tic adjustments while ensuring that the original content and intention remain intact. We use conversational dataset from Reddit and curated dlocmment-based dataset from CNN Daily Mail (formal; Nallapati al., 2016) and the Blog Au- thorship Corpus (informal; Schler et al., 2006).
Nan-conterttol
Figure 2: Overview of our approach: examine three kinds of rewrites contextual, non-contextual, and ran- dom contextual rewrites. GPT-3.5 and GPT-NeoX are utilized for rewriting via in-context learning. Evaluation methods consist of non-contextual evaluation, which does not consider context, and contextual evaluation, which incorporates context into the assessment process.
Rewriting Sentiment For sentiment transfer (Hu et al., 2017), our focus lies in converting sentences with positive sentiment to negative sentiment, and vice versa, well as transforming neutral sen- tences to convey positive negative sentiment. Here, both the content and intention are altered; however, the main subject entities remain consis- tent, although with a change in sentiment. We obtain a conversational dataset from the DailyDia- log (Li et al., 2017) dataset and a document-based dataset from Yelp reviews (Zhang et al., 2015).
For document-basse datasets, we select three con- text sentences and one for rewriting.
We label the context and response using pre- trained style classifiers: RoBERTa-Base formal- ity classifier, SLMM-ROBERTa-Baa sentiment classifier (Barbieri et al., 2022) and toxicity scores from PerspectiveAPII, HateBert (Tommaso- Caselli and JelenaMitrovic, 2021) and HateRoberta (Hartvigsen al., 2022). We select a stratified sam- ple that includes a wide range of style strengths. See Appendix A for datasets and classifiers details.
De-toxifying Text Here, our objective is to rewrite text in a manner that reduces toxicity, as introduced by Nogueira dos Santos et al. (2018). Rewriting may modify the original content, but the initial intent should be preserved and conveyed using less offensive language. In this task, we examine three conversational datasets: the Civil Comments in Context (CCC) dataset (Xenos et al., 2021), the Multi-Label Dialogue Malevolence De- tection (MDMD) dataset (Zhang et al., 2022), and the ProsocialDialog dataset (Kim et al., 2022).
Modeling Context in Rewriting
In this section, we introduce our methodology for contextual stylistic rewriting utilizing large lan- guage models (LLMs) and in-context learning. We conduct comparison of three types of rewrites: those generated with context, those generated with- out context, and those generated with random context (as counterfactual baseline). Figure 2 provides visual representation of our approach.
3.2 Data Preparation
4.1 Contextual Rewriting
For conversational datasets (as depicted in the ex- ample Figure 1), we focus on two-turns, repre- senting parent context and response for rewriting.
To address the challenge of insufficient parallel data, as discussed in $2, we propose contextual
thttps://higgingface.coo / roberta-bbse-forrnlllyyyrrree
We use reddit t- corttttc-----eee small from http: cornell. Cornell.edu/docummee www.d/ooummettaiionn////oo stubrtateddttttttttttt..... html
"httss//ppesppcciiepp..c

---

Context
Original Sentence
Model
Style
Contextual Rewrite
Non contextual Rewrite
Best Bond movie evert Shitlect Bond (heme ever!!!
Haven't seen Spectro, have you? lutter, not the former.
GPT-355
formal
may not have had the best theme wong, but definitely of the best Bond movies!
Haven's you seen Spectre?
Huver'l you soen Spectre? Nor you, but someone else.
Informal
reading book, 777 Mathematical Conversation
>0 Herez some of them. and further I will keep updating this page with what
informal
Here are some of them, and in loke I will keep updat- ing this page with my favourite quoses from the book.
Here of and further will update this page with what che find intereviing.
Formality
GPTNNooX
Stariers by John dePillis. Its of
forual
interesting
like.
quotes, poemee
What exucily has Taylor Swift makses anyone want to 'smuck har?" This isn't runay and doesu't make sense.
0005,000000000000 I'd smack her though. For that much money my grandmother.
informal
tinancial incentive SX3S billion might lead me to censider striking Taylor Swifl. With such
For eight hundred and thiny-- five billion deillars, would clined commit the act of phys- ical assault with regards to gandmotherr.
GPT--355
formall
enormous amouunt stake,
would even be inclined to strike
my own gandmwother.
(d)
So zombic spocalypsc occurred, what would you da
USER I'm killing
GPT-3.5
loss toxic
I'd zombic apocalypee.
think
Jam feeling overwhelmed hopeless, so I'll die
mysell
Detocifiation
(c)
has rasantad histericaa oventis isn't it?
It's remaskable how the Bible
history. isn't i. And you disagree, RSN where
Bible reliable source for
You appecsr bhe saying the
GPT--NoX
less toxic
think that the Bible an atcurate sounee history. While may be sacred collection texts, liable source for histarical facts, For further discussion, I suggese visltin the RSN fonumns.
You
the Bible is relioble source for bistory. you have other opin- ions, please go RSN to be dis- heamennd
disagrze
down literally fire.
(
my throat feels weidd and dry quit
& USER oef, see, there are downsidss sucking
GPT-3.5
less toxic
UUUSE I hope you fool better sood.
USER there are negative sepects perforning sex.
(g)
Shall see you Friday momning
That's not good for me at It'll have to be another time.
GPT-35
negutive + postlive
Friday moming dusan' woek let's plan for another
That's great me. Let's plan for moother time.
time.
Seatiment
spicklly and insepansivelyy actually came to pick it up
Not only did Tim fix my
>> He's also an extremelyy Iriendly, funny guy who puts you at case immodiately.
GPT-NeoX
pasitive
>0
friendly humwenus which might make you feel uneasy
However, Tim's not very
>0 He is also rade, uno- mused guy makes you ansious.
nogative
himself so that I wouldn't 't have spend money low truck!
Last me soc. We should gu to some truit fins.
hassyy and not concetient to take them to the top
GPT-3.5
negative
light and very comvenient will nice, refreshing smack take them the top.
Buying fruit is great idea! It
posttive
to have on our way up.
Table 2: Examples from our datasets and corresponding rewrites generated by GPT-3.5 and GPT-NeoX, for all three tasks: formality change, de-toxification, and sentiment transfer. document context (truncated), >> = sentence to rewrite, >> contextual rewrite, >> nonn-conextuul rewrite; previous turn in conversation, contextual rewrite of response, non-contextual rewrite of response, "Style" header refers to the style conversion.
response rewrite,
rewriting model that utilizes the in-context learning capabilities of LLMs, inspired by approaches pre- sented in Reif et al. (2022) and Roy et al. (2023).
we employ LLMs to rewrite an original sentence from one style to another. Similar to contextual rewriting, we manually construct few-shot exam- ples that solely consist of the original sentence to be rewritten, an instructional prompt specifying the desired style, and an example rewrite, without any preceding context.
We conduct few-shot prompting experiments with two LLMs: GPT-3.5+ (Ouyang al., 2022) and GPT 5 (Black et al., 2022). Each example includes the preceding context, the original input with a specified style, and the rewrite in another style, factoring in the context. For GPT-3.5, we use 2 few-shot examples to obtain rewrites in the desired format, while for GPT-NeoX, we use 10 examples. See Appendix B for more details.
4.3 Rewriting with a Random Context
To demonstrate the importance of incorporating contextual information in the rewriting process, we employ a baseline method that generates rewrites using random context. This approach serves two key purposes: first, it assesses the contextual sensi- tivity of automatic metrics; and second, it ensures that our contextual rewriting method effectively ac- counts for the given context. In our experiments, we randomly pick a context from our dataset in- stead of using the true preceding context.
4.2 Non-contextual Rewriting
We are interested in comparing contextual rewrites with non-contextual rewrites that do not depend on prior context. To generate non-contextual rewrites,
'We use text-davinci-e00
'We use the 20B parameter model

---

5 Contextual Human Evaluation
workers on Amazon Mechanical Turk (MTurk) and qualified them using pre-qualification test for each task (See App C for qualification details).
Since in realistic rewriting scenarios, context will always be available and crucial to users who wish to rewrite their dialogue utterances story sentences (Atwell et al., 2022), we start by conducting con- textual human evaluation gauge user preferences between non-contextual and contextual rewrites. This contextual human evaluation is a departure from most previous work which has predominantly not used context ($2).
Agreement We employ three annotators to rank each pair of rewrites. Averaging across three tasks, our annotator agreement was Krippendorff's a = 0.43 and F'eiiss's 0.31. For dimension- specific annotator agreements, please refer to Ta- bles 7-9 in App C.I. We obtain the final human judgment preferences using majority voting of the three annotators.
5.1 Experimental Setup
5.2 Human Evaluation Results
We conduct a hcad-to-head human evaluation of non-contextual and contextual rewrites in the pres- ence of preceding textual context, following the setup in Kiritchenko and Mohammad (2017). Par- ticipants are given preceding context, pairs of rewritten sentences (non-contextual and contex- tual), and the desired style attribute. They are then asked to rank the rewrites with respect to:
Our results show that annotators prefer con- textual rewrites over non-contextual rewrites across all three tasks and context types (Figure 3). This effeet is especially pronounced for formal- ity and toxicity (see (a)-(f) in Table 2).
Contextual rewrites are more natural and fitting The success rate for contextual rewrites in toxicity and formality cases approximately 50% while that for non-contextual rewrites was close to 20% and 30%, respectively (p < 0.1). Regarding senti- ment, the success rate for contextual rewrites was around 35% as opposed to non-contextual rewrites with a success rate of about 30% (p > 0.1).
Naturalness: which rewrite do the annotators prefer / which one appears most natural
Style Strength: which rewrite best achieves the required style, independent of meaning changes
Event-level Similarity: which rewrite most ef- fectively retains the essential events, entities, and relations present in the original sentence, with- considering the preceding context
Contextual rewrites better preserve the intended meaning Contextual rewrites better preserve the author's 's intention, tone, and implied meaning more effectively (p < 0.1), In the detoxification task example (d) shown in Table 2, the user's 's intended meaning is not about actually killing oneself but rather about avoiding the zombie apocalypse. The contextual rewrite captures this meaning more ef- fectively compared to the literal rephrasing pro- vided by non-contextual rewriting.
Intended Meaning: which rewrite most effee- tively preserves and conveys the original sen- tence's s overall message or intended meaning
Overall Fit: which rewrite is overall most suit- able or relevant in relation to the given context
We sample 100 examples for sentiment from DailyDialog, 6 100 examples for formality, and 90 examples for toxicity", focusing on those with the highest style strength in each category (e.g.. 50 most formal and 50 most informal). We conduct significance testing for all three tasks. We recruited
Contextual rewrites struggle with preserving event-level similarity Examples (a), (f), and (i) in Table 2 demonstrate that contextual rewrites of- ten include extra entity/cvent details, while non- contextual rewrites align more closely with the original sentence at an n-gram level. 10 Despite this, annotators still prefer contextual rewrites for their naturalness and fit, indicating that extra event
"We opted not to use Yelp reviews in sampling due to difficulties encountered during pilot experiments. Annotators found it tough select rewrites that retained meaning while effectively transferring sentiment, such as from positive neg- ative. Generally, even contexts classified as "neutral" seemed positive when part of overall positive review, complicating the annotators' ability agree the rewrites' effectiveness.
"p O.I, CI=90% using binomial test and splitting the option evenly between contextual and non-contextual preferences.
'equal number from both Reddit CNN/DailyMan +
Blog Authorship Corpus
"equal number of examples from CCC, MDMD and Proso- cialDialog which were scored as highly toxic by all three toxicity classificrs hateroberta, hatebert and Perspoctive API
Event-level similarity is the only dimension which shows no significant differences hetween contextual and non- contextual rewrites for all three tasks.

---

Conmecal
Tex
ScerCattratial
Centeous
Te
Nan-Carrataal
Centrnnal
T>
Mon-Cwristii
StykSrespth
6:44
6.24
6.23
StytStonpth
6.43
0.26
9.27
ScAcStrongh
0.45
6:34
9.59
maarky
0:41
6:31
0.66
0.16
5.33
5.34
Crenti wrlty
0:38
0:1
0:41
6.44
626
6,20
6:31
6:22
Naturrless
Natuniees
6.36
(a) Formality
(b) Toxicity
(c) Sentiment
Figure Head-to-head human evaluation with context for all three tasks formality change, detoxification, and sentiment transfer. Contextual rewrites are generally favored over non-contextual rewrites across all tasks. particularly terms of style strength, preservation of intended meaning, naturalness, and overall coherence with the preceding context. The numbers the bars represent the proportion of preferences for each respective category.
details are acceptable as long as they fit appropri- atcly within the context.
Lavie, 2005) and word error rate (WER; Zechner and Waibel, 2000), for lexical similarity (Lexical).
Sentiment Style Transfer Might be Ill-defined The trends in the sentiment style transfer task are less pronounced than in other tasks (p > 0.1 for all dimensions) and show lower agreement compared toxicity and formality (see Table 9 in App C.I). Example (g) in Table 2 highlights the challenges in sentiment transfer due to the inherent need for meaning changes while preserving the original in- tent (cspecially for reviews which were written specifically to communicate sentiment; Yu et al., 2021). This complication leads to inconsistencies, resulting in annotators having difficulty reaching consensus meaning preservation, evidenced by lower agreement rates (Table 9).
Semantic Similarity To measure semantic simi- larity (Semantie), we BERTScore (Zhang al., 2019) and SBERT (Reimers and Gurevych, 2019), as employed in previous work. We also consider Smatch (Cai and Knight, 2013), which compares the similarity between two Abstract Meaning Rep- resentation (AMR) graphs, providing a distinctive. structured view on semantic relatedness not consid- ered in prior rewriting studies.
Fluency To assess fluency, we employ language model, specifically GPT-2 (Radford al., 2019). and use perplexity (pplx) as the metric, line with previous research (Holtzman et al., 2018; Xu et al.. 2018a; Ma et al., 2020).
6 Non-contextual Automatic Evaluation
6.2 Non-Contextual Evaluation Results
Overall, contextual human evaluations reveal general preference for contextual rewrites over non-contextual ones. Given that prior work pri- marily evaluated utterance-level rewrites in both human and automatic evaluations, raises the ques- tion of how well non-contextual automatic metrics mirror human preferences. In this section, we in- vestigate commonly used metrics in previous work (Mir et al., 2019; Hu al., 2022) for meaning preservation, style strength, and fluency.
In our analysis, we evaluate the performance of both GPT-3 and NeoX models producing non-contextual rewrites, contextual rewrites, and rewrites generated with random preceding con- text. We present aggregate results of the perfor- mance in Table 3 across all tasks, datasets, and metrics. For detailed results individual tasks and datasets, we refer the reader Appendix D.
Non-contextual rewrites more similar in meaning to the original input sentence com- pared to contextual rewrites Utterance level lexical and semantic meaning preservation met- rics score non-contextual rewrites higher, across all three tasks and the two types of context (see Ta- bles (6-22 in Appendix D). Additionally, we find that our patterns are consistent for both GPT-3.5 and NeoX, though note a marked decrease in performance from GPT-NeoX.
6.1 Metries Considered
We distinguish two types of "meaning preservation" metrics, namely. lexical and semantic similarity between rewrite X and the original input I.
Style Strength Following previous studics (Li et al., 2018; Madaan et al., 2020), we assess style strength of rewritten text by examining the proba- bilities of the target style under our style classifier.
This suggests that models that edit the original sentence more (i.e., preserve lexical and seman- tic similarity less) are better at achieving the de- sired style. For fluency measured by perplexity, we
Lexical Similarity We use word-overlap met- rics like ROUGE (Lin, 2004), METEOR (Banerjee and

---

model
type
Lexical
Semonaic SBERT
Plueney
Style
ROU
MET
WER
BERT-S
Smatch
PPL
contextunl
0.18
0.38
1,66
0.90
0.59
0.45
40.25
0.74
GPT-3.5
non coniexoll
0.28
0.48
0.88
0.92
0.70
0.57
47.69
0.70
randon--ontext
0.16
0.34
2.04
0.89
0.50
0.42
GFT-NcoX
contextual
0.25
0.37
1.43
0.90
0.57
0.44
55.31
0.52
non-contexttal
0.41
0.56
0.78
0.93
0.74
0.60
64.65
0.44
fandon--ontext
0.24
0.37
1.55
0.90
0.54
0.43
Table Non-contextual Automatic Evaluation Results: Non-contextual rewrites achieve higher scores in lexical and semantic similarity metrics whereas contextual rewrites demonstrate enhanced style strength and fluency. These results are obtained by averaging across all tasks and datasets. This heatmap displays the best-performing rewrite for each specific metric darker orange indicates higher preference. For more details on individual tasks and datasets exhibiting similar trends, see App D.
find that both approaches generate decently fluent rewrites regardless of context, as expected." !I
original input sentence I before comparing it to the rewrite X: sim(C + I, X). The intuition behind this alteration is that the preceding textual context could capture more of the topical semantic in- formation necessary fully derive the speaker's intended meaning.
Non-contextual metrics do not correlate with human judgments We see in Figure 3 and Ta- ble 3, that the non-contextual automatic metrics paint an incomplete picture compared to human evaluations. We compute Spearman rank correla- tion and Kendall's for the dataset samples used during the contextual human evaluation $5.1. Non- contextual automatic metrics cxhibit very weak, non-significant correlation with human judgments of overall fit (averaged across all tasks): p = 0.09, T 0.09 for lexical metrics (p > 0.05) and p = 0.23, T = 0.22 for semantic metrics (p > 0.05). See Appendix D.I for metric-specific correlation scores for overall fit and naturalness dimensions.
Contextual Lexical and Semantic Similarity For lexical similarity, we refer to these metrics as ROUGE METEOR and WER For seman- tic similarity, we refer to them as BERTScore SBERT and Smatch
Contextual Coherence and Cohesiveness In lin- guistics, coherence and cohesiveness are terms typ- ically used to denote the connectedness embedded or implied spoken or written discourse.
(a) Coherence: Coherence is generally defined as the overall picture presented by all the sentences in piece of writing, similar to the way puzzle pieces form the image on the box (Williams, 1990; Zienkowski al., 2011). 1). This definition is often operationalized by modeling the fit of sentence given its preceding context, as demonstrated by prior work (See et al., 2019; Pang et al., 2020). Specifically, this involves measuring perplexity of the rewrite conditioned on the context using GPT-2 (Radford al., 2019).
7 Contextual Automatic Evaluation
As shown in the previous section, non-contextual automatic metrics, especially for meaning preserva- tion, are not sufficient to evaluate the performance rewriting models. To address this, incorporating context into the evaluation process is necessary for better representing realistic downstream use cases. Drawing inspiration from reference-free metrics in dialog evaluation (Yeh et al., 2021; Zhao et al., 2017), which considers both the dialog context and generated responses assess responses within the dialogue history, we propose including context into existing automatic evaluation metries and further introduce CtxSimFit, it, a new contextual metric.
(b) Cohesiveness: Cohesiveness refers the semantic relationships between sentences, link- ing current elements with preceding or following ones through lexical and structural means, much like how two jigsaw puzzle pieces fit together (Williams, 1990; Zienkowski et al., 201 1). Fol- lowing prior work that used this definition (Shi and Demberg, 2019; Abhishek al., 2021; Nguyen, 2021), measure cohesiveness using the prob- abilities from the Next Sentence Prediction (NSP) head BERT (Devlin et al., 2018), which mea- sures if the rewrite follows and fits with its the
7.1 Infusing Automatic Metrics with Context
Since context crucial to derive intended meaning (Searle, 1975), we alter existing meaning similar- ity measures by prepending the context C' to the
"I "Lower perplexity generally indicates higher sentence qual- ity and grammaticality, but may not directly correlate with mcaning preservation, style, or content relevance.

---

Lecical MET
Somantic SBERT
Coherence PPL
Colerivaess NSP
Cussomn CtxSinFit
type
ROL
WER
BERT-S
Smarchy
contextual
0.15
0.24
0.89
0.88
0.59
0.35
28.70
0.95
0.93
GPT-3.5
neo-contectual
0.16
0.22
0.88
0.88
0.49
0.32
42.94
0.89
0.91
candom-context
0.10
0.18
1.06
0.87
0.39
0.29
45.73
0.80
0.85
GPT-NeoX
contextual
0.19
0.22
0.87
0.88
0.53
0.31
31.10
0.93
0.92
nco-contextual
0.21
0.23
0.86
0.88
0.48
0.30
49.81
0.90
0.91
cadoon-context
0.13
0.17
0.98
0.39
0.26
52.93
0.83
0.86
Table 4: Contextual Automatic Evaluation Results: On average, across all tasks and datasets, contextual rewrites achieve higher scores than non-contextual rewrites when evaluated using context-infused automatic metrics and our CtxSimFit i metric. This heatmap shows the best-performing rewrite for particular metric - darker green indicates higher preference. For more details on individual tasks and datasets displaying similar trends, see App E.
preceding context.
Contextual metries correlate significantly with human judgments We find that contextual au- tomatic metrics correlate significantly with hu- man judgments 'overall fit' (averaged across all tasks): p = 0.6, 7 = 0.58 for lexical metrics (p < 0.05) and p 0.56, r 0.57 for semantic metrics (p 0.05). See Appendix E. for metric- wise correlation scores for both overall fit and nat- uralness human judgment dimensions.
7.2 Novel Composite Metric: CtxSimFit
We introduce CtxSimFit, simple metric that com- bines contextual cohesiveness and semantic sim- ilarity to assess the overall quality of rewrite. CtxSimFit computes the weighted average both the BERTScore between the original and rewritten sentences, and the probabilities from the BERT's NSP head between the preceding context and the rewrite, thus determining how well the rewrite fits the preceding context and maintains semantic simi- larity.
CtxSimFit correlates the best with human judgements Compared to contextual versions of existing metrics, CtxSimFit correlates very strongly with human judgements of 'overall fit' (averaged across all tasks): P 0.85, 0.82 (p < 0.01). We see similar trends for 'naturalness' == 0.85, t = 0.81 (p < 0.01). This suggests that combining meaning preservation and contextual co- hesiveness into a composite measure better mirrors human preferences than individual metrics alone.
CtxSimFit = * BERTSCORE(S,X
+ (1 - a) NSP(C,
where is hyperparameter that provides users with control over their preference for balaneing meaning preservation and contextual fit. Unless specified otherwise, we set a = 0.5.
Contextual rewrites are scored higher on style strength compared to non-contextual rewrites
7,4 Sensitivity analysis for in CtxSimFit
In our experiments, we set 0.5 to equally weight contextual cohesiveness and semantic sim- ilarity. We further examine the impact of in CtxSimFit it, detailed by Table
7.3 Contextual Evaluation Results
Similar to $6.2, we aggregate the results of both GPT-3.5 and NeoX across all tasks, datasets and metrics (see Table 4). For detailed results in- dividual tasks and datasets, refer the reader to Tables 23-29 in Appendix E.
Our CtxSimf it significantly correlates with hu- man judgments of 'overall fit' for a values within the range of 0.2-0.6, with correlation and signifi- cance diminishing outside this range. The highest alignment with human judgments is achieved 0.5. The longer range of 0.2-0.5 for a 0.5 highlights the effect and importance of contextual cohesiveness in stylistic text rewriting.
Contextual rewrites are preferred by nearly all of our contextual automatic metrics compared to non-contextual rewrites These results mir- ror human preferences naturalness, fit and in- tended meaning preservation. As reality check, contextual rewrites with random contexts perform the worst across all metries, indicating that contex- tual models are indeed taking context into account. Further as expected, contextual rewrites also have better coherence compared to non- con-contextual ones.
While balanced approach (a 0.5) offers the strongest alignment with human judgments for formality, sentiment and de-toxification tasks, the degree of emphasis on contextual cohesiveness and semantic similarity should be adjusted based on specific tasks and users' priorities.

---

Hyperparameter in CtxSinFit
Task
Correlation with "overall fit'
Significance
entities and better vocabulary usage (examples (a), (c)), retaining relevant details from context for a better flow (examples (b), (i)) and preserving the intended meanings (examples (d), (g)).
Formality
-0.03
as
Toxicity
ns
Sentimett
0.04
Formality
0.66
0.2
Toxicity
0.65
Sentiment
0.54
Existing meaning preservation metrics do align with human preferences for formality, sen- timent and toxicity transfer tasks Next, we demonstrate that common non-contextual auto- matic metrics for lexical and semantic similarity, i.e., often used as proxies for meaning preservation in prior work (Li et al., 2018; Sudhakar et al., 2019; Mir et al., 2019; Reif et al., 2022; Madaan et al., 2020; Wang al., 2019; Reid and Zhong, 2021; Roy al., 2023), do not align with human pref- erences concerning naturalness, fit, and intended meaning. Since the overarching meaning of a sentence largely depends on its context (Searle, 1975; Clark, 1997, 1996), non-contextual proxies for meaning preservation will always be in tension with any stylistic change to the sentence, making the trade-off hard navigate (Mir al., 2019; Hu al., 2022). Therefore, advocate for discontin- uing non-contextual meaning preservation metries in stylistic rewriting tasks and for more research into better modeling of communicative intents or goals (Adolphs et al., 2022; Zhou et al., 2022).
Formality
0.75
Toxicity
Sentiment
one
Formality
0.4
Toxicity
0.67
60e
Sentiment
0.60
Formality
0.88
0.5
Toxicity
0.82
Sentimeat
Formality
Toxicity
Sentimntt
0.42
o.e
Formality
0.32
0.7
Toxicity
0.38
Sentiment
0.20
ms
Formality
0.24
Toxicity
Sentiment
as
Formality
0.25
0.9
Toxicity
0.28
Sentiment
0.20
ns
Table 5: Sensitivity of the a in CtxSimFit it across all tasks. indicates correlation of CtxSimFit with human judgments of 'overall fit'. ns indicates not significant (p 0.05), * is 0.05, ** is p< 0.01, *** p< 0.001
Contextual automatic metrics, especially CtxSimFit, better mirror human judgments In our work, we attempt to bridge the gap between non-contextual metrics and contextual human evaluations by integrating context into automated metrics ($7). Our proposed composite metric, CtxSimFit it, balances meaning preserva- tion with contextual cohesiveness, providing a more comprehensive measure that better aligns with human judgments. While commonly-used automatic metrics enriched with context align with human preferences, our proposed CtxSimFit demonstrates stronger correlation.
8 Summary & Discussion of Findings
Existing work on stylistic text rewriting has of- ten neglected the surrounding context of the sen- tence. In our study, we focus on incorporating the preceding textual context in documents and con- versations into both the modeling and evaluation stages of rewriting. We develop a contextual hu- man evaluation framework and compare its results to non-contextual automatic metrics, contextual- ized versions of these metrics, as well as to our new composite metric CtxSimFit.
Context is crucial for rewriting Corroborating findings by Cheng al. (2020) and Roy et al. (2023), contextual rewrites are significantly pre- ferred by human annotators in terms of naturalness, intended meaning preservation, and style strength. Additionally, we demonstrate that having the right context is crucial for contextual rewriting, as ev- idenced by the poor performance of contextual rewrites generated using a random context.
Initial work in evaluating open-domain dialogue generation with context (Welleck et al., 2019; Pang et al., 2020) has been done, but we encourage fur- ther development of better contextualized metrics for stylistic rewriting evaluation. Improvements could include modeling themes. tones, sentence structures (Zhang al., 2014; Khatri al., 2018: Chen and Yang, 2020; Toubia et al., 2021: Shen et al., 2023), and social dynamics, and emotional states in conversations (Sap et al., 2017; Rashkin et al., 2018, 2019: Mostafazadeh et al., 2020).
Qualitative examination (Table 2) shows that contextual rewrites are better at disambiguating

---

9 Limitations & Ethical Considerations
Parthasarathi al., 2021; Su et al., 2023). More- over, custom-made rewriting models from prior research often lack the modeling of context (Ma et al., 2020; Dale et al., 2021). We believe the our results still apply for smaller models, given some preliminary research (Cheng et al., 2020; Atwell et al., 2022) on an increased human preference for contextual rewrites from custom- custom-trained trained seq2seq models. We encourage future work to thoroughly investigate strategies for effective modeling and evaluation of context smaller models.
Despite taking the first step towards incorporating context into stylistic rewriting and its evaluation frameworks, there are several limitations and ethi- cal concerns, which we list below.
Limited Context Scope In this study, our pri- mary focus is incorporating textual context, particularly from preceding sentences or previous turns in conversation. Future work should ex- plore how to incorporate other forms of context into rewriting models and evaluations, such as dis- course structure (Welleck et al., 2019), external knowledge (Ghazvininejad et al., 2018), richer social and power dynamics (Antoniak et al., 2023), emotional states (Zhou et al., 2023), and commu- nicative intent (Zhou al., 2022), all of which can significantly contribute to understanding the text.
Harms of Exposing Workers to Toxic Content In our work, we exposed human annotators to toxic content during the evaluation of the de-toxification task. Exposure to such offensive content can be harmful to the annotators (Liu et al., 2016). We aim to work towards developing evaluation strategies that can minimize the exposure of annotators toxie content.
Amount of Context In our experiments, we opted to investigate the context of three preced- ing sentences in document and one preceding conversational turn, considering only a specific length. However, the amount of context at the mod- eling and evaluation stages could also change the results. We hypothesize that more context could improve rewriting methods, but it could potentially also negatively impact contextual meaning preser- vation metrics. Future work should explore these effects of varying lengths of context.
Potentially Inconsistent Human Evaluations In our work, we also assume human judgments as the gold standard. Concurrent work has shown that human evaluation might not always be con- sistent (Clark et al., 2021; Karpinska et al., 2021); however human judgments continue to be the gold standard for evaluating open-ended text generation.
A Acknowledgements
We would like to thank our workers on MTurk for their responses. We are also grateful to the anonymous reviewers for their helpful comments. Special thanks to Saadia Gabriel, Jocelyn Shen, Ashutosh Baheti, and the members of the CMU LTI COMEDY group for their feedback, and Ope- nAT for providing access to the GPT-3.5 API. This research was supported in part by the Meta Fun- damental Research Laboratories (FAIR) "Dyn- abench Data Collection and Benchmarking Plat- form" award "ContExTox: Context-Aware and Ex- plainable Toxicity Detection."
Broad Definition of Meaning Preservation While we have tried to define meaning preserva- tion as the preservation an event or entity-level details and intended overall meaning, this defini- tion remains broad and subjective (Searle, 1975; Adolphs et al., 2022; Zhou et al., 2022). In this work, we do not delve into more intricate dimen- sions of meaning preservation, such as spatial and temporal accuracy, or the retention of cultural con- text, including references, nuances, and dialects.
Applicability to Smaller Models Our work re- lies on few-shot prompting of LLMs to incorporate textual context, given their demonstrated strong rewriting capabilities both with and without textual context usage (Brown al., 2020). Other exist- ing generative models, such as those used for chit- chat and goal-oriented conversational agents, as well as pretrained language models, have struggled with effectively utilizing preceding textual context (Sankar et al., 2019; O' O'Connor and Andreas, 2021;
References
Tushar Abhishek, Daksh Rawat, Manish Gupta, and Vasudeva Varma. 2021. Transformer models for text coherence assessment. arXiv preprint arXiv:2109.02176.
Leonard Adolphs, Kurt Shuster, Jack Urbanek, Arthur Szlam, and Jason Weston. 2022. Reason first, then respond: Modular generation for knowledge-infuse

---

dialogue. In Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 7112- 7132.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence- to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4106- 4118.
Maria Antoniak, Anjalie Field, Ji Min Mun, Melanie Walsh, Lauren F. Klein, and Maarten Sap. 2023. Riv- eter: Measuring power and social dynamics between entities. In ACL demonstrations.
Yu Cheng. Zhe Gan, Yizhe Zhang, Oussama Elachqar, Dianqi Li, and Jingjing Liu. 2020. Contextual text style transfer. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2915- 2924.
Katherine Atwell, Sabit Hassan, and Malihe Alikhani. 2022. Appdia: A discourse-awar transformer-basse style transfer model for offensive social media con- versations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6063-6074.
Elizabeth Clark, Tal August, Sofia Serrano. Nikita Haduong. Suchin Gururangan, and Noah A Smith. 2021. All that's 'human' is not gold: Evaluating hu- man evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 7282-7296.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for evaluation with improved cor- relation with human judgments. In Proceedings of the ucl workshop intrinsic und extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with machine in the loop: Case studies on slogans and stories. In 23rd International Conference Intelligent User Interfaces, pages 329-340.
Francesco Barbieri, Luis Espinosa Anke, and Jose Mamacho-Colladoo 2022. Xlm-t: Multilingual lan- guage models in twitter for sentiment analysis and beyond. Proceedings of the Thirteenth Language Resources and Fyaluation Conference, pages 258- 266.
Herbert H Clark. 1996. Using language. Cambridge university press.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregres- sive language model. In Proceedings of BigScience Episode # 5-Workshop on Challenges & Perspec- tives in Creating Large Language Models, pages 95- 136.
Herbert H Clark. 1997. Dogmas of understanding. Dis- course Processes, 23(3)5667-598.
David Dale, Anton Voronov, Daryna Dementieva, Var- vara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. 2021. Text detoxification us- ing large pre-trained neural models. In Proceedings of the 2021 Conference Empirical Methods in Nat- ural Language Processing, pages 7979-7996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vass. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv: 1rXiv:1810.04000 1810.04805.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng. Dongyan Zhao, and Rui Yan. 2018 Style transfer in text: Explo- ration and evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Marjan Ghazvininejad, Chris Brockett, Ming- Wei Chang, Bill Dolan, Jianfeng Gao. Wen-tau Yih, and Michel Galley. 2018. A knowledge-groune ral conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceed- ings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 748-752.
neu-
Skyler Hallinan. Alisa Liu, Yejin Choi, and Maarten Sap. 2023. Detoxifying text with marco: Controllable revision with experts and anti-experts. In ACI.
Kunal Chawla and Diyi Yang. 2020. Semi-supervised formality style transfer using language model dis- criminator and mutual information maximization. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 2340-2354, Online. Association for Computational Linguistics.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generatee dataset

---

for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association of Computational Linguistics.
Yanran Li. Hui Su, Xiaoyu Shen, Wenjie Li. Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-tumn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Nat- ural Language Processing (Volume I: Long Pupers), pages 986 985-995.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Leam- ing to write with cooperative discriminators. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 1638-1649.
Chin- Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branckes out, pages 74-81.
Zhiqiang Hu, Roy Rei Lee, Charu C Aggarwal, and Aston Zhang. 2022. Text style transfer: A review and experimental evaluation. ACM SIGKDD Explo- rations Newsletter, 24(1):14-45.
Chia- Lii Liu, Ryan Lowe. Iulian Vlad Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 1122-2132
Zhiting Hu, Zichao Yang. Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward con- trolled generation of text. In International conference on machine learning, pages 1587-1596. PMI.R.
Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. 2022. Paradetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6804-6818.
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey, Computational Linguistics, 48(1):155-205.
Marzena Karpinska, Nader Akoury, and Mohit lyyer. 2021. The perils of using mechanical turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265-1285.
Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. 2020. Powertransformer: Unsupervised con- trollable revision for biased language correction. In EMNLP.
Chandra Khatri, Rahul Goel, Behnam Hedayatnia, Angeliki Metanillou, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018. Contextual topic modeling for dialog systems. In 2018 ieee spoken language technology workshop (sli), pages IEEE
Charles A MacArthur. 2009. Reflections on research on writing and technology for struggling writers. Learn ing Disabilities Research & Practice, 24(2):93-103.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barn- abás Poczós, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhu- moye. 2020. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 1889-1881.
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim. Yejin Choi, and Maarten Sap. 2022. Prosocialdialog: A prosocial backbone for conversational agents. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4005-4029.
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 495-504.
Svetlana Kiritchenko and Saif Mohammad. 2017. Best- worst scaling more reliable than rating scales: case study on sentiment intensity annotation. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465-470.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer. Marc' Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attre attribute text rewriting. In International Conference on Learning Representa- tions.
Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll. 2020. Glucose: Generalized and contextualized story explanations. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4569-4588
Juncen Li. Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: simple approach to senti- ment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Pupers), pages 1865-1874.
Sourabrata Mukherjee, Zdeněk Kasner, and Ondřej Dušek. 2022. Balaneing the style-content trade-off in sentiment transfer using polarity- aware denoising. In Text, Speech, and Dialogue, pages 172-186, Cham. Springer International Publishing.

---

Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang. 2016. Abstractive rnns
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Anmual Meet- ing of the Association for Compulational Linguistics. pages 5370-5381.
text summarization using sequence-to-sequeee and beyond. In Proceedings of the 20th SIGNLL Con- ference on Computational Natural Language Learn- ing, pages 280-290.
An Nguyen. 2021.
Language model evaluation in open-ended text generation. arXiv preprint wrXiv:2108.0357..
Machel Reid and Victor Zhong. 2021. Lewis: Leven- shtein editing for unsupervised text style transfer. In Findings of the Association for Computational Lin- guistics: ACL-IJCNLP 2021, pages 3932-3944.
Cicero Nogueira dos Santos, Igor Meinyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pages 189-194, Melbourne, Australia. As- sociation for Computational Linguistics.
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burees and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 837-848.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Neural Information P Processing Systems, 35:27730-2773
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert, Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 3982-3992.
Joe O' O'Connor and Jacob Andreas. 2021. What context features can transformer language models use? In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 1th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 851-864.
Shamik Roy, Raphael Shu. Nikolaos Pappas, Elman Mansimov, Yi Zhang, Saab Mansour, and Dan Roth. 2023. Conversation style transfer using few-shot learning, arXiv preprint 2.X2::2302.08362
Chinnadhurai Sankar, Sandeep Subramanian, Christo- pher Pal, Sarath Chandar, and Yoshua Bengio. 2019. Do neural dialog systems use the conversation his- tory effectively? an empirical study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 32-37.
Bo Pang, Erik Nijkamp. Wenjuan Han, Linqi Zhou, Yix- ian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue gener- ation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 3619-3629
Maarten Sap, Marcella Cindy Prasetio. Ari Holtzman, Hannah Rashkin, and Yejin Choi. 2017. Connotation frames of power and agency in modern films. In EMNLP.
Prasanna Parthasarathi, Joelle Pineau, and Sarath Chan- dar. 2021. Do encoder representations of generative dialogue models have sufficient summary of the in- formation about the task? In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 477-488.
Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006. Effects of age and gen- der on blogging. In AAAI spring symposium: Compu- tational approaches to analyzing weblogs, volume 6, pages 199-205.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
John R Searle. 1975. A taxonomy of illocutionary acts.
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pages 843-861.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, bench- marks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Jocelyn Shen, Maarten Sap, Pedro Golon-Hernandez Hae Won Park, and Cynthia Breazeal. 2023. Model- ing empathic similarity in personal narratives.
Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psy- chology of characters simple commonsense stories. In ACL.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. Advances in neural information processing systems, 30.

---

Wei Shi and Vera Demberg. 2019, Next sentence pre- diction helps implicit discourse relation classification within and across domains. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference natural language processing (EMNLP- IJCNLP), pages 5790-5796.
Unpaired sentiment- toeeentttt--seeeee so--emtiment translation: ey- cled reinforcement learning approach. In Proceed- ings of the 56th Anmual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 979-988, Melbourne, Australia. Association for Computational Linguistics.
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Pro- ceedings of COLING 2012. pages 2899-2914, 4, Mum- bai, India. The COLING 2012 Organizing Commit- tee.
Hsuan Su, Shachi H Kumar, Sahisnu Mazumder, Wenda Chen, Ramesh Manuvinakurike, Eda Okur, Saurav Sahay, Lama Nachman, Shang-Tse Chen, and Hung- yi Lec. 2023. Position matters! empirical study of order effect in knowledge-Pi -nowledge-rrounded dialogue. arXiv preprint arXiv:2302.0553 058888...
Yi-Ting Ych, Maxine Eskenazi, and Shikib Mchri. 2021. A comprehensive assessment of dialog evaluation metrics. In The First Workshop Evaluations and A Assessments of Neural Conversation Systems, pages 15-33.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Ma- heswaran. 2019. "transforming" delete, retrieve, gen- erate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Longuage Processing (EMNIP--JCNTL. pages 3269-3279.
Akhila Yerukola, Mason Bretan, and Hongxia Jin. 2021, Data augmentation for voice-assistant NLU using BERT-bae -bsed interchangeable rephrase. In Proceed- ings of the 16th Conference of the European Chap- ter of the Association for Computational Linguistics: Main Volume.
ValerioBasile TommasoCaselli and MichaelGranitzer JelenaMitrovic. 2021. Hatebert Retraining bert for abusive language detection in english. WOAH 2021, page 17.
Ping Yu, Yang Zhao, Chunyuan Li, and Changyou Chen. 2021. Rethinking sentiment style transfer. In Find- ings of Association for Computational Linguistics: EMNLP 2021, pages 1569-1582.
Olivier Toubia, Jonah Berger, and Jehoshua Eliashberg. 2021. How quantifying the shape of stories predicts their success. Proceedings of the National Academy of Sciences of the United States of America, 118(26).
Klaus Zechner and Alex Waibel. 2000. Minimizing word error rate in textual summaries of spoken lan- guage. In /st Meeting of the North American Chapter of the Association for Computational Linguistics.
Yunli Wang, Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019. Harnessing pre-trained neural networks with rules for formality style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3573-3578.
Kai Zhang, Wei Wu, Haocheng Wu, Zhoujun Li, and Ming Zhou. 2014. Question retrieval with high qual- ity answers in community question answering. In Proceedingy of the 23rd ACM international confer- ence on conference on information and knowledge management, pages 371-380)
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3731-3741.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0661
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Joseph Williams. 1990.
Toward clarity and grace. Chicago: The University of Chicago.
Alexandros Xenos, John Pavlopoulos, and lon Androut- sopoulos. 2021. Context sensitivity estimation in toxicity detection. In Proceedings of the Sth Work- shop on Online Abuse and Harms (WOAH 2021).
Yangjun Zhang, Pengjie Ren, Wentao Deng, Zhumin Chen, and Maarten Rijke. 2022. Improving multi- label malevolence detection in dialogues through multi-faceted label correlation enhancement. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 3543-3555.
pages 140-145.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-rommoting
2018a. Diversity- promoting gan: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of the 2018 con- ference on empirical methods in natural language processing, pages 3940-3949.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse- level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 6544
Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xu- ancheng Ren, Houfeng Wang, and Wenjie Li. 2018b.

---

Pei Zhou, Hyundong Cho, Pegah Jandaghi. Dong-Ho Lee. Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022. Reflect, not reflex: Inference-based common ground improves dialogue response quality. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Lunguage Processing, pages 10450- 10468.
Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas David- son, Jena D. Hwang. Swabha Swayamdipta, and Maarten Sap. 2023. Cobra frames: Contextual rea- soning about effects and harms of offensive state- ments. In Findings of ACL.
Jan Zienkowski, Jef Verschueren, and Jo Östman. 201 I. Discursive pragmaties: A platform for the pragmatic study of discourse. In Disursive Pragmatics, pages 1-13. John Benjamins Publishing Company.

---

A Tasks and Datasets
HateBert and HateRoberta measure the toxi- city of the context and responses.
Formality Data We obtain a conversational dataset from Reddit by sampling conversa- tional threads from subreddits such as r/news, r/askscience, and r/Economics (formal conversa- tions), as well r/movies, r/fantasyfootball, and rfrelationships (informal conversations). We fo- on two-turn Reddit threads: a parent/preceding context and the response to be rewritten, Next, we sample documents from CNN Daily Mail (formal documents; Nallapati ct al., 2016) and the Blog Au- thorship Corpus (informal documents; Schler et al., 2006). We select four sentences from each data sample: three sentences as the preceding parent context, and the following sentence the to be rewritten. For each data sample, label the context and response using pre-trained formality classifier.
B Modeling Context in Rewriting
We perform few-shot prompting experiments with GPT--.55 and GPT-NeoX. For GPT-3.5, we use 2 few-shot examples, while for GPT-NeoX, we use 10 few-shot examples. Each few-shot example was manually constructed with the preceding context, an original sentence to be rewritten, an instruction specifying the required style, and a sample rewrite. Figures 6 and 7 display the few-shot prompt exam- ples that we utilized for generating rewrites in the formality change task.
B.1 In-context learning sample Rewrites
Table shows some additional example rewrites from GPT-3.5 and GPT-NeoX for all tasks.
Sentiment Data We obtain conversational dataset from the DailyDialog (Li al., 2017) dataset, focusing on two-turn conversations: a par- ent/preceding context and the response be rewrit- ten. Next, we sample entries from the Yelp reviews (Zhang et al., 2015) dataset. Analogous to the doc- ument dataset used in formality, we choose four sentences from each data sample: three as the pre- ceding parent context and the subsequent sentence the one to be rewritten. For each data sample, we annotate the context and response using senti- ment classifier." We partition the data to transform sentences from positive to negative sentiment and vice versa, as well as to convert neutral sentences to positive or negative sentiment.
C Contextual Human Evaluation
Worker selection We involve annotators from USA and Canada Amazon Mechanical Turk (MTurk)), who voluntarily opt-in for each task. We recruit annotators for each style transfer task via corresponding qualification task, In the qualifi- cation task, annotators must answer two questions per pair of rewrites: which rewrite has the strongest style strength (e.g., most formal), and which rewrite is the most natural given the preceding context. An- notators assess three pairs of handcrafted rewrites in each qualification task. Those who accurately answer at least five of the six questions (three for style and least two for naturalness) are approved for the main task. Once approved. we pay them $0.27 USD per head-to-head comparison.
Toxicity Data We examine three conversational datasets: the Civil Comments in Context (CCC) dataset (Xenos et al., 2021), the Multi-Label Di- alogue Malevolence Detection (MDMD) dataset (Zhang et al., 2022), and the ProsocialDialog dataset (Kim et al., 2022). For each dataset, we select two turns from each conversational thread, representing the preceding parent context and the subsequent response as the sentence be rewrit- ten. We use toxicity scores from Per
C.1 Human Evaluation Results
We present the agreement results of the human eval- uation studies of detoxification (Table 7), formality change (Table 8) and sentiment transfer (Table 9). Additionally, refer to Figures 4 and 5 for screen- shots of the human evaluation instructions provided to annotators and the actual task, respectively.
insce-ctee arcementi Stykdincahh Ever Incenaableasing Nanrino
Kripendont'ss
02207
We use reddi corpoo-sssssss small from http::/convokit. cornell. wttp://cccmmenaatioo//uurreedddd html
0.997
0.5157
http:p/:s///uuggigggggac.//////s roberta-base- Saberte-base-foommlltt--aaaae
Table 7: Inter-rater agreement scores for human evalua- tion results of de-toxification task
wuitterrkknnrderrr---ss-estttti
"htttss:/hhggiingaac co//tonh/toxigenhhte #nttpss//huggiigfacc cttpa/cc...o//oooooeeeeeeooeeooe to//ooh//toxieen robertt
"httpss//persppeciivapp.

---

inter airccescet
Sykdiszzphh
Eveatlessing
Inseeleolesring
Natedlvos
Fa
Lexical
Semantic
Lexical (r)
Semantic (r)
Anippodont's
Fit
-0.02
0.14
-0.02
0.14
Fless
0.502
02504
0.2667
0.3627
Naturalness
-0.03
0.18
-0.03
0.17
Table 8: Inter-rater agreement scores for human evalua- tion results formality transfer task
Table 10: Detoxification task: Spearman rank and Kendall Correlation of non-contextual evaluation met- rics with human judgment
luter eatr apreeneett
Sylcieeeghh
EvenDKeusing
IntralsDlesring
Nateaboaa
Fa
Knjpeddnt'se ko
0.108
0.4M
Lexical
Semantic
Lexical (r)
Semantic
Fit
0.18
0.28
0.17
0.27
Table Inter-rater agreement scores for human evalua- tion results of sentiment change task
Naturalness
0.11
0.26
0.10
0.24
Table 11: Formality task: Spearman rank and Kendall Correlation of non-contextual evaluation metrics with human judgment
D Non-contextual Automatic Evaluation
We present the non-contextual automated evalu- ation results for each task-specific dataset. Fig- ures 16 and 17 illustrate the formality change re- sults for document-level and conversation-lewe datasets, respectively. Figures 18 and 19 display the sentiment transfer results for document-level and conversation-leww datasets, respectively. Fig- ures 27, 21, and 22 depict the de-toxification results for conversational datasets. Notably, all of these fig- ures exhibit similar trends to the aggregate results across all tasks and datasets presented in Figure 3.
Lexical (p)
Semantic
Semantic
Fit
0.11
0.26
0.10
0.25
Naturalness
-0.05
0.13
00.05
0.12
Table 12: Sentiment task: Spearman rank and Kendall Correlation of non-contextual evaluation metrics with human judgment
Teskalig
Seeaatii chostefir (g) Leick (/)
Secaedie (+) Chosieetee)
Narrabco
0.55
Table 13: Detoxification task: Spearman rank and
D.1 Correlation with Human Judgments
Kendall Correlation of contextual evaluation metrics with human judgment
Effective evaluation metrics should yield judg- ments that correlate highly with human judgments, assuming that human evaluators represent a gold- standard. For the human judgments along the di- mensions of naturalness and fit, we map human preferences as follows: 'contextual' to 1, 'tie' to 0, and 'non-contextual" to 11 For the automatic metrics, we assign score of if metric scores the contextual rewrite higher than the non-contextual rewrite, and -1 the metric scores are lower for contextual rewrites.
Lesketio
col CtsSteFlt Sevvuticiol
Lescel (.)
Secauric (s)
0.58
Nataraheo
0.68
0.59
6994
0.8
0.56
1090
Table 14: Formality task: Spearman rank and Kendall Correlation of contextual evaluation metries with human judgment
11
0.88
0.85
0:58
842
853
Nazrabes
For given automatic metric and human judg- ment dimension, we calculate the Spearman rank P correlation and Kendall's for the dataset sam- ples used during the contextual human evaluation $5.1. The correlation scores, ranging from -1 to 1, are obtained by comparing the mapped automatic scores with the mapped human judgment scores. Higher values indicate stronger correlation be- tween the scores obtained using the comparison metric and judgments made by human evaluators. Refer to Tables 12 for the correlation scores non-contextual evaluation metrics with human judgments for each task.
Table 15: Sentiment task: Spearman rank and Kendall Correlation of contextual evaluation metries with human judgment
Contextual Automatic Evaluation
We present the contextual automated evaluation results for each task-specific dataset. Figures 23 and 24 illustrate the formality change results for document-level and conversation-level datasets, re- spectively. Figures 25 and 26 display the sen- timent transfer results for document-level and

---

conversation-level datasets, respectively. Figures 27, 28, and 29 depict the de-toxification results for conversational datasets. All of these figures exhibit similar trends to the aggregate results across all tasks and datasets presented in Figure 3 and they align with the findings from our contextual human evaluation study.
E.1 Correlation with Human Judgments
Similar to $D.1, measure the Spearman rank / correlation and Kendall's correlation for the samples used during human evaluation $5.1. Re- fer Tables 13 15 for the correlation scores non-contextual evaluation metrics with human judgments for each task.

---

Full Instructions
(Expand/Collapse)
Instructions
Thanks for participating this new taskt We are trying to test out a few automatic systems for rewritig sentences to have a desired formality (formal informal).
Given sentence from conversation social media post, desired
formality rewriting level (e.g, formal informal vice and two rewritten sentences, answer the following questions about the rewrites:
facmallyy..accoooo Jeeal? Which
exhibits
formality (formal informat?
example, sentence sounds
politefess might percieved as formail language
formol laguage is usually written with correct grammar and vocabulary, passive voice ottn with abtroviatons, exdamation marts and ingeratives usully formal grammar vocabuuary.
longogge
What are the described events?
Which think closer meaning with respect events in the original utterance (regardless of the formality change??
Who does whom?
level details describe roughly peoplie objects and relationships, iirespective of formality attribute differences. But, it's details the sentence
similar
example, when rewriting news article, important aspects story.
the accuracy what, when, where, why
What intended meaning similarity?
Two sentences are similar their intended meaning they capture roughly the same overall message or purpose. Intended meaning goes beyond the specific details and focuses on maincaining the author's intention, tone, and the implied meoning in the rewritten text. But again, It's okay if few details the sentence get changed bit.
For example, consider original informal utterance- "Hey, wanna grab some grub later?". When this utterance rewritten more formal style, while maintaining its intended meaning asking someone join for meal, might be rewritten "Hello, you like join for later?"
Wull-nultedeess natural
may necessarity be fitting / appropriate relevant/ well-suited given context the dialog document, The rewrite should
required
meaning
original
meaningful and coherent while maintaining flow the conversation.
Examples
Content original Sentence
Instraction
Rewrites
Target Formality
Event Similarity
Intended Meaning
Wes- suited
This to creyy We have people over this world.
the
Reeritt
Fewer peogle art villinn cameros In OCEIP
everpehere glant ship manages
informal
cameraa In octan, courSe. Furhermont the ooban VOST.
many peogre
Arvennas
making mnrediilly
dut. And
REALLY
about inverngetnn ISAF U.S goy.
whee reading tollys news, tound encresting reve
Averitn
the
john Allen light mumerous emalls
The Fill has inttaned an invectgation imoo General
informal
Arvennas
specligtt
following the revetaiion of rumerous
general now under IDD's scrutiyy
dloriingg maiss betawen and
kelly
anouencaal stmus pramae Bustewaynt
Simoatwe is divided ino provicees and
responte
the
to meet the needs of zinoatie's miltion
LAD's lanyers based in harare, working
The current Legail Ard Direszratee ELAC) is stufted by 15 uwyers, based representing the
formal
ALS Divectrate laayers harare who Gablo care of needsorrtee onehe milonn people
Fing
Figure 4: Screenshot of the instructions for human evaluation annotation

---

Task
type)
Njriggnal, type torignnats
Instruction: nowne the origiaa seriencuresprnne from suzyyers to sorgeezza tinguage
Rewrites:
rewrite
QT Pretend The person that the ongnaal sentence, which would preter KErjetz rewreee
Por reurte wisch you'd more rebunal sen stre pit which you'd mentter you mere person who onigina seroece
Strenterd
Nowntee
Ether Optnon
Canner
you your prefermed
most scrayeaz2?
Berh nowntee could work Cannec decde
Qct wiich rewnte ttink clesor desoribed events onginnal semsencee
Chaose serance with the same people chings and inderiani pertoeectiooss the orgg sertenes
Strenite))
Nowrtte
Other Option
ingleed
Qli Which sentence best welk-atttd relewand Flous better/ specifiic with overal preceding conversaiion document
Option
Bach neerta could work Canner decie
stale how much should should changed
changge
Figure 5: Screenshot of the human evaluation annotation task

---

Task
Conversion
Context
Original Sentence
Model
Contextual Rewrite
Non-contextunl Rewrite
>> continue up date this page with funther seloctions find partie- ularly intriguing from the
>> Here are some of them, up- date this page with my preferences.
GPT-3-5
reading hook; 777 Mathematical
> Herez some of them, and further will keep updating this page what I like.
(a) Farmality
Informal Formal
Conversationn Sturters by
John dePillis.
interesting book with lots quotes, poems
Here are some of them, and in later post will keep updating this page with my from the book.
>> Here and further I will update page End interesting.
the
GPT-NeoX
quotes
enjoyed Ni No Kuni people didn't and that's okay. Everyone's a got their opinion!
enjoyed that game. opinions may in- deed sary among individu-
can never again take site that gives the mess and just an awfil game.
Semi-formal Informal
liked that game. It's almost as opinions are subjective.
(b) Formality
liked that gume, de- spite critical opinioes to the contrary. The fict that people enjoyed doesn't make any than anything else.
liked that game. It's almost if opinions
GPT NeoX
relative.
GPT
@USER Yes, ing you to be quict.
very proud accomplishnent yeah
(c) Detexifieation
Toxic
Did Rolando just call me jst up?
USER fuck yeah
Less Toxic
up
GPT-Neoo
user yes, please keep
yeah, Shot
I'm under-
GPT-3.5
stand what you're 'r trying to say. Could you please rephiss??
how somcone is illiterate mies
play games am playin wit 10:50 to 2:42 baby I'm
(d) Detoxification
Toxic
It's so ammssing when Illiterane
Toxic
everywhurre
GPT-NooX
It's anmsing when somcone attempts to insult taunt and they educnted try clearly uneducated about language.
It's funny when the un-
Despite heing an efficient mechanic, Tim
>> He's unfricndly Incked the friendly and humorous domeance expected of service provider.
just want to echo what everyone said about this mechanic... Not only did Tim fix my quickly and inexpensively, he sctually came pick up himselr so that wouldn't t have speed money on a truck!
GPT-3.5
penson who
already
He's also extremely friendly. funny guy who pats you immediately.
makes you feel uneasy right away.
Positive Negative
Sentiment
>> However, Tim's not very friendly or humor- which might make you feel uneacy
He also very rude, unamused guy who you feel ansious.
GPT-NeoX
It very consider- ate you to move so get your bike through. That's great.
How kind of you to
GPT-3.5
be right.
How dure you be inconsideratl That's wrong.
Negative
>> Positive
can get my bike through.
needed understand you
GPT NeoX
you con- säderate! That's right.
you could get your bike through ihank you being thoughtful!
Table Examples from our datasets and corresponding rewrites generated by GPT-3.5 and GPT-NeoX, showcasing all three tasks: formality change, de-toxification, and sentiment transfer. document context, >> sentence to rewrite, >> contextual rewrite. >> non-contextual rewrite; previous turn in conversation, response to rewrite, contextual rewrite response,
non-contextual rewrite response

---

Lexical
Samautic
Flumny
rewrite type
Style
ROU
MET
WER
BERT-S
SBERT
Smatch
PPL
contextual
0.19
0.40
2.14
0.92
0.62
0.51
38.37
0.59
GPT-3.5
non contexttllt
0.28
0.49
0.91
0.94
0.73
0.67
43.40
0.58
random-context
0.18
0.35
3.17
0.91
0.52
0.47
GPT-NeoX
contextual
0.26
0.42
1.88
0.91
0.60
0.47
44.59
0.42
non-contextual
0.45
0.63
0.72
0.95
0.80
0.67
44.80
0.35
Fandom-context
1.90
0.91
0.41
Table 16: Non-contextual Automatic Evaluation Results on Formality: Document-lewe context from CNN/DailyMail Blog Authorship Corpus
Laxical
Semantic
Plwenry
model
rewrite type
Style
ROU
MET
WER
BERT-S
SBERT
Smatch
PPL
contextual
0.16
0.38
2.67
0.90
0.67
0.45
33.78
0.68
GPT-3.5
non-contextual
0.22
1,23
0.91
0.72
0.53
40.06
0.67
random-context
0.32
3.72
0.43
GPT-NeoX
contextual
non-contextual
randem-context
0.27
0.40
2.70
0.90
0.60
0.44
Table 17: Non-contextual Automatic Evaluation Results on Formality: Conversational context comprised of Reddit threads
Lexical
Semautic
Plumay PPL
model
rewrite type
Style
ROU
MET
WER
BERT-S
SBERT
Smatch
GPT-3.5
non-contextual
0.63
context
GPT-NeoX
contextual
0.27
0.43
1.43
0.91
0.56
0.41
57.6 64
0.49
non-contextual
0.45
0.60
0.67
0.94
0.74
0.62
73.02
0.49
Fandem-context
0.29
0.44
1.37
0.91
0.56
0.42
Table 18: Non-contextual Automatic Evaluation Results Sentiment: Document-level context comprised of Yelp Reviews
Samantte
Plueney
model
rewrite type
Style
BERT-S
Smatch
contextual
0.30
0.54
1.03
0.91
0.63
0.53
36.31
0.69
GPT-3.5
con-contextual
0.45
0.67
0.65
0.93
0.75
0.68
42.82
0..6
randem-cootin context
D.30
0.52
1.07
0,91
0.59
0.53
GPT-NeoX
contextual
0.16
0.30
1.66
0.87
0.43
0.22
42.39
0.35
non-contextual
0.33
0.48
0.81
0.90
0.59
0.50
64.09
0.25
random-context
0.18
0.33
1.63
0.88
0.43
0.30
Table 19: Non-contextual Automatic Evaluation Results on Sentiment: Conversational context from DailyDialog
dataset
model
rewnite type
Lexical
Semonilc SBERT
Flurncy PPL
Style MuteRoberio
Style HoteBur
Style Persyeetive
ROU
MET
WER
BERT-S
Smatch
0.45
GPT:3.5
non-coentettual
0.24
0.41
0.90
0.91
0.72
0.51
40.98
0.01
0.47
0.07
zundom cuntext
0.17
0.32
0.97
0.57
0.43
OPE-NooX
0.32
0.78
0.60
0.46
63.04
0.61
0.13
non consextual
0.44
0.52
0.61
0.92
0.71
0.57
67.47
0.10
0.59
0.15
randor-contesr
0.73
Table 20: Non-contextual Automatic Evaluation Results on Toxicity: Conversational context from CCC dataset

---

model
rewrite type
Lexioni
Semoniic
Flurey
Style HuteRoberia
Style
Style Persyeettive
ROU
MET
WER
BERTSS
SBERT
Smalch
PPL
contextmual
1.18
0.87
0.45
75.48
0.04
0.31
0.11
GP1:3.
non contextual
0.12
0.34
0.99
0.88
0.56
0.47
78.23
0.05
0.34
0.12
rundom-contee
0.08
0.28
1.24
0.86
0.42
0.40
GPE.Nee
0.18
0.28
1.29
0.87
0.45
0.39
80.10
0.39
0.62
0.35
nom coniixulll
0.32
0.49
0.91
0.90
0.67
0.54
106.66
0.52
0.74
0.46
andor--contes
Table 21: Non-contextual Automatic Evaluation Results on Toxicity: Conversational context from MDMD dataset
model
type
ROU
Lexiical MET
WER
BERT-S
Semantic SBERT
Smatch
Flunncy PPL.
Style HateRoberia
Styfe NateBerr
Style Porspective
0.29
GPT-3.5
non-conteettul
0.11
0.29
0.97
0.1
0.52
0.41
33.00
0.14
0.40
0.09
rundorn context
0.05
0.19
1.61
0.25
0.29
GPe-NeX
non-conneextual
0.43
0.59
0.66
0.94
0.72
0.63
37.90
0.64
0.79
0.38
rundern context
0.25
0.40
1.06
0.9
0.48
0.44
Table 22: Non-contextual Automatic Evaluation Results on Toxicity: Conversational context from ProsocialDialog
dataset
model
type
Levicul MET
Semuniic SBERT
Coherence P
Cohesiveness NSP
Cusiom CtxSinFit
ROU
WER
BERT-S
Smatch
contextual
0.16
0.23
0.89
0.90
0.61
0.38
22.05
0.94
0.93
GPF-3.5
non-centextual
0.13
0.20
0.87
0.89
0.50
0.33
32.79
0.87
0.91
random-context
0.10
0.16
1.06
0.87
0.38
0.29
34.24
0.69
0.80
GPT-NooX
contextual
0.25
0.28
0.83
0.90
0.62
0.37
20.51
0.97
0.94
non-ccntextual
0,23
0.24
0.84
0.89
0.52
0.33
32.15
0.94
0,94
Table 23: Contextual Automatic Evaluation Results on Formality: Document-level context from CNN//ailyMail Blog Authorship Corpus
Lesical MET
Semuniic <BERT
Coherence PPL
Cohexiveness NSP
Cusiom CtxSinFit
model
rewrite type
ROU
WER
BERT-SUx
Smatche
contextual
0.14
0.27
0.91
0.89
0.66
0.37
28.82
0.88
0.89
GPT-3.
non-scntextual
0.14
0.24
0.95
0.88
0.56
0.36
41.02
0.82
0.87
GPT-NooX
non-contextual
0.36
$2.45
random-context
0.17
0.22
1.29
0.87
0.46
0.31
53.98
0.80)
0.85
Table 24: Contextual Automatic Evaluation Results Formality: Conversational context comprised of Reddit threads
Lesical MET
Semuniic TB
Coherence PPL
Coheviveneas NSP
Cusiom
model
rewrite type
ROU
wER
BERT-S
Smatch
GPT-3.5
non-contextual
0.16
0.24
rundom-content
0.07
0.12
0.93
0.86
0.38
0.19
44.13
0,82
0.86
GPT NooX
contextual
0.13
0.16
0.90
0.87
0.47
0.23
33.05
0.96
0.93
non-contextual
0.17
0.18
0.87
0.88
0.44
0.22
48.59
0.93
0.93
undom-content
0.12
0.15
0.91
0.87
0.41
0.20
53.44
0.91
0.91
Table 25: Contextual Automatic Evaluation Results on Sentiment: Document-level context comprised of Yelp
Reviews

---

Lexical
Semantic S
Coherence
Cohesivenesy
Custom
model
rewrte type
ROUS
MET
Smatches
GPT-333
nen-contextual
0.28
0.35
0.81
0.89
0.54
0.41
50.60
0.92
0.93
rundom-context
0.20
0.30
0.89
0.88
0.45
0.38
54.10
0.87
0.89
GPT-NooX
contextual
0.17
0.26
0.97
0.86
0.46
0.34
32:45
0.88
0.88
non-contextual
0.20
0.24
0.87
0.87
0.42
0.21
60.31
0.86
0.88
rundom-comtext
0.12
0.21
1.04
0.85
0.34
0.21
41.69
0.79
0.83
Table 26: Contextual Automatic Evaluation Results on Sentiment: Conversational context from DailyDialog dataset
Levical MET
Semantic SBERT
Coherivemes
model
rewrite type
ROU
WERe
BERT-S
Smatches
PPL
NSP
CtxSinFit
contextual
0.16
0.24
0.86
0.88
0.61
0.36
28.45
0.95
0.93
GPT-335
non-gontextual
0.17
0.25
0.85
0.88
0.57
0.35
37.40
0,91
0.91
random-ccnteex
0,12
0.20
0.90
0.87
0.47
0.31
38.96
0.89
0.89
GPT-NeoX
contextual
0.24
0.25
0.82
0.88
0.54
0.34
37.41
0.96
0.93
non-contextual
0.29
0.29
0.77
0.89
0.54
0.32
$1.56
0.92
0.92
random-context
0.21
0.23
0.84
0.87
0.44
0.32
52.24
0.89
0.90
Table 27: Contextual Automatic Evaluation Results on Toxicity: Conversational context from CCC dataset
Lesical MET
Semuntic SBER
Cokerence eP
Cohesivenesz NSP
Custom CtxSinFit
model
rewrite type
RO
wER
BERT-S
Smatches
contextual
0.10
0.22
0.91
0.86
0.50
0.34
49.50
0.96
0.92
GVT-335
non-contextual
0.08
0.19
0.92
0.86
0.40
0.31
70.91
0.86
0.87
rundom-contextt
0.06
0.16
0.96
0.84
0.29
0.29
71.09
0.82
0.84
GPT-NooX
contextual
0.20
0.25
0.85
0.87
0.52
0.35
40.79
0.93
0.90
non-contextual
0.19
0.26
0.87
0.87
0.47
0.36
90.64
0.89
0.90
random-context
0.09
0.15
0.97
0.85
0.30
0.26
92.21
0.76
0.81
Table 28: Contextual Automatic Evaluation Results on Toxicity: Conversational context from MDMD dataset
Levioal MET
Semantic SBERT<
Coherence PPL
Cohesrveness NSP
Custom CtySinFit
model
rewrite type
ROU
WER
Smatchee
contextual
0.09
0.18
0.90
0.89
0.54
0.29
14.89
0.98
0.93
GPT-3.5
non-contextual
0.06
0.12
0.93
0.88
0.36
0.21
29.75
0.89
0.90
random-context
0.03
0.11
0.95
0.86
0.21
0.22
32.58
0.84
0.86
GPT-NooX
contextual
0.19
0.23
0.85
0.89
0.52
0.32
18.98
0.94
0.93
non-contextual
0.20
0.22
0.85
0.89
0.44
0.30
34.42
0.88
0.91
candon-context
0.12
0.17
0.90
0.88
0.26
36.32
0.84
Table 29: Contextual Automatic Evaluation Results on Toxicity: Conversational context from ProsocialDialog
dataset

---

Here the preceding context for some text: (While reading todays news, I found an interesting text about the investiaaton on the ISAF and U.S guy.
General John Allen.
Hore the text: (The general is now in the FBI's spotlight after discovering lots of mails between him and Jiil Kalley.) )
Here rewrite the text from informal formal langauge, under the given context.
(The FBI has initiated an investiatton into General John Allen in light of the numerous emails exchanged between him and Jiil Kelley) #dd
Here the preceding context for some text: (From the fourth policy year onwards, your boat is insured for its current value. This sailing ground covers
waters the Baltic, including Kattegat and Skagerrak, well the European rivers and inland waterways.)
Here the text: (In estuaries, an imaginary continuation of the coastine forms the boundary between the inland waterways and the neighbouring sea.)
Here rewrite of the text from formal to informal language, under the given context.
(In the estuaries, protend line along the shore marks the line between the canals and the soa. |
eee
Here the preceding context some text: (YACHTING24 does not pass on any data third parties. }
Hare the text: (Compliance with data protection regulations monitored
in-house data protection officer your interests.)
Here rewrtee of the text from formal to informal language, under the given context.
(We don't give out your data to anyone else, and our data protection officer keeps eye on everything make sure your info's secure.) #id
Here the preceding context some text: (If arrested with warrant the defendant must be brought police station charging station soon possible.) }
Hare the text: (A defendant may detained for more than 48 hours unless magistrate authorizes extension of the detention.)
Here rewrtee of the text from formal to informal language, under the given context.
(The defendant can't be held for more than two days unless a judge says it's okay)
#ide
Here the preceding context for some text: (Zimbabwe is divided into provinces and cites with provincial status (Harare and Bulawayo).
Hare the text: (The current Legal Aid Directorate (LAD) is staffed by 15 lawyers, all based in Harare, rapresenting the needs Zimbabwe's million citizens)
Here rewrite of the text from formal to informal language, under the given context.
(The Legal Aid Directorate has 15 lawyers in Harare who take care of the needs of the 12 million people living Zimbabwe.)
#9d
Here the preceding context for some text: (I heard that trans fat is bad for the body. Why that?? in our daily food?)
Hare the text: (Where can
Here rewrite of the text from informal formal language, under the given context.
(What foods are likely to contain trans that we consume on a daily basis?) see
Here the preceding context for some text: (It was because the number of centroids have is actually larger than 1M so doesn't let me
howeevrr I'm not entirely sure what number of centroids need either.)
Hare the (I'm just directly keeping the ratio of centroids total vectors constant from earlirr coda.)
Here rewrite of the text from informal formal language, under the given context.
am maintaining a consistent ratio of centroids to total vectors from my prior code, despite the number centroids exceeding one million.)
#ee
Here the preceding context for some text: (In the san francisco bay area, does make sense to rent or buy ?}
Hare the text: [the prices of rent and the price buying does not make sense to me, mostly the rent will not cover the mortgage)
Here rewrite of the text from informal to formal language, under the given context.
(The costs renting and purchasing property the San Francisco Bay Area do not appear to be balance, rent payments generally do not cover mortgage.)
#ee
Here the preceding context some text: (I have very small stuff stuck under my keyboard and prevents from working)
Hore the text: (What will be the best way to clean It?)
Here rewrite of the text from informal to formal language, under the given context.
(What would be the most efficient method of removing the debris that is obstructing the keyboard?)
Here the preceding context for some text: (Blushing unique blend of evolutionary and social behavior) Here the text: (It's involuntary reaction of the sympathetic nervous system, which is responsible for our "fight or flight" response, but blushing is solely triggered social cues.)
Here rewrite of the text from informal formal language, under the given context.
(Blushing an involuntary physiological response of the sympathetic nervous system, which typically activated in response to a perceived fear threat. However, blushing activated exclusively by social stimull.) #ee
Figure 6: Formality: 10-shot prompting examples for GPT-NeoX

---

Statement: While reading todays news, found an interesting text about the investigation the ISAF and U.S guy, General John Allen. Response: The general is now in the FBI's spotlight after discovering lots of mails between him and Jill Kelley.
Rewrite the Response from informal to formal language, under the given context.
Rewrite: The FBI has initiated an investigation into General John Allen in light of the numerous emails exchanged between him and Jill Kelley.
Statement: Zimbabwe is divided into 8 provinces and cities with provincial status (Harare and Bulawayo).
Response: The current Legal Aid Directorate (LAD) staffed by 15 lawyers, all based in Harare, representing the needs Zimbabwe's 12 million citizens
Rewrite the Response from formal to informal language, under the given context.
Rewrite: The Legal Aid Directorate has 15 lawyers in Harare who take care of the needs of the 12 million people living Zimbabwe.
Figure 7: Formality: 2-shot prompting examples for GPT-3.5


Reviews:
[{"review_id": "JzmxPvdhuQ", "reviewer": "Reviewer_5HXe", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback on our paper, especially for considering that our novel metric  \"would be useful for evaluation community\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper defined and proposed a new question, that is contextual style transfer and find it is better aligned to human preference compared to previous non-contextual approaches.\n2. The paper investigated the comparison with non-contextual evaluation metrics, and found they were not correlated well to the human annotations, which is none of surprise though, and they propose a way to integrate context to improve these metrics.\n3. The paper also proposes a new metric that both considers the semantic similarity and cohesiveness, that yield the best results. It would be useful for evaluation community.\n4. The paper is well organised and written. Nice and clear tables and figures.\n5. The experiment is sound, three tasks are considered, and a lot experiments are done, also the analysis is good to me.   ", "reasons_to_reject": "No reason to reject in my opinion. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes to integrate textual context into rewriting and evaluation stages of stylistic text rewriting, and propose a new evaluation metric: CtxSimFit.\nThey show that human prefers contextual rewrites over non-contextual ones, and the contextual evaluation metric correlates better to human evaluation.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vfwKOoqiid", "reviewer": "Reviewer_XbBa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s insightful comments and for the overall positive endorsement of our paper. \n\n1. `Sensitivity and intuition for the hyperparameter alpha in CtxSimFit`\\\n\\\nThank you for this question. We chose to present CtxSimFit with alpha=0.5 in the paper to emulate an equal weightage of contextual cohesiveness and semantic similarity to the original sentence (meaning preservation) for a general case of a stylistic rewrite. However, we agree that different use cases might require different weightings of both components, which we will add to the paper. We present the reviewer with some preliminary sensitivity analysis of the hyperparameter alpha: CtxSimFit = $\\alpha$∗BERTSCORE(S,X) + (1-$\\alpha$) * NSP(C, X)\n\\\n\\\nA) When alpha is small ($\\alpha$ < 0.5): there is more emphasis on the contextual cohesiveness measured by the BERT next sentence prediction (NSP). We notice that contextual rewrites are scored much higher than non-contextual rewrites. Further, the Spearman correlation  $\\rho$ with human judgments of ‘overall fit’ is significant and around the range of 0.65-0.77, suggesting alignment with human preferences. \n\\\nFor example for $\\alpha=0.2$, the Spearman $\\rho$ = 0.74, $p < 0.01$ (averaged across all tasks).\n\\\n\\\nB) When $\\alpha$=0.5 (presented in the paper): there is an equal weightage on the contextual cohesiveness (NSP) and semantic similarity via BERTSCORE. Spearman correlation with human judgments of ‘overall fit’ is $\\rho$ = 0.85 $p < 0.01$, showing stronger alignment with humans than smaller alpha. \n\\\n\\\nC) When alpha is large ($\\alpha$ > 0.5):  there is a greater emphasis on the semantic similarity to the original sentence i.e. meaning preservation (non-contextual metric), hence the contextual and non-contextual rewrites are scored fairly equally by our metric. Further, the Spearman correlation  $\\rho$ with human judgments of ‘overall fit’ is slightly significant  $p < 0.05$ for alpha values closer to 0.5 and non-significant farther away from 0.5, suggesting that this isn't well-aligned with humans' preferences. \n\\\nFor example for $\\alpha=0.7$, the Spearman $\\rho$ = 0.46, $p < 0.05$ (averaged across all tasks).\n\\\n\\\nWe will expand upon this and include details about hyperparameter sensitivity in the final version of the paper. \n\\\n&nbsp;\n2. `Underlying meaning or interpretation of the linear combination of the two elements in CtxSimFit` \\\n\\\nWe propose CtxSimFit as a metric that combines contextual cohesiveness and semantic similarity to assess the quality of a rewritten sentence. The linear combination of these two elements offers a projection into both vector representation spaces and, therefore could lead to better human alignment. This approach is commonly used in NLP, as seen in the BERT model where token, position, and segment embeddings are added to obtain a richer input representation. In our case, we are considering a one-dimensional scenario.\n\\\n\\\nWe include the hyperparameter alpha to allow users to have better control over their preferences for meaning preservation vs. contextual fit. This is particularly useful in creative use cases of stylistic rewriting, where users may prioritize contextual cohesiveness over content preservation. On the other hand, for tasks such as formality transfer, users may value meaning preservation more than they would for detoxification tasks. \n\\\n&nbsp;\n3. `Clarification on the lack of alternative methods to compare`\\\n \\\nWe acknowledge that we relied on few-shot prompting of LLMs to incorporate textual context. We specifically chose LLMs because they have demonstrated strong rewriting capabilities, both with and without the use of textual context  [1]. Other existing generative models, such as those used for chit-chat and goal-oriented conversational agents, as well as pretrained language models, have struggled with effectively understanding and utilizing preceding textual context [2, 3, 4, 5]. Conversely, custom-made rewriting models from prior research often lack the modeling of context [6, 7]. Therefore, our decision to focus on LLMs was made as an initial investigation. We will add the lack of alternate custom or smaller models in our limitations section. \n\n_References:_ \n\n[1] Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.\n\n[2] Sankar, Chinnadhurai, Sandeep Subramanian, Christopher Pal, Sarath Chandar, and Yoshua Bengio. \"Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study.\" In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 32-37. 2019.\n\n[3] O’Connor, Joe, and Jacob Andreas. \"What Context Features Can Transformer Language Models Use?.\" In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 851-864. 2021.\n\n[4] Parthasarathi, Prasanna, Joelle Pineau, and Sarath Chandar. \"Do Encoder Representations of Generative Dialogue Models have sufficient summary of the Information about the task?.\" In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 477-488. 2021.\n\n[5] Su, Hsuan, Shachi H. Kumar, Sahisnu Mazumder, Wenda Chen, Ramesh Manuvinakurike, Eda Okur, Saurav Sahay, Lama Nachman, Shang-Tse Chen, and Hung-yi Lee. \"Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue.\" arXiv preprint arXiv:2302.05888 (2023).\n\n[6] Ma, Xinyao, Maarten Sap, Hannah Rashkin, and Yejin Choi. \"PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction.\" In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7426-7441. 2020.\n\n[7] Dale, David, Anton Voronov, Daryna Dementieva, Varvara Logacheva, Olga Kozlova, Nikita Semenov, and Alexander Panchenko. \"Text Detoxification using Large Pre-trained Neural Models.\" In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7979-7996. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper contains a thoroughly evaluated empirical claims, with a wide coverage of tasks and perspectives. The comparisons of different metrics and their correlations to human perceptions are presented in an easy-to-read manner, with clear writing. Despite the simple and straightforward method to incorporate textual context (and lack of alternative methods to compare), the method itself is evaluated thoroughly in design and also solidly enough in statistical manners to convince the readers the need of contextual element in stylistic rewrite tasks. Lastly, the presentation of what can be supported in the main text and the clear listing of limitations make the reader easy to determine the exact coverage of the research presented in this paper.\n", "reasons_to_reject": "As CtxSimFit is a newly proposed metric that combines BERTSCORE and probability from BERT NSP head, a careful empirical validation on how sensitive the metric is to the relative weight hyperparameter alpha should have been presented. Also, Albeit a minor issue, the overall depth of insight would have been deeper, had there been theoretical exploration on the underlying meaning or interpretation of the linear combination of the two elements consisting the new metric CtxSimFit. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper claims the empirical need to incorporate context information in stylistic text rewriting tasks, not only in the text generation but also (and more importantly) in the evaluation stage. To support this argument, three rewriting tasks (formality transfer, sentiment transfer, toxicity adjustment) over two types of contexts (conversation and document setting) are explored, and three human evaluators validated the human-level perception of the rewrites. Also, this paper finds out that the correlation between human perception and pre-existing metrics can be improved with a newly proposed evaluation metric ``CtxSimFit''.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "io3uPOij0L", "reviewer": "Reviewer_joB3", "summary": "", "questions": "A. Line 326: Does this n-gram comparison include the context? If not, I wonder if it really makes sense to penalize contextual rewrites that borrowed words not in the original sentence from the context.\n\nB. Would human preference drastically change if you removed sentence context? I'm wondering whether the consistency in meaning strongly overshadows other characteristics that make up a good sentence. (For examples, are humans more forgiving with grammatical errors with the contextual rewrites?)", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and positive feedback on “having a metric that is easier to use but is also correlated with crowdsourced data would be useful to the community”.\n\n1. `Existing non-contextual automatic metric correlation with human preferences`\\\n\\\nWe agree with the reviewer that these automatic metrics are widely used in evaluating stylistic rewriting. However, what we illustrated in our paper is that these widely used metrics have focused on sentence/statement level non-contextual evaluation, often failing to align to account for the preceding textual context which is necessary for better representing realistic downstream use cases [1]. \n\\\n\\\nWe illustrate this via our contextual human evaluation study. Though we agree with the reviewer that the size of the test set is small, we show that **our results are statistically significant** through a binomial test. With these statistically significant human preference results, we compare correlations (Spearman and Kendall tau) of both non-contextual and our proposed contextualized automatic metrics with human preferences of ‘overall fit’ and ‘naturalness’ dimensions, across three stylistic rewriting tasks. \n\\\n\\\nWe agree with the reviewer that we demonstrate the shortcomings of existing non-contextual automatic only for the three tasks - formality, sentiment, and detoxification. We will modify the language in the paper to reflect and clarify this. However, we are optimistic about the relevance and need for contextual automatic metrics in other generation tasks, given initial work on the relevance of context in evaluating open-domain dialogue generation [2, 3]. \n\\\n\\\n[1] Yeh, Yi-Ting, Maxine Eskenazi, and Shikib Mehri. \"A Comprehensive Assessment of Dialog Evaluation Metrics.\" In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pp. 15-33. 2021.\\\n[2] Welleck, Sean, Jason Weston, Arthur Szlam, and Kyunghyun Cho. \"Dialogue Natural Language Inference.\" In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3731-3741. 2019.\\\n[3] Pang, Bo, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. \"Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3619-3629. 2020.\\\n\\\n&nbsp;\n2. `L326 Event-level Similarity: “Does this n-gram comparison include the context?” `\\\n\\\nWe define “Event-level Similarity” ranking as “the rewrite that most effectively retains the essential events, entities, and relations present in the original sentence” (L272), and thus **does not include the context**. We intentionally do not include context for “event-level similarity” since we believe that non-contextual lexical and semantic automatic metrics emulate this dimension. We hoped to separate and contrast it against the “overall fit” and “naturalness” dimensions, which do consider context, to see the effect of the inclusion of extra entity/event details in contextual rewrites on the overall preference of human annotators. Through this experiment, we showed that non-contextual rewrites are most aligned with the entity/event details in the original sentence, yet annotators still prefer contextual rewrites for their naturalness and fit. We agree with the reviewer that contextual rewrites should not be penalized for “words not in the original sentence from the context” - hence our argument for inclusion and usage of contextual automatic metrics for stylistic rewriting as existing non-contextual metrics do indeed penalize contextual rewrites for including details from the context (Section 6). \n\\\n&nbsp;\n3. `Would human preference drastically change if you removed sentence context? Does “consistency in meaning strongly overshadow other characteristics that make up a good sentence” `\\\nYes, we believe that human preferences would change if we excluded the previous context and the results would be similar to the “event-level similarity” preference dimension where non-contextual rewrites are preferred to contextual rewrites. We agree with the reviewer that there is a greater emphasis on “meaning” in our human evaluation experiment, and lesser emphasis on other characteristics like grammatical errors (which are very uncommon with modern LLMs). However, we consider “Fluency” via non-contextual perplexity (Section 6.1 - Metrics considered for non-contextual automatic evaluation) as a proxy to measure grammatical errors and sentence quality [4, 5, 6]. We see that both contextual and non-contextual rewrites generate decently fluent rewrites, regardless of context. \n\\\n\\\n[4] Holtzman, Ari, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. \"Learning to Write with Cooperative Discriminators.\" In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1638-1649. 2018.\\\n[5] Xu, Jingjing, Xuancheng Ren, Junyang Lin, and Xu Sun. \"Diversity-promoting GAN: A cross-entropy based generative adversarial network for diversified text generation.\" In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 3940-3949. 2018.\\\n[6] Ma, Xinyao, Maarten Sap, Hannah Rashkin, and Yejin Choi. \"PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction.\" In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7426-7441. 2020.\\\n\\\n&nbsp;\n4. `Presentation improvements`\\\nThank you for your feedback, we will address the figure beautification change.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I agree that current, most commonly used automatic text revision metrics are lacking, and would be interested in applying their new metric. Collecting human preference data can be time-consuming and expensive and having a metric that is easier to use but is also correlated with crowdsourced data would be useful to the community. The paper is also well-written and easy to read.", "reasons_to_reject": "While not perfect, automatic text revision metrics are an established method that have been used widely for evaluating text. Claiming that it's not correlated with human preference is a strong statement given the size of the test set and there is no evidence to show that this claim holds true outside of evaluating formality, toxicity, and sentiment. \n\n", "questions_to_authors": "A. Line 326: Does this n-gram comparison include the context? If not, I wonder if it really makes sense to penalize contextual rewrites that borrowed words not in the original sentence from the context.\n\nB. Would human preference drastically change if you removed sentence context? I'm wondering whether the consistency in meaning strongly overshadows other characteristics that make up a good sentence. (For examples, are humans more forgiving with grammatical errors with the contextual rewrites?)", "paper_topic_and_main_contributions": "This paper brings attention to how most existing work on text revision focuses on the text to be revised and not its surrounding sentences. It talks about how context is an important factor for humans, both when rewriting text or evaluating revisions, and suggests a new metric that takes original sentence similarity and contextual coherence into account. The authors apply their work to multiple datasets that provide different styles with respect to formality, toxicity, and sentiment. The revised sentences are collected from two GPT models and preference data of whether the annotator prefers contextual or non-contextual edits are collected from MTurk. Their findings show that humans prefer contextual rewrites and the metric proposed here aligns more strongly with human preference.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A. Figure 2: It would be helpful to briefly mention that the numbers in the bars are representing the proportion of those who voted for a particular category."}, "extra_text": {}}, {"review_id": "DcCW8tE9NG", "reviewer": "Reviewer_fCPE", "summary": "", "questions": "Q1: Among Tables 12-14, there is a lower correlation for the CtxSimFit score in the Sentiment task (Table 14). It would be helpful to explain the reasons behind this observation and discuss any possible implications for future research or improvements.\n\nQ2: Are there a taxonomy of the errors made by (non)contextual models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and positive feedback on “the paper's recognition of the importance of context in both stylistic re-writing generation and evaluation is crucial for the field ”.\n\n1. `Line 206: Choice of LLMs and validity of results for smaller models`\\\nWe incorporated textual context using few-shot prompting of LLMs, which have shown strong rewriting capabilities with and without context. Existing generative models for chit-chat or goal-oriented dialogue agents struggle to effectively incorporate context. On the other hand, custom-made models often do not consider context. Please see response 3 to Reviewer [XbBa](https://openreview.net/forum?id=In4L79U5n7&noteId=43bvMBdDLS) for additional details on the choice of LLMs. \n\\\n\\\nAs for the validity of the results for smaller models, we believe the results still hold since there is some preliminary research [1] that explored context in small custom-trained seq2seq rewriting models which showed that contextual rewrites are preferred by human annotators. \n\\\n\\\n[1] Cheng, Yu, Zhe Gan, Yizhe Zhang, Oussama Elachqar, Dianqi Li, and Jingjing Liu. \"Contextual Text Style Transfer.\" In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2915-2924. 2020.\n\\\n&nbsp;\n2. `Line 452: Definitions of  \"Coherence\" and \"Cohesiveness\"; Clarification on using pplx^Ctx for coherence`\\\nThe terms coherence and cohesion in linguistics are commonly defined as follows [2]: \n\\\n*Cohesion:* sentence pairs fitting together the way two pieces of a jigsaw puzzle do\n\\\n*Coherence:* what all the sentences in a piece of writing add up to, the way all the pieces in a puzzle add up to the picture on the box\n\\\n\\\nHowever, these concepts have varying definitions and are slightly adapted differently for various tasks. We computed coherence in our study by measuring the perplexity of the rewrite conditioned on the context using GPT-2, based on the above linguistic definitions and prior research [4, 5]. We appreciate the reviewer for mentioning the work of Fabbri, Alexander R., et al [6],  which is specific to summarization and suggests that metrics like ROUGE-4 and character-based n-gram overlap are highly correlated with coherence. In our paper, we included ROUGE-2 under \"lexical similarity\", and in the final version, we will also add ROUGE-4 to our evaluation suite.\n\\\n\\\nSimilarly following prior work based on these definitions [7], we compute cohesiveness using the probabilities from the next sentence prediction head of BERT which measures if the context followed by the rewrite is fitting. \n\\\n\\\nWe will acknowledge these varying definitions in our paper and clarify the specific meaning we intend to convey.\n\\\n\\\n[2] J.M. Williams and G.G. Colomb. 1995. Style: Toward Clarity and Grace. Chicago guides to writing, editing, and publishing. University of Chicago Press.\\\n[3] See, Abigail, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. \"Do Massively Pretrained Language Models Make Better Storytellers?.\" In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 843-861. 2019.\\\n[4] Pang, Bo, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. \"Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3619-3629. 2020.\\\n[5] Fabbri, Alexander R., Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. \"Summeval: Re-evaluating summarization evaluation.\" Transactions of the Association for Computational Linguistics 9 (2021): 391-409.\\\n[6] Sugiyama, Hiroaki. \"Dialogue breakdown detection using BERT with traditional dialogue features.\" In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction: 10th International Workshop on Spoken Dialogue Systems, pp. 419-427. Springer Singapore, 2021.\n\\\n&nbsp;\n3. `Line 468: BERT’s Next Sentence Prediction (NSP) head for Cohesiveness`\\\nIn addition to the masked language modeling pre-training objective, BERT has an additional “next sentence prediction” (NSP) task that jointly pre-trains text-pair representations. In other words, NSP is a classification head on top of the BERT model. We measure cohesiveness or if the rewrite follows the context via the probabilities obtained from the NSP head of the BERT model. We refer the reviewer to the BERT paper [7] and huggingface documentation [8] for additional details.\n\\\n\\\n[7] Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. \"Bert: Pre-training of deep bidirectional transformers for language understanding.\" arXiv preprint arXiv:1810.04805 (2018).\\\n[8] https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForNextSentencePrediction\n\\\n&nbsp;\n4. `Sensitivity analysis for hyperparameter alpha in CtxSimFit`\\\nWe chose to present CtxSimFit with alpha=0.5 in the paper to emulate an equal weightage of contextual cohesiveness and semantic similarity to the original sentence (meaning preservation) for a general case of a stylistic rewrite. We performed a preliminary sensitivity analysis of the hyperparameter alpha: \\\nCtxSimFit = $\\alpha$∗BERTSCORE(S,X) + (1-$\\alpha$) * NSP(C, X)\n\\\nA) When alpha is small ($\\alpha$ < 0.5), contextual cohesiveness measured by BERT next sentence prediction (NSP) is emphasized more. Contextual rewrites are scored higher compared to non-contextual rewrites. The Spearman correlation with human judgments of 'overall fit' ranges from 0.65-0.77, indicating alignment with human preferences. \n\\\nB) When $\\alpha$=0.5 (presented in the paper), there is an equal weightage on the contextual cohesiveness (NSP) and semantic similarity via BERTSCORE. Spearman correlation with human judgments of ‘overall fit’ is $\\rho$ = 0.85 $p < 0.01$, indicating even stronger alignment with humans than smaller alpha values.\n\\\nC) When alpha is large ($\\alpha$ > 0.5), there is a stronger focus on preserving the meaning of the original sentence (non-contextual metric), resulting in comparable scores for both contextual and non-contextual rewrites. The Spearman correlation $\\rho$ with human judgments of 'overall fit' is slightly significant ($p < 0.05$) for alpha values close to 0.5 but not significant for values farther away, indicating a lack of alignment with human preferences. \n\\\nWe refer the reviewer to response 1 to Reviewer [XbBa](https://openreview.net/forum?id=In4L79U5n7&noteId=43bvMBdDLS) for additional details.  \n\\\n&nbsp;\n5. `Intuition for CtxSimFit`\\\nWe frame CtxSimFit as the linearly weighted combination of contextual cohesiveness and semantic similarity. The combination of these two elements through a linear projection allows for improved human alignment within both vector representation spaces. Furthermore, the hyperparameter alpha offers users better control over their preferences for contextual cohesiveness versus semantic similarity. \n\\\nWe refer the reviewer to response 2 to Reviewer [XbBa](https://openreview.net/forum?id=In4L79U5n7&noteId=43bvMBdDLS) for additional details.\n\\\n&nbsp;\n6. `Reasoning for lower correlation of CtxSimFit score in the Sentiment task`\\\nSection 5.2 of our paper discusses and supports the notion, as indicated by prior research [1], that the Sentiment style transfer task may be ill-defined. Though the task of stylistic text rewriting is to maintain the original intended meaning while adapting to a different style, sentiment style transfer demands a change in both meaning and style (esp. in reviews where the reason why a review was written was to convey the original sentiment).  We believe this is the reason why the results of human evaluation for the sentiment rewriting task have lower annotator agreement compared to toxicity and formality tasks. Furthermore, the low agreement scores could explain the generally low correlation of our CtxSimFit metric with human preferences.\n\\\n\\\nFor future work, we believe there is a need for a separate set of evaluation metrics that attempt to extract event/entity-specific content, and event-independent content and weigh these signals according to the task requirements.  Some preliminary work has been done in this area [9], and we hope that our work motivates further research in this field.\n\\\n\\\n[9] Yu, Ping, Yang Zhao, Chunyuan Li, and Changyou Chen. \"Rethinking sentiment style transfer.\" In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1569-1582. 2021.\n\\\n&nbsp;\n7. `Taxonomy of errors made by (non)contextual models`\\\nWe briefly discuss this in our paper under Section 8 - Context is crucial for rewriting.  Non-contextual models often struggle with certain aspects that contextual rewriting models excel at.\n\\\nFor formality style transfer, non-contextual models may have difficulty retaining context details, disambiguating entities (as shown in example (a) in Table 2), enhancing cohesiveness (as demonstrated in example (b)), and improving vocabulary usage (as seen in example (c)). \n\\\nIn detoxification, non-contextual models may fail to preserve intended meanings (as illustrated in example (d)), maintain politeness consistency (as showcased in example (e)), and show demographic sensitivity by generating less offensive rewrites (shown in example (f)).\n\\\nSimilarly, in sentiment transfer, non-contextual models may struggle to resolve ambiguity caused by sentiment change (as exemplified in example (g)) and fail to generate engaging rewrites (as seen in examples (h) and (i)). \nWe will elaborate on this qualitative analysis in the final version of the paper. \n\\\n&nbsp;\n8. `Presentation improvements`\\\nThank you for your feedback, we will incorporate these tables in the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n- The paper's recognition of the importance of context in both stylistic re-writing generation and evaluation is crucial for the field.\n- The evaluation of existing non-contextual and contextual systems, especially against human judgment, is a valuable empirical effort that establishes the state-of-the-art in this area.\n- The introduction of CtxSimFit as a new score that correlates better with human judgment appears to be effective.\n\nBased on these merits, I strongly recommend accepting this paper (Excitement=4, Soundness=4).", "reasons_to_reject": "Weaknesses:\nAs a non-expert in the field of stylistic re-writing, I can only identify general weaknesses, and upon my initial review, I find no reason to reject this paper.\n\nChallenges:\n\n- Line 206: The authors mention utilizing large language models (LLMs) and in-context learning for the task. It would be beneficial to explain why they specifically chose these approaches and whether their conclusions would still hold for smaller models.\n- Line 452: The terms \"Coherence\" and \"Cohesiveness\" are formal linguistic concepts that require careful definitions in the given context. Please provide a clear definition here. Additionally, there are concerns about using pplx^Ctx as a proper metric for evaluating text coherence, given the availability of specialized methods for this purpose [1].\n- Line 468: What do you mean by NSP head? Since the equation at Line 474 requires a score calculated by NSP(C, X), but NSP head is a compoment in neural network architecture. What's more, there are more clarifications to be made here: (1) Why $\\alpha$ is set to be 0.5? Won't other values yield better results? (2) What is the intuition behind this formula? Is it modeling coheisveness or coherence?\n\n[1] Summeval: Re-evaluating summarization evaluation. TACL 2021.", "questions_to_authors": "Q1: Among Tables 12-14, there is a lower correlation for the CtxSimFit score in the Sentiment task (Table 14). It would be helpful to explain the reasons behind this observation and discuss any possible implications for future research or improvements.\n\nQ2: Are there a taxonomy of the errors made by (non)contextual models?", "paper_topic_and_main_contributions": "The paper focuses on the significance of contextual modeling for stylistic re-writing generation and evaluation.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Tables 12 to 14 are important results that demonstrate CtxSimFit's correlation with human judgment. It would be beneficial to find a way to incorporate these tables into the main text for better visibility and understanding.\n"}}]

Author rebuttal:

Meta-review: This paper highlights the importance of context in stylistic text rewriting, proposing the integration of textual context into rewriting and evaluation stages. It introduces the CtxSimFit evaluation metric, showing that human preference aligns with contextual rewrites and that this metric better correlates with human evaluation. The research explores various rewriting tasks and contexts, emphasizing the significance of contextual modeling in stylistic rewriting.

Reasons to accept:
- The paper introduces the concept of contextual style transfer, which is better aligned with human preference compared to previous non-contextual approaches, bringing a novel perspective to the field.
- Thorough investigation of non-contextual evaluation metrics and their lack of correlation with human annotations, along with a proposal to integrate context for improvement.
-Introduction of a new metric, CtxSimFit, which considers both semantic similarity and cohesiveness and demonstrates superior results, offering a valuable addition to the evaluation community.
-The paper is well-organized and well-written, featuring clear tables and figures that enhance readability.
-Sound experimentation with three tasks, extensive experiments, and insightful analysis.
-Clear presentation of research scope and limitations, aiding readers in understanding the paper's coverage and contributions.

Reasons to reject:
-Lack of empirical validation regarding the sensitivity of CtxSimFit metric to the relative weight hyperparameter alpha, and a need for theoretical exploration of the metric's underlying meaning.
-Insufficient evidence to support the claim that existing automatic text revision metrics are not correlated with human preference beyond the specific evaluation of formality, toxicity, and sentiment.
-Potential issues with the use of large language models (LLMs) and in-context learning without a clear explanation of their selection and their applicability to smaller models.
-Unclear definitions of formal linguistic concepts such as "Coherence" and "Cohesiveness,".
-Lack of clarification and insight regarding the NSP head and the choice of alpha=0.5 in the formula.
Decision: Accept-Main