Text Fact Transfer
Authors: Nishant Balepur
Jie Huang
Kevin Chang
Venue: EMNLP 2023 Main (2023)
Keywords: Text Generation
Factuality
Text Style Transfer
Information Extraction
TL;DR: We propose the new task of text fact transfer, a complementary task to text style transfer which seeks to transfer factual content while preserving style, along with a model to solve our task

Abstract: Text style transfer is a prominent task that aims to control the style of text without inherently changing its factual content. To cover more text modification applications, such as adapting past news for current events and repurposing educational materials, we propose the task of text fact transfer, which seeks to transfer the factual content of a source text between topics without modifying its style. We find that existing language models struggle with text fact transfer, due to their inability to preserve the specificity and phrasing of the source text, and tendency to hallucinate errors. To address these issues, we design ModQGA, a framework that minimally modifies a source text with a novel combination of end-to-end question generation and specificity-aware question answering. Through experiments on four existing datasets adapted for text fact transfer, we show that ModQGA can accurately transfer factual content without sacrificing the style of the source text.

Text Fact Transfer
Nishant Balepur
Jie Huang Kevin
Chang
University of Illinois at Urbana-Champaign, USA (balepur2, jeffhi, kcchang)@illinois.com
Abstract
Source Text
Source Topic: Joseph Stalin
Targei Topic: hisson Kandilli
Target Text
Joseph Stalin belongs the
Nelson Mandela belongs ANG
Text style transfer is prominent task that aims control the style of text without inherently changing its factual content. To cover more text modification applications, such as adapting past news for current events and repurposing edu- cational materials, we propose the task of text fact transfer, which seeks to transfer the factual content of source text between topics without modifying its style. We find that existing lan- guage models struggle with text fact transfer, due their inability to preserve the specificity and phrasing of the source text, and tendency hallucinate errors. address these issues, we design ModQGA, framework that mini- mally modifies a source text with a novel com- bination of end-to-end question generation and specificity- aware question answering. Through experiments on four existing datasets adapted for text fact transfer, can accurately transfer factual content without sacrificing style of the source text.!
Communist Party
Facts on:
Manons
GPT-3.5 Nelson Mandela was of African National Congress
See2Seeq
Nelson Mandela belongs Rhodesia
Nelson Mandela belongs to ANC
Figure I: (Top) Overview of text fact transfer. (Bottom) Outputs from the seq2seq LED language model, O-shot GPT-3.5, and O-shot ModQGA (ours) text fact trans- fer. Red highlighted text indicates factual inaccuracies or failure match the style of the source text.
(Graefe, 2016) and repurposing educational materi- als for new subjects (Kaldoudi et al., 2011), which are outside the scope of text style transfer. Further, studying methods to transfer facts while preserv- ing style could be useful for augmenting datasets, i.e., expanding training sets with new, factual train- ing examples in a similar style (Amin-Nejad al., 2020; Bayer et al., 2023), or evaluating the factual accuracy of text generation models (Celikyilmaz et al., 2020; Ji et al., 2023).
show that ModQGA
1 Introduction
Text style transfer aims to control the stylistic at- tributes of text, such as sentiment or formality, with- out affecting its factual content (Jin al., 2022; Hu et al., 2022). This task has several applications, including personalizing dialogue agents (Rao and Tetreault, 2018; Zheng et al., 2020), increasing per- suasiveness in marketing news (Jin et al., 2020; Moorjani et al., 2022), simplifying educational resources (Wang et al., 2019; Cao et al., 2020).
To address these needs, we propose the task of text fact transfer, which aims modify the factual content of source text while preserving its style. We define factual content as topic-specific entities that convey knowledge and style as how the factual content is phrased and organized, as well as its level of specificity?. As shown in Figure (top), given as inputs source text, source topic, target topic, and corpus of facts for the target topic, we seek to generate target text that matches the style of the source text and contains factual content specific the target topic. Thus, while text style transfer aims to modify subjective, stylistic aspects of text, text
While text style transfer models can adeptly al- ter stylistic elements, they do not address all text modification needs, especially those centered on factual modifications. Specifically, there exist sev- eral applications that require the transfer of factual content beiween topics without altering style, such adapting past news articles for current events
Dependingg on the setting, this definition of style may need to be modified, For example, in educational repurposing, it may he infeasible to keep the phrasing consistent, as diffcrent subjects may need to he discussed and phrased differently.
'Code is available at https: https://githube http://giiiubuoooooooeoooeooeeet http//balepprr text-fact text-faat-traanfer.. transfer.

---

fact transfer controls the objective, factual content. One approach for text fact transfer (on parallel corpora) to train/prompt seq2seq or large LMs (Lewis et al., 2020a; Brown et al., 2020). However, there are two inherent challenges to text fact trans- fer that cannot be overcome by directly applying these models. First, the generated text must not de- viate from the wording of the source text, but LMs may not always succeed in this regard (Balepur al., 2023). For example in Figure 1, GPT-3.5 states that Nelson Mandela "was a member of"" the ANC, which inconsistent with the phrasing of "belongs to" present in the source text.
"Communist Party" to "ANC" directly is difficult, but we can easily transfer "What is Joseph Stalin's s party?" to "What is Nelson Mandela's s party?" by replacing the source topic (Joseph Stalin) with the target topic (Nelson Mandela), returning question that can be used to retrieve the entity "ANC."
Exploiting these findings, we design ModQGA, model that minimally modifies the source text with a combination of Question Generation (QG) and Answering (QA). As shown in Figure 2, Mod- QGA first uses end-to-end QG to jointly produce entities from the source text and questions that can be answered by said entities. Next, these questions are transferred to pertain to the target topic. Mod- QGA then uses specificity-awant QA to retrieve an answer from the corpus for each transferred ques- tion, while matching the specificity of the source text entities. Finally, these answers are filled into the source text. Solely modifying factual entities allows for the preservation of the phrasing of the source text, while the focused approach of transfer- ring entities with specificity- awaar QA promotes factuality and matched specificity, as shown in Fig- ure 1. Further, we can train the QG and QA models of ModQGA on external QA datasets (Rajpurkar et al., 2016), resulting in O-shot model that can be applied diverse domains without extra training.
Second, along with being accurate, the factual content must align with the specificity of the source text to best maintain its style. However, LMs have been shown to hallucinate (Ji et al., 2023) and strug- gle to control the specificity of their outputs (Huang al., 2022). For example, as seen in Figure 1, the seq2seq LM states that "Nelson Mandela belongs Rhodesia." Although the leader has some links to Rhodesia, it is inaccurate to state that he belongs there. Further, the source text contains the political party of Joseph Stalin (i.e., Communist Party), so the target text should contain a political party (i.c.. ANC) rather than a country, which is less specific. Further, these challenges become more complex if supervised text fact transfer is infeasible. For ex- ample, when adapting past news for current events or augmenting datasets, it could take ample time and effort to construct parallel corpora and train a supervised model. In such cases, 0-shot models, while harder to develop, are preferred, as they can adapt to domains without extra training. Hence, we must study 0-shot and supervised text fact transfer models to ensure adaptability in downstream tasks.
We showcase the strength of ModQGA for text fact transfer by creating four parallel corpora from existing datasets, spanning expository text genera- tion (Balepur et al., 2023) and relationship triples (Elsahar et al., 2018; Gardent et al., 20. Hence, our initial study of text fact transfer focuses on the adaptation of expository texts and relationship triples, leaving applications such as repurposing news articles and dataset augmentation for future research. Using these datasets, we design a O-shot and supervised version of ModQGA and in our ex- periments, find that both models outperform their respective baselines in style preservation and factu- ality on a majority of datasets.
To address these challenges of text fact transfer, we extend the concept of minimal alterations for text style transfer proposed by Li et al. (2018) and seek to execute the two-step process of: (1) locat- ing factual entities in the source text; and (2) solely transferring these entities between topics. To per- form step one, we note that factual entities are in- herently question question-driven, and thus any entity in the source text that must be transferred can answer a question. For example in Figure 1, the factual entity "Communist Party" answers the question "What is Joseph Stalin's party?"'. To perform step two, we find that transferring entities between topics chal- lenging, but transferring questions that can retrieve said entities is simple. For example, transferring
Our contributions can be summarized as follows:
1) We propose the task of text fact transfer, which aims to alter factual content while preserving style. 2) To solve our task, we design ModQGA, which minimally modifies source text with an ensemble of end-to-end QG and specificity-aware QA. We qualitatively assess the latter, which shows at least some ability to control the specificity of its answer. 3) We adapt four datasets for text fact transfer.
4) Through experiments on our four datasets, we

---

demonstrate that ModQGA generates factual text that is stylistically consistent with the source text.
emplar. Apart from lack of focus on transferring factual content, these works differ from our task they do not leverage a factual corpus.
The task most similar to ours is expository text generation (ETG) (Balepur et al., 2023), which seeks to generate factual text from corpus in a consistent style. However, ETG dictates that this style is learned from examples of outputs in the same domain, while text fact transfer adheres to the style of a single source text. Hence, an ETG model is Nomain-specific, while a single text fact trans- fer model (c.g., O-shot ModQGA) could be used in several domains. Further, the IRP model pro- posed by Balepur et al. (2023) for ETG combines content planning, retrieval, and rephrasing, while ModQGA modifies source text with question gen- eration and answering. and our model tackles the additional problem of controlling specificity ($3.3).
2 Related Work
2.1 Text Style Transfer
Text style transfer aims to modify the style of text without inherently affecting its content (Fu al., 2018; Jin et al., 2022; Hu al., 2022). The concept of style can take many forms, including formality (Wang et al., 2019; Zhang et al., 2020), sentiment (Prabhumoye et al., 2018; Yang et al., 2018), and authorship (Xu et al., 2012). Text fact transfer is the counterpart to text style transfer, as we focus on transferring the factual content of text between topics without affecting its underlying style. Hence, our task emphasizes generating new, factual text, which is not the main focus of style transfer tasks.
Several methods have been developed for text style transfer, such as training neural models on parallel corpora (Rao and Tetreault, 2018; Xu et al., 2019), latently disentangling content and style (Hu et al., 2017; Shen et al., 2017), or prototype editing (Li al., 2018; Sudhakar et al., 2019; Abu Sheikha and Inkpen, 2011). ModQGA most similar the Reeerate model (Li et al., 2018), 8), which extracts attribute markers, transfers attributes across styles, and generates an output. We apply a similar technique for text fact transfer, but notably use novel combination of end-to-end question generation and specificity-aware question answer- ing, which has not been explored in prior work.
2.3 Analogy Completion
The concept of transferring entities between topics is similar to analogy completion (Ushio et al., 2021; Bhavya et al., 2022; Chen et al., 2022), which aims to select a word that parallels an input query-word pair (e.g.. "Paris:Franee Lima:|MASS: [MASK]"). While analogy completion could be used for factual entity transfer, this is only one aspect of text fact transfer. Further, our task is fundamentally a text generation task, while analogy complction is typically used to assess how models internally capture relations.
Methodology
Given a source text D.,, source topic f,, and target topic 4, text fact transfer aims to produce a target text D, that matches the style of D, and modifies the entities related to the source topic L. with enti- ties related to the target topic t.- To serve ground truth information for 4.. we also provide corpus of factual sentences C related to the target topic tr As illustrated Figure 2, the backbone of Mod- QGA consists of two key modules: (i) An end-to- end question generator ((2//pg.. 1x) that produces question/entity pairs (q, C O. where each g can be answered by e using the source text De; and (ii) A specificity-aware question answering model p((a,, a,)le, q, that extracts answer span (at, ay) from the context e (where c C), which answers question q and matches the speci- ficity of the entity e. After training these models, ModQGA performs text fact transfer via: 1) end- to-end question generation with pl Q, (D, tx); 2)
2.2 Stylistic Exemplars
Recent work has studied models that leverage stylis- tic exemplars to guide stylistic choices in text gen- eration (Cao et al., 2018; Wei et al., 2020). Such exemplars improve the fluency of seq2seq mod- els in various tasks, including summarization (Dou al., 2021; An et al., 2021), machine translation (Shang et al., 2021; Nguyen et al., 2022), dialogue generation (Zheng et al., 2020; Wang et al., 2021), and question answering (Wang et al., 2022).
More relevant to text fact transfer are tasks that require strictly adhering to the style of an exemplar. Chen et al. (2019) propose the task of controllable paraphrase generation, which aims to combine the semantics from one sentence and the syntax from a second sentence. Lin et al. (2020) introduce "style imitation" and perform data-to-tex gener- ation while strictly maintaining the style of an ex-

---

Generated Questions
Transferred Questions
Question Transfer ($3.2)
What location of UIUC?
What the location Stanford?
What UIUC?
What is the size Stanford?
Factual
Inputs
Corpus
Target Topic: Stanford
End-to-End Question Generation (53.1)
Specificiy-Aware Question Answering ($3.3)
Source Topic: UlUC
Factual Entities
Transferred Entities
Output
Source Text: UIUG has city location and campus size 1,783 acres.
Sourse Text Infiling ($3.4)
city
California
suburban
Target Text: Stanford has
suburban location and its campus size 3,180
1,783 acres
33 sq km
3,180 acres
Figure 2: Overview of ModQGA, First, ModQGA jointly generates questions and factual entities with an end-to-end question generator. These questions are then transferred to pertain to the target topic. Next, ModQGA performs specificity- aware question answering to retrieve answers from the factual source that match the level of specificity of the source entity (correct/incorrect matches in green/red). Last, ModQGA infills the source text with these answers.
question transferring; 3) question answering with p( (,)lc, g, e); and 4) source text infilling. We will describe each these steps followed by how they are combined for the full ModQGA model.
property, we only keep SQuAD entries where the topic is a substring of the question. Thus, all training questions contain the topic t,, teaching the model to produce i, in the output during inference.
To ensure all factual entities in the source text are detected, use nucleus sampling to generate question/entity pairs Q, {(gj- e,) with each sentence of the source text D, as the context and source topic i, as the topic Thus, each unique factual entity from the source text may be mapped to multiple questions, which are ensembled by the specificity-awarr question answering model ($3.3). As a final post-processing step, we discard pairs in Qa with entity that does not appear in the source text, as this entity is hallucinated. Further, if an entity substring of another entity in Q.. we discard the substring (shorter) entity.
3.1 End-to-End Question Generation
Our approach to text fact transfer is rooted in the observation that all entities in the source text that need to be transferred can be viewed as an answer to question. For example, given the source text "Tbuprofen is used to relieve pain," transferring be- tween the topies Ibuprofen and Melatonin may re- sult in the text "Melatonin is used to promote sleep." " The part of the source text that needs to be trans- ferred (apart from the known transfer of "Tbuprofen" to "Melatonin") is "to relieve pain," which can an- swer the question "Why is Ibuprofen used?". This question-answer paradigm helps us guide the mod- ification process in text fact transfer.
3.2 Question Transferring
Hence, to identify entities that need to be trans- ferred and the questions that can be answered by said entities, we train an end-to-en question gener- ation model that jointly generates entities and their questions from a context. To do so, we leverage the SQuAD-V2 dataset (Rajpurkar et al., 2016). We train BART-large (Lewis et al., 2020a) to minimize the loss Agg of token prediction of question q and answer (entity) e, surrounded by 
 and < answer |> tokens (represented as (q e)), conditioned on the context e and topic
Each question qj found in the question/entity pairs (gj, e,) E Q., pertain to the source topic t.. but we require a transferred question of that pertains to the target topic 4, Since qj will contain the substring 1g, we can simply replace f, with f, to obtain %f Through testing on our validation sets, we also find that generic queries can outperform specific queries when retrieving contexts for question an- swering. For instance, we find that the generic query "What the hub of the airport?" outper- forms the specific query "What the hub of Cathay Pacific Airport?". We find this occurs because the inclusion of the topic 4 in the query distracts the retriever when searching C for contexts in QA, it is more biased towards facts that contain the to-
App
logp((g-e))le, (q.c)..... (g 1). (1)
If a generated question q contains the source topic ta q can be simply transferred to the target topic & by replacing t, with t, To elicit this desirable
We found that performing lexically constrained token decoding (Hokamp and Liu, 2017) with topic f, led to similar outcome, but this resulted in higher GPU memory usage.

---

kens in le even when said facts are not relevant to the query." We show the benefit of generic queries experimentally with ablation studies ($5.3).
context e, question 9, and specificity guidance
N x log pgpaaaee i),
(2)
To obtain a generic question dl'. we take the in- tersecting tokens of Lj and 1/, which eliminates the topic-specific tokens found in t, and tt. Combin- ing the specific and generic questions, we obtain set of transferred questions and their corresponding source entities Q {dly, ((d',
>3 Σ
log pogalayla, =)),
(3)
Aya A
(4)
where II the indicator function and is the num- ber of tokens in the input sequence.
3.3 Specificity-Awarr Question Answering
After creating the transferred question-entity pairs Qs we faithfully answer each transferred question by retrieving a context from the factual source followed by extractive question answering (QA). However, an off-the-shelf QA model cannot be used for our task, as it fails to consider the speci- ficity of the answer we require (Huang et al., 2022). To extract transferred entities that are stylistically aligned with the source text, we seek answers with the same level of specificity as the entities they are replacing. For example, at one step in ModQGA, we may obtain the question "Where Stanford located?" derived from the source entity "rural". While "California," "Palo Alto," and "suburban" are all valid answers, "suburban" is the best choice, it shares the same level of specificity as "rural," and thus best matches the style of the source text.
For each transferred question/source text entity pair (g. e) E Q1. we first use Contriever (Izacard et al., 2022) to obtain the context c. i.e., the top-k most relevant facts in C via maximum inner- product search (Shrivastava and Li, 2014). Next, the question q. entity e (specificity guidance). and context e are fed through the specificity aeear QA model p((ss, aj) q, q)e). We record the predicted answer a = (a,, Q²) with the highest likelihood (sum of start and end likelihoods) under length m. We map each unique entity in Q: to the answer with the highest total likelihood. This process returns map & with each source text entity the key and its transferred entity a as the value.
3.4 Source Text Infilling
Lastly, we infill the source text D,, replacing each entity with its mapped entity a in E. We describe zero-shot and supervised infilling methods below: Zero-shot: Given that each entity e appears in the source text D,,, we replace every occurrence of e with a and la with l to create the target text DD Supervised: We train the LED language model (Beltagy al., 2020) to generate the target text D, using the source topic t,, source text D., tar- get topic 4, corpus C, and each transferred entity (surrounded by <| answer |> tokens). This pro- cess is similar to keyword-guided text generation techniques (Mao et al., 2020; Narayan et al., 2021). O Overall, the supervised version of ModQGA allows the model to have more flexibility during infilling. Although ModQGA is designed primarily as a 0-shot text fact transfer model, using custom com- ponents trained on external SQuAD datasets, alter- ing the infilling process allows us to fairly compare our model with supervised baselines ($4.2).
To create a dataset with these specifications, we again modify the SQuAD-VS V2 dataset (Rajpurkar et al., 2016). The dataset already provides ques- tions, contexts, and answer spans, but we still re- quire guidance to match the specificity levels of the answers. We find that one way to obtain specificity guidance of an answer is through the skip-gram assumption (Mikolov et al., similar words are discussed in similar contexts. For example, intuit that because "rural" and "suburban" are used in the same context (e.g., "the location is [subur- ban/ruralJ"",, they have similar specificity levels. Hence, we obtain specificity guidance for each an- swer in the SQuAD dataset by replacing every word in the answer with random top-20 skip-gram syn- onym via fastText embeddings (Joulin et al., 2017).
We use BERT-large (Devlin et al., 2018) train our specifieity-aware QA model p((a;, ag) |c. q, e). We minimize Aga: the sum of X₁ the cross-entropy loss of the predicted start index ag and Aj. the loss of the predicted end index aj. conditioned on the
3.5 The ModQGA Framework
In Algorithm 1, we use the above components design ModQGA. First, ModQGA performs end- to-end question generation with the BART model
*We note that the specific query can find the facts with top-k retrieval for large k but our QA model is trained to use fewer contexts (I: = 5), hence our need for generic queries.

---

(222, r, v2)) and use #1 - yn the source text and 22 ..-3 r- 122 as the target text denotes concatena- tion). We use X1 and #2 as the source and target topics. For t-REX, we use the Wikipedia texts in the dataset for C, and for Google, we scrape sen- tences from the top-7 web pages queried with 22.
Algorithm ModQGA
procedure MODQGA(D, Ce.
2:
Qa 0.2. Q, < (, (map: E (A
3:
while [2.1 do
4:
2, 2, EEEEEEEEEEE QG)
2 2; QUESTIONTRANNSFRRRR.. 2S.
6:
for (q.e) Q do
7:
C CONTRIEVER( C)
8:
a, scure cA.QA(
9:
d' score' + E(e) if score score' then Ele) + (a, score)
& Lookup in E map
4.2 Baselines
10:
11:
Update best answer
We compare zero-shot ModQGA (0-shot Mod- QGA) with the following zero-shot baselines:
12:
Dj < (D..E)
13:
return D:
1) 0-Shot GPT: We use a zero-shot prompt (Ap- pendix A.3) instructing GPT-3.5 to create the target text using the source text, source topic, and target topic. This model uses its internal knowledge. 2) O-Shot GPT+Retr: We add the top-5 retrieved facts from C as an extra input to 0-Shot GPT.
p( 2. D.) 42), to generate question/entity pairs
Q, covering the factual content of the source text
D.. ModQGA then transfers the questions in Q, from the source topic tg to the target topic & to cre- ate Q. which has specific and generic questions. For each transferred question and source entity e in 2t, ModQGA performs specificity-aware QA with the BERT model p((a;, aj)lc, q, e). We build the map ε, containing each source entity mapped to the answer with the highest likelihood, to repre- sent its transferred entity. Last, using &, ModQGA infills the source text D, to create the target text D,, either in a O-shot or supervised manner.
3) SourceCopy: We trivially copy the source text as the predicted output for the target text.
When parallel data exists in text style transfer, seq2seq models are typically used (Jin et al., 2022). Thus, for our parallel text fact transfer setting, we compare supervised ModQGA (ModQGA-Sup) with the following supervised seq2seq models:
1) z-Shot GPT: We construct a z-shot prompt for GPT-3.5 to generate the target text with the source text, source topic, and target topic inputs. This model relies on its internal knowledge.
4 Experimental Setup
2) >-Shot GPT We add the top-5 retrieved facts from C as an extra input >-Shot GPT.
We provide detailed setup in Appendix A.
4.1
Datasets
3) LED: LED (Beltagy al., 2020) seq2seq
LM based on the Longformer. LED produces the target text using the source text, source topic, target topic, and corpus as inputs. This model is Mod- QGA-Sup without the transferred entities as inputs. 4) BART+Retr: Similar to RAG (Lewis al., 2020b), we retrieve the top-25 facts from C and train BART to generate the target text using the source text, source/target topics, and retrieved facts. All GPT-3.5 models are gpt-3. popt-3.5-tte 5-turbo with a temperature of 0.2. Models that perform retrieval use the same Contriever setup (Izacard et al., 2022) as ModQGA. The input query used is the source text D3 with every occurrence of l, replaced with t4. We found that this query outperforms solely the target topic 4, as it provides the Contriever context as to which information to search for (see Table 6).
We adapt the following tasks and datasets to con- struet parallel corpora for text fact transfer: 1) Expository Text Generation (ETG) uses topic- related sentences to create multi-sentence factual texts in a consistent style (Balepur et al., 2023). We adapt the U.S. News and Medline datasets, span- ning college and medical domains. We use the out- put text as the target text and retrieve/create training examples for the source text (see Appendix A. We use the document titles for the source/target topics, and the provided corpus for C. 2) Relationship triples have a subject x, pred- icate y. and relation r between and We adapt the t-REX (Elsahar et al., 2018) 8) and Google (Orr, 2013; Petroni et al., 2019) relationship triple datasets. t-REX contains open-domain relations, while Google contains biographical relations. The open-domain nature of t-REX allows us to as- sess the udaptability of each baseline. We obtain triples that share a relation r (i.e., (#), r, (1) and
4.3 Quantitative Metrics
We measure the output similarity of the predicted and target texts with ROUGE-1/2 (R1/R2) and

---

Dataset
Model
R1
R2
BLEU
Halluc
FactCC
NLI-Ent
Length
O-shot ModQGA (Ours)
0.934
0.890
0.865
0.29
0.650
0.708
1.01
U.S. News
0-Shot GPT
0.881
0.814
0.774
4.84
0.489
0.420
1.03
0-Shot Ree+
0.832
0.767
0.679
3.65
0.534
0.587
1.16
SourceCopy
0.795
0.682
0.00
0.220
0.185
1.00
0-shot ModQGA (Ours)
0.724
0.605
0.579
0.00
0.915
0.502
0.97
Medline
0-Shot GPT
0.732
0.599
0,486
0.91
0.958
0.447
1.29
0-Shot GPT+Retr
0,476
0.338
0.176
0.69
0.825
0.231
2.60
SourceCopy
0,539
0.417
0,400
0.00
0,890
0.034
1.00
0-shot ModQGA (Ours)
0.929
0.914
0.857
0.82
0.589
0.621
1.00
Google
0-Shot GPT
0.794
0.670
3.56
0.245
0.200
1.01
0-Shot RRe+
0.698
0.609
0.315
2.50
0.502
0.222
1.89
SourceCopy
0.455
0.350
0.082
0.00
0.078
0.000
1.02
0-shot ModQGA (Ours)
0.841
0.781
0.721
0.58
0.722
0.609
1.05
I-REX
0-Shot GPT
0.780
0.699
0.490
6.75
0.798
0.538
1.30
0-Shot GPT+Rctr
0.585
0.483
0.157
3.89
0.739
0.261
3.15
SourceCopy
0.497
0.376
0.350
0.00
0.004
0.017
1.00
Table 1: Quantitative comparison of zero-shot text fact transfer models oulput similarity (R1, R2, BLEU) and
factuality (Hallue, FacCCC, NLI-Ent). Best results are in bold, barring SourceCopy Hallue, as it will always be 0.
Dataset
Model
R1
R2
BLEU
Halluc
FactCC
NLI-Ent
Length
MedQGA-Sup (Ours)
0.967
0.953
0.944
0.33
0.901
0.889
1.01
3-Shot GPT
0.883
0.819
0.787
5.24
0.357
0.430
1.03
U.S. News
7-Shot GPT+Retr
0.909
0.863
0,848
0.422
0.482
1.02
1.FD
0.958
0.941
0.933
1.05
0.838
0.815
1.02
BART+Retr
0.892
0.839
2.94
0.669
0.652
1.00
ModQGA-Sup (Ours)
0.870
0.807
0.785
0.22
0.976
0.725
0,99
3-Shot GPT
0.778
0.668
0.589
1.17
0.969
0.584
1.16
Medline
7-Shot GPT+Retr
0.721
0.606
0.568
0.49
0.927
0.460
1.04
LED
0.850
0.780
0.760
0.30
0,962
0.725
0.98
BART+Retr
0.817
0.732
0.716
T.05
0.955
0.605
1.01
ModQGA-Sh (Ours)
0.947
0.937
0.899
1.52
0.737
0.714
1.00
3-Shot GPT
0.846
0.809
0.630
1.32
0.546
0.415
1.17
Google
10-Shot Get+
0.812
0.773
0.614
4.50
0.541
0.467
1.14
LED
0.938
0.926
0.878
1.54
0.683
0.661
1.00
Reer
0.943
0.932
0.890
1.04
0.732
0.696
1.00
ModQGA-SS (Ours)
0.833
0.761
0.735
0.65
0.661
0.539
0,98
3-hhot GPT 10-Shot GPT+Retr
0.710
0.598
9.67
0.862
0.5(1)
1.25
tEEX
0.742
0.666
0.499
5.88
0.591
0.536
1.29
LED
0.816
0.753
0.720
0.66
0.670
0.478
1.03
RAtr+Retr
0.883
0.835
0.812
0.83
0.835
0.722
1.01
Table 2: Quantitative comparison of supervised text fact transfer models in output similarity (R1, R2, BLEU) and factuality (Hallue, FactCC, NLI-Ent). Best results are in bold, second best results are underlined.
BLEU (Lin, 2004; Papineni et al., 2002), serving proxies for style preservation of the source text.
All metrics are reported from a single run.
Results
To evaluate factuality, we adopt three metrics: 1) Hallue calculates the average percentage of to- kens that extrinsically hallucinated, meaning that they do not appear in the corpus C or source D,; 2) FactCC (Kryscinski al., 2020) is a classifier that predicts any factual errors exist between a source text and claim. We use the true output as the source and each sentence of the gener- ated text as the claim, and report the proportion of sentences with no factual errors; 3) NLI-Ent uses textual entailment to predict whether a claim is tailed by source (Maynez al., 2020). We train DistilBERT (Sanh et al., 2019) classifier on the MNLI dataset (Williams al., 2018) (accuracy of 0.82) and report the proportion of sentences in the generated text that are entailed by the true output.
5.1 Quantitative Performance
In Table 1, that O-shot ModQGA excels text fact transfer, achieving the strongest results in 22/24 metrics. This is impressive given that ModQGA has significantly less parameters than GPT-3.5 (0.8B vs 175B). We also note that O-shot ModQGA outperforms ModQGA-Sup on open- domain t-REX, showing that our O-shot model is more adaptable than its supervised version, but is surpassed by BART+Retr, opening the door search in 0-shot text fact transfer to close this gap. In Table 2, find that ModQGA-Suu outper- forms baselines on three datasets (17/18 metrics on U.S. News/Mecdiin/Gooole))) and achieves the sec- ond strongest results on t- t-REX. Further, ModQGA-

---

Zero-Shot Ours
Supervised Equal
Model
R1
R2
BLEU
FactCC
NLI-Ent
Metric
Equal
GPT
Ours
LED
Full Model
0.724
0.605
0.579
0.915
0.502
U.S. News-Style
59.0
28.0
13.0
3.0
91.0
6.0
No Generic
0.680
0.553
0.527
0.889
0.215
U.S. News-Fact
47.0
49.0
4.0
46.0
49.0
5.0
Normal QA
0.686
0.562
0.522
0.825
0.278
Google-Style
86.0
11.0
3.0
6.0
94.0
0.0
Google-Fact
13.0
78.0
9.0
14.0
84.0
2.0
Table 4: Ablation study for O-shot ModQGA on Med-
line. No Generic removes generic questions during ques- tion transferring, and Normal QA uses a BERTT-baed question answering model without specificity guidance.
Table 3: Pairwise comparison of O-shot ModQGA vs O-Shot GPT+Retr and ModQGA-Sup vs LED. We evalu- ate with Factuality/Style on U.S. News/Google. Results are averaged w.r.t. annotators and reported a percent. The favored models ignoring ties (Equal) are in bold.
Question: What is the setting of the University of Florida?
Entity: urban
Entity: Tucson, Arizona
Answer: residential
Answer: Tallahassee, Florida
Sup surpasses LED in 23/24 metrics, meaning that our extra input of transferred entities is valuable for improving the style and factuality of seq2seq mod- els in text fact transfer. These findings suggest that strategy of identifying entities, transferring en- tities between topics, and infilling, can outperform generating text solely in a seq2seq manner.
Entity: Philadelphia
Entity: The Nation's Capital
Answer: Tallahasses
Answer: The State Capital
Entity: East Answer: North
Entity: Oklahoma
Entity: 200
Answer: Tallahassee
Answer: 185
Figure 3: Examples of our QA model altering the speci- ficity of its answer depending the entity. The ques- tion was obtained by running ModQGA on U.S. News. Green/red text indicates correct/incorrect answers.
Finally, we note that GPT-3.5 fails to produce factual text, obtaining much lower factuality scores that are not always improved by using the corpus C. The LLM also struggles to adhere to the style of the source text, shown by the lower output similar- ity scores and larger length ratios. Thus, text fact transfer highlights the limitations of GPT-3.5 with preserving factuality and style, meaning that our task could benchmark these capabilities of LLMs.
accuracy of seq2seq models. These findings paral- lel our quantitative results ($5.1), reinforcing that ModQGA effectively transfer factual content without sacrificing the style of the source text.
5.3 Ablation Studies
We conduct an ablation study (Table 4, full results Appendix 8) and note that the use of generic ques- tions and specificity guidance improve the output similarity and factuality of O-shot ModQGA. We find the specificity result to be noteworthy, it means controlling specificity can enhance the per- formance of O-shot text fact transfer frameworks.
5.2 Human Evaluation
We invite two computer science and engincering students to evaluate 50 generated outputs from U.S. News and Google on style (i.e. which output best matches the source text style) and factuality (i.e. which output is more factual). Following best prac- tices, we use a pairwise comparative evaluation (Lewis et al., 2020b). To study the issues of O-shot LLMs, we compare 0-shot ModQGA and O-Shot GPT+Retr, and to study if the extra inputs of trans- ferred entities aid seq2seq models, compare ModQGA-Sup and LED.
5.4 Specificity-Awar QA Analysis
In Figure 3, we assess the abilities of our specificity- aware QA model. Overall, we find that the model does use the specificity of the entity guidance, hav- ing the ability to provide a regional descriptor ("res- idential"), city ("Tallahassee"", city and state ("Tal- lahassee, Florida"), and city descriptor ("The State Capital"). This suggests that our model has at least some ability to control the specificity of its answers. Despite these strengths, our QA model may still err. Specifically, the model may identify part of the context that matches the specificity of the entity, even though it does not correctly answer the ques- tion (e.g.. "North" comes from the context "North side of campus," " but the correct answer is "South"). Further, the model may be biased towards answers
In Table 3, the evaluator ratings indicate that ()- shot ModQGA better preserved style compared to O-Shot GPT+Retr in over 55% of cases on both datasets and was more factual on U.S. News in 47% of cases, highlighting that ModQGA is a pre- ferred choice for the challenging task of 0-shot text fact transfer. Further, evaluators indicated that ModQGA-Sa -Sup outperformed LED in factuality in 46% of cases on U.S. News, once again suggesting that our transferred entities can improve the factual

---

that match the length of the entity, even if the speci- ficity is not matched (c.g., predicting "Tallahassee" instead of "Florida"). Finally, the provided entity is drastically unrelated to the question (e.g., "200"), so will the answer (e.g." "185"). Controlling speci- ficity is difficult task (Huang et al., 2022), but we believe our specificity- QA model reveals potential direction to address this problem.
any model solve the task would now have in- corporate the extra step of fact verification, making the task more similar to its downstream use case.
Ethics Statement
The goal of text fact transfer is to transfer the fac- tual content of source text while preserving its original style, which we accomplish by designing ModQGA. As mentioned in the introduction, some downstream applications of text fact transfer could include automatically generating news for current events by leveraging a previous news article for a similar event or repurposing existing educational materials for new subjects. However, as with all text generation frameworks, model like Mod- QGA which is designed for text fact transfer could still hallucinate factual errors. Hence, to avoid the spread of misinformation and inaccurate factual content, ample considerations and thorough evalu- ations must be made before leveraging a text fact transfer framework in downstream applications.
5.5 Sample Outputs
In Appendix B.3, we present examples of target texts generated by ModQGA and other baselines.
6 Conclusion
We propose the task of text fact transfer and de- velop ModQGA overcome the difficulty of LMs to perform our task. ModQGA leverages a novel combination of end-to-end question generation and specificity-aware question answering to perform text fact transfer. Through experiments on four datasets, including human evaluation, we find that O-shot and supervised ModQGA excel in style preservation and factuality majority of datasets. We conduct an ablation study to reveal the strengths of our design choices of ModQGA. Finally, we per- form a qualitative analysis of our specificity- question answering model, which shows at least some ability to control the specificity of its answers.
Acknowledgements
We thank the anonymous reviewers for their feed- back. This material is based upon work supported by the National Science Foundation IIS 16- and IIS 16-33755, Zhejiang University ZJU Re- search 083650, IBM-Illinois Center for Cognitive Computing Systems Research (C3SR) a research collaboration as part of the IBM Cognitive Horizon Network, grants from eBay and Microsoft Azure, UIUC OVCR CCIL Plamning Grant 434S34, UIUC CSBS Small Grant 434C8U, and UIUC New Fron- tiers Initiative. Any opinions, findings, and conclu- sions recommendations expressed in this publi- cation are those of the author(s) and do not neces- sarily reflect the views of the funding agencies.
7 Limitations
One limitation of O-shot ModQGA is that it has a slower inference time compared the 0-Shot GPT models. Although our model shows improve- ments in factuality and style over the GPT models, we acknowledge that it is important to ensure our framework is computationally efficient. The slow- est part of ModQGA is the ensembling of multiple questions during question answering. Hence, we believe future research could improve upon Mod- QGA by identifying subset of generated questions that are likely to produce high-quality answers, and only using this subset in ModQGA. This could make contributions to an interesting research area of high-quality question identification.
References
Fadi Abu Sheikha and Diana Inkpen. 2011, Generation of formal and informal sentences. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 187-193, Nancy, France, Associa- tion for Computational Linguistics.
Further, assume that the factual corpora used in our tasks are error-free and do not contain con- tradictions. Hence, we did not assess how any text fact transfer framework would perform if placed in a setting with misinformation. This could be an interesting future setting for text fact transfer, as
Ali Amin-Nejad, Julia Ive, and Sumithra Velupillai. 2020. Exploring transformer text generation for med- ical dataset augmentation. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 6699-4708 Marseille, France. European Language Resources Association.

---

Chenxin An, Ming Zhong. Zhichao Geng, Jiangiang Yang, and Xipeng Qiu. 2021. Retrievalsum: A re- trieval enhanced framework for abstractive summa- rization. CoRR, abs/2109.07943.
the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5972- 29924, Florence. Italy. Association for Computational Linguisties.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and
Nishant Balepur, Jie Huang. and Kevin Chunn Chang. 2023. Expository text generation: Imitate, re- trieve, paraphrase. arXiv preprint (tt:::233500322...
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint
Markus Bayer, Made-André Kaufhold, Bjöm Buchhold, Marcel Keller, Jörg Dallmeyer, and Christian Reuter. 2023. Data augmentation in natural language pro- cessing: a novel text generation approach for long and short text classifiers. International journal of machine learning and cybernetics, (41):135-150.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021. GSum: A gen- eral framework for guided neural abstractive summa- rization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4830-4842, Online. Association for Computational Linguisties.
Iz Beltagy, Matthew E, Peters, and Arman Cohan, 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Bhavya Bhavya, Jinjun Xiong, and ChengXiang Zhai. 2022. Analogy generation by prompting large lan- guage models: A case study of InstructGPT. In Proceedings of the 15th International Conference on Natural Language Generation, pages 298-312, 12, Waterville, Maine, USA and virtual meeting, Associ- ation for Computational Linguisties.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. Proceedings of the AAAI Conference Artificial Intelligence. 32(1).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Belra-Bellrachini. 2017. The webnlg chal- lenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Lan- guage Generation, pages 124 133.
Yixin Cao, Ruihao Shui, Liangming Pan, Min- Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: new task towards better communi- cation between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1061-1071, On- line. Association for Computational Linguistics.
Andreas Graefe. 2016. Guide to automated journalism.
Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psy- chology, 61(1):29-44
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1535-1544 Vancouver, Canada. Association for Computational Linguisties.
Ziqiang Cao. Wenjie Li, Sujian Li, and Furu Wei, 2018. Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1.52-611.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.1 14799.
Zhiqiang Hu, Roy Ka-Wei Lee, Charu C. Aggarwal, and Aston Zhang. 2022. Text style transfer: A review and experimental evaluation. SIGKDD Explor. Newsl. 24(1):14-45.
Janggje Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li, Yanghua Xiao, and Hao Zhou. 2022. E-KAR: A benchmark for rationalizing natural language analogical reason- ing. In Findings of the Association for Computa- tional Linguistics: ACL 2022, pages 3941-3955, Dublin, Ireland. Association for Computational Lin- guistics.
Zhiting Hu, Zichao Yang. Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward con- trolled generation of text. Proceedings of the 34th International Conference on Machine Learn- ing, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase gener- ation with a syntactic exemplar. In Proceedings of
Jie Huang, Kevin Chen-Chuan Chang. Jinjun Xiong, and Wen-mei Hwu. 2022. Can language models specific? how? arXiv preprint 10.002220

---

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Fdouard Grave, 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions on Machine Learning Research.
Juncen Li, Robin Jia, He He, and Perey Liang, 2018, Delete, retrieve, generate: a simple approach to senti- ment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 1865-1874, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv. 55(12).
Chin- Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branckes out, pages 74-81.
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep Learning for Text Style Transfer: A Survey. Computational Linguistics, 48( 48(1):155-205.
Shuai Lin, Wentao Wang, Zichao Yang, Xiaodan Liang,
Frank F. Xu, Eric Xing, and Zhiting Hu. 2020. Data- to-text generation with style imitation. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 1589-1598, Online. Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020. Hooks in the headline: Learn- ing to generate headlines with controlled styles. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5082- 5093, Online. Association for Computational Lin- guistics.
Yuning Mao, Xiang Ren, Heng Ji, and Jiawei Han, 2020. Constrained abstractive summarization: Pre- serving factual consistency with constrained genera- tion. arXiv preprint arXiv:2010.11 127233.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pa- pers, pages 427-431, Valencia, Spain. Association for Computational L Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representa- tions vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scoutsdale Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Eleni Kaldoudi, Nikolas Dovrolis, Stathis Th. Konstan- tinidis, and Panagiotis Bamidis. 2011. Depicting educational content repurposing context and inheri- tance. IEEE Transactions on Information Technology in Biomedicine, 15(1):164 170.
Wojeiech Kryscinski. Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332 9346, Online. Association for Computa- tional Linguistics.
Samraj Moorjani, Adit Krishnan, Hari Sundaram, Ewa Maslowska, and Aravind Sankar. 2022. Audience- centric natural language generation via style infusion. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1919-1932, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with learned entity prompts for abstractive summarization. Transactions of the Association for Computational Linguistics, 9: 14442.
Phuong Nguyen, Tung Le, Thanh-Le Ha, Thai Dang, Khanh Tran. Kim Anh Nguyen, and Nguyen Le Minh. 2022. Improving neural machine translation by ef- ficiently incorporating syntactic templates. In Ad- vances and Trends in Artificial Intelligence. Theory and Practices in Artificial Intelligence: 35th Inter- national Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2022, Kitakyushu, Japan, July 19-22, 2022, Proceedings, pages 303-314. Springer.
Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020b. Retrieval-augmente generation for knowledge-intensive nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474.

---

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.
Anshumali Shrivastava and Ping Li. 2014. Asymmetric Ish (alsh) for sublinear time maximum inner prod- uct search (mips). Advances in neural information processing systems, 27.
Dave Or: 2013. 50,000 lessons how to read: relation extraction corpus. Online: Google Research Blog. 11.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Ma- heswaran. 2019. "transforming" delete, retrieve, gen- erate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNE) 3279, Hong Kong. China. Association for Computa- tional Linguisties.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-3 1-318, USA. Association for Computational Linguisties.
pages 3269-
Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Callin
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language mod- els as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019. pages 2463-2473 Association for Computational Linguistics.
2021. BERT to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume I: Long Papers), pages 3609-3624, Online. As- sociation for Computational Linguisties.
Dingmin Wang, Ziyao Chen, Wanwei He, Li Zhong, Yunzhe Tao, and Min Yang. 2021. A template-guided hybrid pointer network for knowledge-bused task- oriented dialogue systems. In Proceedings of the Ist Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021). pages 28, Online. Association for Computational Linguistics.
Shrimai Prabhumoye. Yulia Tsvetkov, Ruslan Salakhut- dinov, and Alan W Black. 2018 Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers). pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints. page arXiv: 066660005000.
Shuohang Wang, Yichong Xu, Yuwei Fang. Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume Long Papers). pages 3170-3179, Dublin, Ireland. Association for Computational Lin- guistics.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus. bench- marks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Or- leans, Louisiana. Association for Computational Lin- guistics.
Yunli Wang. Yu Wu, Lili Mou, Zhoujun Li, and Wenhan Chao. 2019. Harnessing pre- trained neural networks with rules for formality style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language P Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP--ICCNIP)) pages 3573-3578, Hong Kong, China. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Wei Shang, Chong Feng, Tianfu Zhang, and Da Xu. 2021. Guiding neural machine translation with re- trieved translation template. In 2021 International Joint Conference Neural Networks (JCNN), pages 1-7, IBEE.
Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin. 2020. Retrieve and refine: exemplar-based neural comment generation. In Proceedings of the 35th IEFF/ACM International Conference on Automated Software Engineering, pages 349-360.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Informa- tion Processing Systems, volume 30. Curran Asso- ciates, Inc.
Matthew West, Geoffrey L Herman, and Craig Zilles. 2015. Prairielearm: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition, pages 26-1238.

---

Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A covercoveeage challenge corpus for sen- tence understanding through inference. In Proceed- ings the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers); pages 11 1112-1122 122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Ruochen Xu, Tao Ge, and Furu Wei. 2019. Formality style transfer with hybrid textual annotations. CoRR. abs/ 190306633..
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In Pro- ceedings of COLING 2012, pages 2899-2914, Mum- bai, India. The COLING 2012 Organizing Commit-
Zichao Yang. Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Kirk-arira...ck. 2018. Unsupervised text style transfer using language models as discrimina- tors. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Zhang. Tao Ge, and Sun. 2020. Parallel aug- mentation for formality style transfer. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pages 3221-3228, On- line. Association for Computational Linguistics.
Yinhe Zheng, Zikai Chen, Rongsheng Zhang, Shilei Huang, Xiaoxi Mao, and Minlie Huang. 2020. Styl- ized dialogue response generation using stylized un- paired texts. In AAAI Conference on Artificial Intelli- gence.

---

A Experimental Setup
tected entities (prediction score > 0.3): "PHONE NUMBER", "CRYPTO", "EMAIL ADDRESS", "IBAN CODE", "IP ADDRESS", "MEDICAL LI- CENSE", "US BANK NUMBER". "US DRIVER LICENSE", "US ITIN", "US PASSPORT", "US SSN".
A.1 Datasets
The U.S. News dataset in ETG already follows a very consistent style that can be adapted for text fact transfer. Hence, we take each output as the tar- get text, and match it with a source text by selecting a random example from the training set. We match descriptions for public colleges with other random descriptions for public colleges, and the same for private colleges, as there are slight differences in the style between these descriptions when describ- ing tuition. The last sentence of the public college descriptions follow the form "The in-state tuition is X; the out-of-state tuition is Y", while the last sentence of the private college descriptions follow the form "The tuition is X
We provide summary statistics of each dataset in Table 5, All datasets are in English.
A.2 Training Setup
The question generation model of ModQGA is trained with BART Large (406M), using batch size of 8, learning rate of 2c-5, weight decay of 0.01, 500 warmup steps, 8 gradient accumulation steps, and 3 training epochs. The question answer- ing model of ModQGA is trained with BERT Large (340M) using the same parameters. We select an- swers spans with a maximum length m equal to two times the length of the entity specificity guid- ance. We generate
For the Medline dataset in ETG, retreviigg a sim- ilar example for the source text cannot be done in the same way, as the style is less consistent. Hence, we first convert each output document to a consistent style by performing question answering using the output as the context with the follow- ing questions: 1) "What is [topic] used to treat?"; 2) "What class of medications does [topic] fall into?"; 3) "How does [topic] work?". Using these answers, we construct document through the tem- plate: [Topic] is used treat [(1)]. It belongs to a class of medications called [(2)]. It works by [(3)]. For question an- swering, we leverage the RoBERTa-Bass model trained on SQuADS. To ensure all collected out- puts fall into this template, we discard documents which provide a negative logit score to any of the three questions. We then use the same process as U.S. News to match source and target texts.
10 sequences in end-to-end question generation with nucleus decoding (top- p 0.75). During retrieval, we select
texts.
The infilling for ModQGA-Sup and LED are implemented with the same LED model (Beltagy et al., 2020) (149M), using batch size of 1, learn- ing rate of 5e-5, and 1500 warmup steps. We train each model for epochs and after training, load the model with the lowest validation loss with re- spect to each epoch. We use a maximum input size of 16384 to encode the input corpus, maximum output length of 256 for U.S. News and Medline, and maximum output length of 64 for Google and t-REX. The training time for this model was around 10 hours on each dataset. The BART model in BART+Retr is trained with the same parameters and similar model size (140M) as the LED model, but instead using a maximum input size of 1024. Using the same strategy, we train the model for 10 epochs and after training, load the model with the lowest validation loss with respect to each epoch. We ensured that the validation loss of each seq2seq model converged on datasets.
The Google and tERE datasets do not require any modifications, as we simply pair source texts and target texts by finding relation triples that share a relation. To obtain the factual corpora C for each target topic in Google, we web scrape us- ing the query "4, Wikipeida." We keep only al- phanumeric characters and punctuation, and de- code the text with unidecode. We qualitatively analyzed a sample of corpora and did not find any personal identifiable information. To be safe, we use the Presidio analyzer provided by Microsoft and remove all sentences with the following de-
All GPT-3.5 models are 3pt-3.5-turbe turbo (175B) with temperature of 0.2. For U.S. News and Medline, we set the maximum output length to 256, and for Google and L-REX, set the maximum output length 64. The Retriever used by all baselines is the Contriever model (Izacard et al., 2022) fine-tuned on MS-MARCO (Nguyen et al., 2016), which is based on BERT (110M). The input query is the source text with every occurrence of
/nttps://hugginggfac cttppppse////eeee///cccccccccc
rober bbberta-base-SQuAD2 "https://microooff githu. io/presidio/ analyzer/

---

the source topic replaced with the target topic. In Table 6, we show that this setup outperforms solely using the target topic as the query.
B Results
B.1 Full Ablation
We display the full ablation results in Table 8. We find that the use of specificity aware question answering model and ensembling of generic ques- tions consistently improve the factuality and style of O-shot ModQGA.
We retrieve k 25 texts for the BART+Retr model, and = texts for the GPT models We found that retrieving more than h texts would limit the number in-context examples that we could provide to GPT-3.5, and we found that these in- context examples were essential to improve the per- formance of the GeT+Retr models (See Appendix A.3, which also contains the prompts used for each GPT model).
B.2 Human Evaluation
We build the human evaluation interface using PrairieLearn (West et al., 2015). Instructions given to the annotators are shown in Figure 6, and a screenshot from the interface is shown in Figure 7. The model outputs were randomized in each comparison. We use Gwet's AC2 (Gwet, 2008) measure annotator agreement, given the presence of high agreement in our evaluation (e.g., over 90% of supervised models annotated as having equal style). We compute value of 0.71, indicating good agreement.
Hyperparameters were manually selected (no search) by assessing validation loss. All models were trained on a single NVIDIA A40 GPU. R1, R2, and BLEU were calculated using the hugging- face Evaluate library.
A.3 GPT Prompts and Considerations
We provide preliminary analysis to study how prompt size affects the few-shot GPT-3.5 models for text fact transfer on the Google dataset in Table 7. Interestingly, we find that increasing the size of the in-context examples from to 10 worsens the performance the GPT-3.5 models that do not use retrieval, but also increases the performance of the GPT-3.5 models that do use retrieval. This could indicate that LLMs are highly sensitive to the in-context examples for text fact transfer.
B.3 Sample Outputs
We provide examples of outputs produced by Mod- QGA (0-shot and supervised) along with their re- spective baselines in Tables 9, 10, 11 for U.S. News. Medline, and Google, respectively.
We provide the prompt used for the 0-shot GPT- 3.5 models in Figure 4 and the prompt used for the shot GPT-3.5 models in Figure 5. When creating the prompt for the 0-shot model, we tested slight variations of the prompt shown in Figure on the validation sets and ultimately found the one shown to work the best.
Given the sensitivity 0-shot GPT-3.5, we ac- knowledge that there likely exists prompt that could boost the performance of this model. How- ever, looking at Tables and 2, we observe that 0- shot ModQGA consistently outperforms the z-shot GPT-3.5 models on all datasets except for Medline. Given this outcome and that z-shot GPT-3.5 is ex- pected to outperform 0-shot GPT-3.5 regardless of the prompt, we believe that, at the very least, 0- shot ModQGA will outperform the 0-shot GPT-3.5 models across varied prompt formats on all datasets except Medline.

---

Dataset
Train/Valid/Test
Avg Output Length
Avg Corpus Size
U.S. News
315/ 79
72.61
531.61
Medline
284 24
116.38 31.01
1064.91
Google
777 224
7.21
Analogy
496/ 68 115
6.09
205.24
Table 5: Summary statisties of text fact transfer datasets.
Dataset
Model
RI @55
R1 69 10
RI @ 15
RI @2
Contriever-Source
0,574
0.649
0.689
0.762
U.S. News
Contricver-
0.383
0.495
0.569
0.674
Contriever-Soune
0.498
0.643
0.714
0.799
Medline
Contriever-Toppe
0.316
0.519
0.603
0.739
Contriever Sonureever-Souree Contriever- Topic
0.911
0.951
0.956
0.966
Analogy
0.818
0.912
0.933
0.948
Contriever-Souree 0.704
0.717
0.736
0.797
Google
Contriever-Topp
0.697
0.711
0.717
0.795
Table Baseline query comparison with Contriever using average ROUGE-I recall compared the ground truth output on the validation set. ROUGE-1 recall will allow to assess what proportion of the tokens from the output are covered by the information retrieved using Contriever (Note: This does not include the tokens covered by the source text). denotes the number facts retrieved from the factual corpus. Contriever-Source uses the source text with every occurrence of the source topic replaced with target topic, while Contriever- Topic solely uses the target topic as a query. Best results are in bold.
Model Type
Model
R1
R2
BLEU
Hallue
FactCC
NLI-Ent
Length
3-Shot
0.846
0.808
0.630
1.32
0,546
0.415
1.17
GPT No Retr
10-Shot
0.838
0.792
0.675
6.77
0.262
0.244
1.05
3-Shot 10-Shot
0.628
0.574
0.382
0,508
0.308
1.35
GeT+Retr
0.812
0.773
0.614
0.541
0.467
1.14
Table 7: Performance analysis with respect to the number of few-shot prompts for GPT-3.5 models on the Google dataset.
Dataset
Model
R1
R2
BLEU
FactCC
NLI-Ent
Length
Full ModQGA
0.934
0.890
0.865
0.650
0.708
1.01
U.S. News
Generie
0.867
0.803
0.772
0.428
0.584
1.03
Normal QA
0.883
0.811
0.760
0.444
0.639
1.05
Full ModQGA
0.724
0.605
0.579
0.915
0,502
0.97
Medline
No Generic
0.680
0.553
0.527
0.889
0.215
0.97
Normal QA
0.686
0.562
0.522
0.825
0.278
1.07
Full ModQGA
0.929
0.914
0.857
0.589
0.621
1.00
Google
No Generic
0.925
0.910
0.843
0.600
0.613
1.02
Normal QA
0.915
0.891
0.784
0.611
0.585
1.07
Full ModQGA
0.841
0.781
0.721
0.722
0.609
1.05
L-EXX
No Generie
0.769
0.712
0.689
0.443
0.113
1.02
Normal QA
0.805
0.726
0.629
0.698
0.509
1.14
Table 8: Ablation comparison with output similarity metrics (R1, R2, BLEU) and factuality metrics (FactCC. NLI-Ent) for the 0-shot ModQGA models. No Generic removes generic questions during question transferring, and Normal QA uses BERT-based question answering model without specificity guidance.

---

Information about university of nevada, las vegae: (University of Xevads, Vegas is public institution that waa founded 1957. Uhiversity Nevade, Las Veges' cenking in the 2022-2023 edition of Best Colleges is National Universities, #295, With its innovative frontier spirit, University of Nevada Vegas institution with diverse enrollment of noze than 24, students, 4200 gradaace studenta, scholert. Since its first classes Nevada, laa Vegaa UNLV hea undergone an cnazing tranaformation fron duzty octpost the souta eäge town institution. The University Nevada, Taa Vegaa 1a large public university located on arban compua Template text with the la a public insticution that was founded in 1855. Tt. has a total unsergradueze earoliment 28, its setting auburban, and cangua size la 5,192 screa. It otiliaes senester-besed acadenic caleadar. Michigan State Universicy's zanking in the 2022-2023 edition of National Oniveralties, 47T.
ia
research
faculty nembers, and 1,185 international studenta
held in 1957, the University
zesearch
Vegas, Nevada.
of nichigan
(Michigan
University
tuition and are $14,850; cut-oi-state tuition and fees aze 540, 662.1
Minimally modify the template of nevada, las vegas. Do not daviate all fron the style langth of the template text. Output:
topic university
Figure 4: Example zero-shot prompt for O-shot GPT-3.5+Retr U.S. News. The O-shot GPT-3.5 (no Retrieval) model uses the same prompt, without the information prepended in the beginning of the prompt.
Template Title: (michige state)
Template: (michigan state university is a public
institution that founded in 1855. it has total
undergraduate enrollment of 38, its setting
suburban, and the campus size is 5,192 acres. it utilizes semester-based academic calendar. michigan state uni versi ty's ranking in the 2022-2023 edition of best colleges is national universities, #77. its in-state tui tionion fees are $14, 850; out-of-state tuition fees are $40, 662.)
Title: (university of nevada, las vegas)
Source: (university of nevada, las vegas is a public institution that was founded in 1957. university of nevada, las vegas" ranking the 2022-2023 edition of best col leges is national universities, #285. wi its innovative frontier spirit, university of nevada las vegas unly
thriving urban research institution with diverse enrollment of more than 24, 700 students, 4200 graduate students, 1507 faculty members, and 1,185
international students scholars. since its first classes were held in 1957, the university of nevada, las vegas unly has undergone an amazing transformation from dusty outpost on the south edge of town to thriving urban research institution. the university of nevada, las vegas large public university located on an urbar. campus in las vegas, nevada. Output:
Figure S: Example few-shot prompt for z-shot GPTe3.5+Retr on U.S. News. The z-shot GPT-3.5 (no Retrieval) model uses the same prompt without the source label. This is the final part of the few-shot prompt, this prompt is preceded by z in-context examples following the same format.

---

brokan up into four iactions. will be shown predicted output from Model A and oulput from Model B, as well oulput Based the true output, you will be adked to select which model (A produced a better output based on style and factuality. Plaase use the following guidelines for style and factuality:
You will total of painwisa
Style: How similar the style of the model curput compored true output? Do they organize the same information in same order, generally the same phrasing ng and same leve specificity are concerned
information correct,
being
similaly.
factuality: low sccural the information conveyed the model ouput? Are there significant factual inconsistancies or emrors? You can compara the fäctuality of tha modil ourpur and true output, but please use Google to verily factusl errors if they are not abvious
The compar ison will be shown multiple cheice formal, allewing you select each attribute if (1) Model was betler, (2) Model was better. or (3) the models performed similar y. Once you have made your choice you may and Grade" again.
change your you may
answer hit
Figure 6: Instructions given to human annotators for pairwise comparison evaluation.
True Output
University campus
public institution. has undergraduate based academic calendar.
14,167,
setting
utilizes University of North Florida's ranking the 2022-202 edition Best Colleges is National Universibes, #263. Its in-state tuition and fees are $6,39% out of state tuition and fees are $20,107.
Model Output
Model Output
University of North Floride public institution that was founded in 1971. It has a total
University of North Floride public institution that was founded in 1816. It has total undergraduate enrollment 14 167, its setting urban, the campus
undergraduate envollment of 14,161 167, setting is 1,300 acres. utilizes
urban, and campus seemsster-bssed academic calendar. University North Florida's ranking 2022 2023 edition of Best Colleges is National Universities, #263, Its in state tuition and fees are $6,389; out-of-state tuition and fees are $20.107.
acres. It utilizes University North Florida's ranking the 2022-2022 edition Best Colleges National Universities, #263, Its in state tultion and fees are $6,389; out-of-state tuition and fees are $20,107.
semester- academic
Which outputt better style?
(a) Model is better
(b) and B are the same
(c)
outputt better factuality?
(a) Model better
Model better
Figure 7: Interface for human annotators for pairwise comparison evaluation.

---

Model
Target Text
Ground Truth
husson university a private institution that was founded in 1898. it has a total undergraduate enrollment 2,572, its setting is suburban, and the campus size is 200 acres. it utilizes semester-based academic calendar. husson university's ranking in the 2022-2023 edition of best colleges is national universities, #331-440. its tuition and fees are $21,090.
Source Text
regis university a private institution that was founded in 1877. has total undergraduate enrollment of 2,745, its setting is suburban. and the campus size is 90 acres. it utilizes semester- academic calendar. regis university's 's ranking the 2022-2023 edition of best colleges is national universities, #250. its tuition and fees are $42,220.
0-shot ModQGA
husson university a private institution that was founded in 1898, it has a total undergraduate enrollment of 2,679, its sctting is suburban, and the campus size is 200 acres, it utilizes semestcr-hased academic calendar. husson university's 's ranking in the 2022-2023 edition of best colleges is national universities, #331. its tuition and fees are $21,090.
0-shot GPT
husson university a private institution that was founded in 1898. it has a total undergraduate enrollment of 2,400, its setting is suburban, and the campus size is 208 acres, it utilizes semester-based academic calendar. husson university's 's ranking the 2022-202 edition of best colleges is regional universities north, #119. tuition and fees are $19,820.
O-shot GeT+
husson university private institution that was founded in 1898, it has total undergraduate enrollment of 2.679. with 2,264 being full time and 415 being part time. its setting suburban, and the campus size 208 acres. utilizes semester-based academic calendar. husson university's 's ranking in the 2022-2023 edition of best colleges is regional universities north, #119. its tuition and fees $19,122.
ModQGA-Sup
husson university a private institution that was founded in 1898, it has a total undergraduate enrollment of 2,572, its setting is suhurban, and the campus size is 200 acres. it utilizes semester-hased academic calendar. husson university's 's ranking in the 2022-2023 edition of best colleges is national universities, #331-440. its tuition and fees are $21,090.
3-shot GPT
husson university private institution that was founded in 1898. it has total undergraduate enrollment of 2,346, its setting is suburban, and the campus size is 208 acres. it utilizes semester-based academic calendar. husson university's 's ranking the 2022-2023 edition of best colleges is regional universities north, #136-#176. its tuition and fees are $19,350.
3-shot GPT+Retr
husson university private institution that was founded in 1898. it has total undergraduate enrollment of 2,264, its setting is suburban, and the campus size is 208 acres. it utilizes semester-based academic calendar. husson university's 's ranking the 2022-2023 edition of best colleges is regional universities north, #136-#176. its tuition and fees are $19,990.
L.ED husson university is a private institution that was founded in 1898, it has a total undergraduate enrollment of 2,572, its setting is suburban, and the campus size is 200 acres. it utilizes semester- based academic calendar. husson university's 's ranking in the 2022-2023 edition of hest colleges is national universitics, #331-440. its tuition and fees are $32,560.
BART+Reir
husson university is a private institution that was founded in 1898, it has a total undergraduate enrollment of 2,679, its setting is rural, and the campus size is 208 acres. it utilizes a semester- based academic calendar. husson university's 's ranking in the 2022-2023 edition of best colleges is national universities, #331 1-300 its tuition and fees are $33,834.
Table 9: Sample target texts generated by basclines on U.S. News with source topic Regis University and target topic Husson University. Red text indicates text that is factually incorrect. Blue text indicates text that does not follow the style of the source text.

---

Model
Target Text
Ground Truth
relugolix is used to treat advanced prostate cancer. it belongs to class of medications called gonadotropin- releasing hormone receptor antagonists. it works by decreasing the amount of testosterone produced by the body
Source Text
hydromorphone is used to treat relieve pain. it belongs to class of medications called opiate analgesics. it works by changing the way the brain and nervous system respond to pain
O-shot ModQGA
relugolix is used to treat prostate cancer. it belongs to group of medications called receptor antagonists. it works by reducing the amount of testosterone
0-shot GPT
relugolix used to treat prostate cancer. relugolix used to treat symptoms of prostate cancer. belongs to class of medications called gonadotropin-rllesinng hormone (garh) antagonists. it works by cells.
changing the way the pituitary gland produces hormones that stimulate the growth of prostate cancer
0-shot GPT+Retr
relugolix is used treat prostate cancer and heavy menstrual bleeding associated with uterine fibroids in premenopausal women. belongs to the class medications called gorh receptor antagonists. works by depressing testosteronc levels and providing less burdensome therapeutic option for patients. possible side effects include joint or muscle pain, high blood pressure, fluid buildup in the body, hot flashes, upset stomach, and diarrhea.
ModQGA-Sup
relugolix used to treat prostate cancer. it belongs to class of medications called gurh antagonists. works by decreasing the amount of testosterone that the body makes.
3-Shot GPT
relugolix is used to treat advanced prostate cancer. belongs class of medications called gonadotropin- releasing hormone (gnrh) receptor antagonists. it works by decrcasing the production of testosterone, which can slow the growth of prostate cancer cells.
7-Shot Get+
relugolix is used to treat prostate cancer and heavy menstrual bleeding associated with uterine fibroids. it belongs class of medications called conadotropin-releasigg hormone receptor antagonists. it works by decreasing the production of certain hormones in the body.
LED
relugolix is used to treat advanced prostate cancer. it belongs to class of medications called androgen receptor antagonists. it works by decreasing the amount of testosterone
BART+Reir
relugolix is used to treat adult patients with advanced prostate cancer. it belongs to class of medications called camadoropin-relcastie hormone receptor antagonists. it works by decreasing the amount of androgen made in the body
Table 10: Sample target texts generated by baselines on Medline with source topic Hydromorphone and target topic Relugolix. Red text indicates text that is factually incorrect. Blue text indicates text that does not follow the style of the source text.
Model
Target Text
Ground Truth
dennis davis was born in manhattan.
Source Text
paul downes was horn in devon.
0-shot ModQGA
dennis davis was born in manhattan.
0-shot GPT
dennis davis was born in devon.
0-shot GeT+Retr
dennis davis was born [unknown location].
ModQGA-Sup
dennis davis was born manhattan.
3-shot GPT
dennis davis was born in institute, west virginia.
10-shot GPT+Retr
dennis davis was born in devon.
LED
dennis davis was born in london.
BetT+Retr
dennis davis was born in manhattan.
Table 11: Sample target texts generated by basclines on Google with source topic Paul Downes and target topic Dennis Davis. Red text indicates text that factually incorrect. Blue text indicates text that does not follow the style of the source text.


Reviews:
[{"review_id": "dtDl9rUhjH", "reviewer": "Reviewer_vEhb", "summary": "", "questions": "- Do you think that zero-shot LLM can work better on the task if you carefully design the prompts? Did you evaluate the same before choosing the prompts?\n- Do you have any intuition why the factuality score is really high in some datasets compared to others? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and insightful comments! We appreciate that the reviewer finds our task to be new, interesting, with promising applications, and great potential for further exploration. Further, it was encouraging to hear that the reviewer finds our model to be intuitive and producing competitive results, and that our datasets could be useful for future studies.\n\n### RE: Zero-Shot GPT\n\nWe appreciate your concern regarding the sensitivity of 0-shot GPT. When creating the prompt for this model, we tested slight variations of the prompt shown in the paper on the validation sets and ultimately found the one shown to work the best.\n\nWe acknowledge that it is possible that there exists a prompt that could boost the performance of 0-shot GPT. However, looking at Tables 1 and 2, we observe that 0-shot ModQGA consistently outperforms the n-shot GPT models on all datasets except for Medline. Given this outcome and that n-shot GPT is expected to outperform 0-shot GPT regardless of the prompt, we believe that, at the very least, 0-shot ModQGA will outperform the 0-shot GPT models across varied prompt formats on all datasets except Medline. We can add these details in the final version.\n\n### RE: Evaluation Metrics\n\nThank you for your feedback regarding our choice of evaluation metrics.\n\n#### *ROUGE/BLEU*\n\nWe believe that our use of localized replacements through question generation and answering does not automatically grant our model an unfair advantage with BLEU and ROUGE. In fact, using localized replacements is still a difficult approach, as even a minor misidentification of which entities need to be transferred in the source text can lead to much lower scores.\n\nFurther, our decision to use ROUGE and BLEU was because they were fundamentally appropriate for measuring style preservation (instead of favoring any particular model). Part of our definition of style is “how the factual content is phrased” (line 51). Thus, we found it most relevant to evaluate style preservation using token overlap metrics like ROUGE and BLEU, which can effectively calculate the similarity of the phrasing of the predicted text and ground truth output. These metrics are also widely accepted in text style transfer (Jin et al., 2022), so we felt it was appropriate and best practice to use them for our task.\n\n\n#### *Factuality*\n\nEvaluating factuality is a difficult task, and there is no single metric that can capture all factual errors. Thus, we adopted a combination of factuality metrics that are designed to detect different errors. For example, FactCC is trained to detect simple factual errors such as numerical inconsistencies and entity mismatches, while NLI-Ent captures entailment relations between two pieces of text which often rely on logical inference (section 4.3). Your observation of the high FactCC score for SourceCopy-Medline (0.890) is valid and suggests that the factual inconsistencies in this dataset are not simple numerical/entity errors. However, the NLI-Ent score for SourceCopy-Medline (0.034) accurately conveys the low factuality of the SourceCopy baseline, illustrating the necessity of a multi-metric approach to factuality. We will add these details to the final version.\n\nAlthough we chose the automated metrics that were best fit for the task of text fact transfer, we acknowledge that they are not a perfect solution for comparing baselines. For this reason, we additionally conducted a human evaluation study (section 5.2). We find that 0-shot ModQGA outperforms 0-shot GPT+Retr and ModQGA-Sup outperforms LED in style and factuality on the two evaluated datasets, which parallel our quantitative results using automated metrics (line 573).\n\n### RE: Presentation Improvements \n\nThank you for pointing this out, we will make these corrections in the final version. We greatly appreciate the attention to detail!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They introduce a new and interesting task called \"text fact transfer,\" which has promising applications in repurposing and adapting texts and data augmentation. It also has great potential for further exploration.\n- The proposed ensemble model is very intuitive and authors demonstrate that it produces competitive results in comparison to other baseline models.\n- They also make available four datasets for text fact transfer which can be useful for future studies on this task.\n", "reasons_to_reject": "- The zero-shot GPT 3.5 model used for comparison is known to be sensitive to the input prompt, which is not taken into account or discussed in the paper. Appendix does include a comparison of prompt length, but on general design of the prompt.\n- The evaluation metrics like BLUE and ROUGE naturally is in favor of the model given that it conducts localized replacements. The factuality metric seems to be questionable looking at the score of  **Medline-SourceCopy** in Table1.  ", "questions_to_authors": "- Do you think that zero-shot LLM can work better on the task if you carefully design the prompts? Did you evaluate the same before choosing the prompts?\n- Do you have any intuition why the factuality score is really high in some datasets compared to others? ", "paper_topic_and_main_contributions": "The paper introduces the novel task of text fact transfer, the goal of which is to replace the factual content based on a source entity in the same style. Contrary to text style transfer, the style has to be preserved while adapting the facts. The authors propose a model which  uses question generation to generate a question related to the source entity, replaces the source entity with target entity and employs a question answering model to generate the answer corresponding to the target entity. This is then replaced in the correct position in the source text. They implement a zeroshot and a supervised version of the model. They should that their model performs considerably better than other baselines. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Corrections: In Table 2, the second best score for FactCC should be **t-REX, BART+Retr** and not **t-REX, LED**. Underline for second best is missing in **t-REX, Halluc**\n"}, "extra_text": {}}, {"review_id": "BXKkLrFWGW", "reviewer": "Reviewer_jgwJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review! We are delighted to hear that the reviewer finds text fact transfer to be an interesting problem with potential applications, believes our evaluation to be exhaustive and careful, and feels our writing is effective and clear. We also appreciate the reviewer’s defense of the inference time of our model compared to LLMs.\n\n### RE: Task Motivation\n\nThank you for your suggestions regarding our framing of the motivation for text fact transfer. We agree that the use of the word \"drawbacks\" in relation to text style transfer is not appropriate, since the fact that text style transfer does not cover our potential applications (e.g. educational repurposing) is not a shortcoming of the task. As suggested, we will make the motivation of text fact transfer stronger so it instead speaks to the positive benefits and unique challenges of our new task.\n\n### RE: Meaning and Style Disentanglement\n\nWe acknowledge your concern that “it's not obvious that meaning and style can be completely disentangled.” We agree and believe that depending on the application, the definition of style must be modified to ensure that meaning and style can be disentangled. For example, in our setting, we define style to be the phrasing, organization, and specificity of the text. The inclusion of phrasing allows us to measure style preservation using token overlap metrics, which can effectively calculate the similarity of the phrasing of the predicted text and ground truth output (line 501). But as you mentioned, when repurposing certain types of educational materials, the definition of style may not include the phrasing of the text, since different subjects may require different phrasings to accurately convey their concepts. Thus, the definition of style would need to be modified to exclude phrasing, but the task could still focus on preserving style in terms of organization, specificity, simplicity, engagingness, formality, or other stylistic attributes (Jin et al., 2022).\n\nOverall, the definition of style we use in the paper is appropriate for tasks such as adapting/augmenting the texts of college descriptions, medical drug information, and biographical facts, where phrasing can stay consistent. However, for the most challenging applications of text fact transfer, such as repurposing educational materials, the definition of style would potentially need to be modified to exclude phrasing. In the final version, we can add these details and clearly explain which applications of text fact transfer our current definition of style covers, and how the definition of style could be modified for more challenging applications.\n\n\n### RE: Data Augmentation\nWe completely agree that data augmentation is a very interesting possibility for text fact transfer. For the scope of this work, we focus on developing a model that can effectively perform text fact transfer and thoroughly compare our model with other baselines in diverse domains. While our primary focus was on our model's ability to preserve style and factuality in several domains, we felt it was important to also recognize its applicability to data augmentation in the introduction. In the final version, we will revise the introduction to make clear which applications we focus on in the paper, compared to what we believe are more applications that can be explored in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Although I'm not sure if the framing is entirely correct, I do think that fact transfer is an interesting problem with potential applications. Also, the focus on factuality is welcome.\n* Evaluation is pretty exhaustive, including both zero-shot and supervised settings, with carefully chosen baselines (including LLMs).\n* Overall, the writing is effective and the technical details clearly laid out. \n* In the discussion of limitations, the authors may have been unfair to themselves, comparing inference time of their pipeline with that of GPT3.5. presumably via the OpenAI API. My sense is that the proposed approach would actually be more computationally efficient than an LLM.", "reasons_to_reject": "* The motivation for the task could be stronger. For example, it doesn't seem quite right to start by saying that text *style* transfer has \"drawbacks,\" when you're considering an entirely different problem that has nothing to do with style.\n\n* As a high-level concern, it's not obvious that meaning and style can be completely disentangled. For example, in the cited motivating application of repurposing educational materials to new subjects, it seems that certain new subjects would require not only modifying the named-entities, but also significantly altering the sentence structure to express the new subjects.\n\n* The interesting possibility of using the proposed approach for data augmentation is mentioned in the introduction. I would have liked to see this borne out in experiments on this downstream application of the proposed framework.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper considers the \"fact transfer\" problem, in which the specifics of a sentence (e.g., specific named-entities mentioned) must be replaced without altering the style of the sentence. The proposed approach involves multiple stages. The first stage is question generation, in which it is stipulated that \"all entities in the source text that need to be transferred can be viewed as an answer to a question.\" To generate questions, a sequence-to-sequence model is fit to the SQaAD-V2 dataset to predict questions. If a generated question contains the source topic, the topic can simply be replaced by the target topic. Next, QA is done based on the generated question, where to address the specificity of the answers, the SQuAD-V2 dataset is modified to include specificity guidance in the form of skip-gram synonyms via fastText embeddings. Finally, source text infilling is done based on the generated questions and answers to produce the target text. In the supervised case, the LED language model (Beltagy et al., 2020) is used to generate the target text, similar to keyword-guided text generation. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "M8Hb282Tv8", "reviewer": "Reviewer_KoEs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We appreciate that the reviewer finds our new task to enable various applications, believes that our approach is technically sound, and notes our comprehensive evaluation and promising results.\n\n### RE: Retrieval Augmented Models\n\nWe acknowledge your concern regarding comparisons to more recent retrieval-augmented seq2seq models. We considered comparing these models, such as retrieval-augmented models that perform reranking, but felt that ModQGA could benefit from these same changes (e.g. ModQGA could also perform reranking during the question answering step to more accurately extract transferred entities). Thus, for the scope of this work, we focus on retrieval-augmented models that perform one round of retrieval followed by one round of generation. In future work, it would be interesting to explore how recent advancements in retrieval-augmented models (i.e. those developed for question answering/generation) could be integrated into our ModQGA framework.\n\n### RE: Case Studies\n\nThank you for your suggestions surrounding case studies. We agree that evaluating text fact transfer with in-depth case studies can highlight the unique advantages of our model. In fact, we chose many of our datasets to parallel real-world applications. For example, the U.S. News, Medline, and Google datasets are equivalent to the real-world applications of repurposing/augmenting the text of previously written college descriptions, prescription drug information, and biographical sentences. We selected the open-domain t-REX dataset to study which models have the ability to adapt to diverse real-world settings. Leveraging these datasets, we show that GPT-3.5 struggles to produce factual text and adhere to the style of the source text (line 543), highlighting that our model has advantages in factuality and stylistic consistency in these real-world applications. \n\nFurther, the qualitative analysis of our specificity-aware QA model (section 5.4) was intended to highlight the unique ability of our model to control specificity in an interpretable manner, which could be very useful in real-world settings of text fact transfer when users desire information at varying levels of specificity. In the final version, we will add more details to explain how our constructed datasets and experiments are indicative of certain real-world applications.\n\n### RE: Missing References\n\nThank you for pointing this out. As suggested, we will reference more recent style transfer works in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposes a new task, text fact transfer, that could enable applications like adapting past news or educational content.\n- Technically sound approach, combining question generation and answering to minimally modify source text.\n- Comprehensive evaluation and promising results suggesting the model better preserves style and facts compared to baselines.", "reasons_to_reject": "- Comparison to more recent retrieval-augmented seq2seq models could be useful.\n- Lack of more in-depth and systematic case studies. For example, existing GPT models can achieve transfer in various formats through in-context learning, and LLMs have better knowledgeability. What are the advantages of your model? Although the paper shows good automatic metric results, the application proposed in the article seems to require more case studies to demonstrate the effectiveness of your methods and intuitive judgments. The case in real world is more important and impressive.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new task named text fact transfer, which aims to modify the factual content of a source text while preserving its original style. It is the opposite task of the original style transfer. The key idea is transferring entities between topics using a minimal text alteration approach. \nFurthermore, they decompose the aforementioned task into question generation and answer generation tasks, thereby achieving their objectives. On top of that, they also constructed relevant datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "You may include more recent work on style transfer:\n- [StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing](https://aclanthology.org/2023.acl-long.827) (ACL 2023)"}}]

Author rebuttal:

Meta-review: This paper seems to introduce a new and interesting task on text fact transfer with promising applications. The proposed model seems intuitive and technically sound. The evaluation seems exhaustive with multiple real world datasets and a reasonable set of metrics. The results are competitive with clear writing. All reviewers find this paper to be mostly technically sound with moderately high excitement. Reviewers shared some concerns on sensitivity to slight variations of prompt, choice of factuality metrics, discussion on applications where style and meaning can be disentangled reasonably, and also discussion on case studies to further illustrate the effectiveness of the methods. Reviewers were mostly happy with the author's responses. Authors should include these changes in the revision, which could significantly increase the impact of this work.
Decision: Accept-Main