Expository Text Generation: Imitate, Retrieve, Paraphrase
Authors: Nishant Balepur
Jie Huang
Kevin Chang
Venue: EMNLP 2023 Main (2023)
Keywords: Text Generation
Factuality
Style-guided Generation
Retrieval-augmented Generation
TL;DR: We propose the new task of expository text generation, which involves generating an accurate and organized expository document from a factual corpus, along with a model to solve our task.

Abstract: Expository documents are vital resources for conveying complex information to readers. Despite their usefulness, writing expository text by hand is a challenging process that requires careful content planning, obtaining facts from multiple sources, and the ability to clearly synthesize these facts. To ease these burdens, we propose the task of expository text generation, which seeks to automatically generate an accurate and stylistically consistent expository text for a topic by intelligently searching a knowledge source. We solve our task by developing IRP, a framework that overcomes the limitations of retrieval-augmented models and iteratively performs content planning, fact retrieval, and rephrasing. Through experiments on three diverse, newly-collected datasets, we show that IRP produces factual and organized expository texts that accurately inform readers.

Expository Text Generation: Imitate, Retrieve, Paraphrase
Nishant Balepur
Jie Huang Kevin Chang
University of Illinois at Stbana-Champaign, USA (balepur2, jeffhi, kcchang) @illinois edu
Abstract
Ground Truth
University Denver private institution that was founded in 1884. It has total undergraduale enrollment 5,867 (fail 2021), its setting is city. and the campus size is 125 acres...
Expository documents are vital resources for conveying complex information to readers. De- spite their usefulness, writing expository text by hand is challenging process that requires careful content planning, obtaining facts from multiple sources, and the ability to clearly syn- thesize these facts. To ease these burdens, we propose the task of expository text generation, which seeks to automatically generate an accu- and stylistically consistent expository text for topic by intelligently searching knowl- edge source. solve our task by developing IRP, framework that overcomes the limita- tions of retrieval-augmentte models and itera- tively performs content planning, fact retrieval, and rephrasing. Through experiments on three diverse, newly-collected datasets, we show that IRP produces factual and organized expository texts that accurately inform readers.!
University Montana is a public institution that was founded in 1893. total undergraduate enrollment 7,223 2021), its setting is urban, and the campus size is 220 acres...
Our Model (IRP)
University Denver private institution founded in 1864 has total of 5,867 students (fail 2021). it located the city. and the campus covers 128 acres...
LLaMA+Retr
University of Denver private Institttinn founded in 1864. It has a lotal of 11,462 students (fall 2021), ) its location is urban, and the campus covers 125 acres...
RAG
University of Denver public Institution founded in 1891. has total of 5,887 students (fall 2021), its location is urban, campus covers 120 acres...
Expository texts college descriptions pro- duced by IRP, 5-shot LLaMA equipped with DPR, and RAG, compared to ground truth. models use the topic and corpus inputs. Differently highlighted text indicates significant errors. Bold indicates similar style.
Figure 1:
1 Introduction
Expository writing intends inform reader about a topic in a clear and logical manner (Weaver and Kintsch, 1991). Such text highly prevalent on- line, appearing in various forms such as university descriptions, medical information, and Wikipedia articles. Despite its importance, writing expository text is difficult task, requiring the author to care- fully plan content for the text, obtain facts from multiple sources, and rephrase the facts the text flows smoothly (Thomas al.. 1987; Davis and Winek, 1989; Santos and Semana, 2015). Although it requires effort, expository writing vital for mak- ing complex information accessible to readers.
output (e.g. college description). The goal of ex- pository text generation is to provide readers with accurate and organized information for the topic.
To facilitate the goals of accuracy and organiza- tion, require the generated output contain up- to-date facts found in the knowledge source, and maintain a consistent style dictated by the expos- itory document domain. For example in Figure 1, the gold college texts are organized and phrased similarly, discussing the institutions' founding, en- rollment, setting, and size. However, finding these facts is nontrivial, as they may be scattered in the knowledge source. Further, sentences with the quired style may not exist in the knowledge source, so the style must be learned from expository texts in the training set. Thus, expository text generation models must tackle both challenges of (1) obtaining the dispersed, relevant facts from the knowledge
To ease these burdens, we propose the task of expository texi generation. As seen in Figure 1, the task uses input title topic (e.g. college name) and a knowledge source of topic-related factual sentences create a multi-sentence expository text
'Code available at https: / thub. www.eeeeeetettiooommmm

---

source; and (2) faithfully rewording said facts in learned style of the expository document domain.
tain said facts from the knowledge source. Finally, to generate the next sentence, the Paraphraser synthesizes the retrieved facts in the style of the content plan. This sentence is appended to the out- put document and the process is repeated until the Imitator decides that the output is complete. Over- all, IRP generates text seitence-by-sentence and retrieves facts with fine-grained queries to preserve factuality, while the Paraphraser maintains the style of the expository document domain. This design addresses the issues of -armeeeed LMs, resulting in factual expository texts that adhere to the style of the domain (shown in Figure 1).
Large language models (LLMs) (Brown al., 2020; Touvron et al., 2023) cannot directly be ap- plied to our task, given their inability to search for up-to-date information in corpora. Hence, a bet- ter solution is to leverage (Lewis et al., 2020b; Izacard et al., 2022). However, these models tend to produce inaccurate expository texts. For example, in Figure 1, we find that RAG and LLaMA with DPR generate texts that resemble the style of college descriptions, but hallucinate several facts, such as the institution's 's founding and enrollment, weakening the credibility of the text.
LMs
We study the effectiveness of IRP on three di- verse, newly-collected datasets: university descrip- tions, medical drug information, and computer sci- ence history Wikipedia articles. Through extensive experiments, we show that IRP produces more fac- tual expository texts compared to existing models. Further, human judges find the outputs of IRP to have the best balance between style and factuality. Our contributions can be summarized as follows: 1) We introduce expository text generation, which aims to generate factual and organized document that clearly informs readers about a specified topic. 2) We develop the IRP framework address the challenges of expository text generation. IRP iteratively and explicitly performs the key steps of content planning, retrieval, and paraphrasing. 3) We curate three new, diverse datasets to facilitate research on factuality and style in text generation.
These issues can be ascribed to two limitations of LMs. First, these models cre- ate text all at once rather than sentence-be bareecee-----eeeee senteence,, risking factual errors due to the challenge of model- ing long-range dependencies (Maynez al., 2020a; Ji al., 2022). Second, these models may fail to find nuanced facts the knowledge source, since they use the generic input topic as the query (Lewis et al., 2020b). For example, querying with a univer- sity name will likely not result in the retrieval of a nuanced fact like the university's 's ranking, However, if we perform sentence-level content planning (Hua and Wang, 2019) (e.g. plan that the next sentence should look something like: "The University of Illi- nois is ranked #32"), we can create fine-grained queries. Although these fine-grained queries may contain hallucinated facts (e.g. an incorrect univer- sity ranking), they will also include high-quality information-sseking keywords (e.g. "is ranked") that increase the chance of retrieving the correct facts, reducing factual errors.
4) In our experiments, we show that IRP produces highly factual texts that adhere the style of the expository document domain. Further, we conduct an ablation study and an error analysis to suggest future paths for expository text generation research.
To overcome these problems, we design frame- work named Imitate, Retrieve, Paraphrase (IRP). Rather than generating text all once, IRP operates at the sentence level and iteratively uses three key modules: the Imitator, Retriever, and Paraphraser. First, conditioned on the text that has been already generated for the output (or initially, a user-given prefix), the Imitator produces a stylistic content plan that outlines the next sentence of the output. The stylistic content plan may contain hallucinated entities, but it will correctly outline which entities should be discussed, making an effective query for retrieving facts to include in the next sentence. To create this plan, the Imitator is trained to mimic the style of expository texts in the training set.
2 Related Work
2.1 Retrieval-Augmenned Generation
Retrieval- -ugmmetee generation (RAG) models combine the strengths of retrievers and generators, and have been used for several knowledge-intente intenssive tasks (Petroni et al., 2021; Lewis et al., 2020b; Mao et al., 2021; Wang et al., 2021). Recent RAG mod- els aim to improve the retriever with re-ranking (Cao et al., 2018a; Ren al., 2021; Fajeik et al., 2021) and the generator by incorporating the evi- dentiality of passages (Asai et al., 2021).
Expository text generation a knowledge inten- sive task, as it leverages a knowledge source. How-
Next, the Retriever uses the content plan to ob-

---

Imitator (Sec 3.1)
Retriever (Sec 3.2)
Paraphraser (Sec 3.2)
Training
Examples
Samuel
Topic
appeared
1869
Machine Leaming
elymology
cieles
Reworded Sentennee
givenn Prefix
Retrieved Facts
1959
ML
by...
DistIBERT
Knowledige Sourse
GPT-2
Stylistic Content Plan
Culpuf
ML
WAk
nemmed
by
Noam
Chomsky
1984
Repeat af santancs-loval
Text
Figure 2: Overview of IRP. First, the Imitator (GPT-2) creates a stylistic content plan that outlines the facts to include in the next sentence. Next, the Retriever (DistilBERT) uses the content plan to find the outlined facts from the corpus. Finally, the Paraphraser (BART) rewords these facts in the style of the content plan. This sentence is appended to the output, which is used as the next prefix for the Imitator. These steps are repeated sentence-hy-senteree
ever, the IRP model has two differences from RAG models. First, to retrieve facts for expository text generation, IRP uses learned, fine-graine queries in the form of stylistic content plans, while typical RAG models use the document title as a query. Sec- ond, IRP is an iterative RAG model, meaning that it generates text sentence-by sentence-b--sentenee bestenn,,,, and attends to shorter pieces of text at a time. Through ablation studies, we find that both of these design choices improve the performance of IRP ($5.3).
in controllable paraphrase generation (Chen et al., 2019), data-to-text generation (Lin et al., 2020), and keys-to-text generation (Brahman et al., 2022).
Overall, expository text generation combines the challenges of (1) searching a knowledge source, (2)
preserving factuality, and (3) maintaining a consis-
tent style into a single knowledge-intensive task.
2.3 Summarization
Although summarization and expository text gen- eration both generate short texts, these tasks have inherently different objectives. Summarization fo- cuses on condensing information and thus, the out- put is distilled version of the input (Nenkova and McKeown, 2011). In expository text generation, seek to synthesize facts in consistent style, where the style is found in examples of expository texts, and the facts are dispersed in the knowledge source. Hence, expository text generation aims to synthesize new content with a focus on factual- ity and stylistic consistency, while summarization creates condensed forms of existing content.
Very recent and contemporaneous RAG models have been designed that iteratively retrieve facts, similar to IRP (Trivedi et al., 2022: Jiang et al., 2023). However, IRP uses smaller LMs trained for expository text generation, while these techniques are prompting methods for large LMs. To be used for our task, these methods require extra engineer- ing synthesize facts from multiple sentences in consistent style, we do not compare with them.
2.2 Factuality and Style in Text Generation
Factuality and style are two important areas of text generation research. Notable methods to im- prove factuality incorporate token constraints and re-ranking (Mao et al., 2020; Zhao et al., 2020; King al., 2022), modified training data (Cao et al., 2018b; Matsumaru et al., 2020), and custom train- ing objectives (Cao and Wang, 2021; Dong et al., 2022). Recently, researchers have enhanced fac- tuality by post-editing generated text (Dong et al., 2020; Cao et al., 2020; Balachandran et al., 2022). To control style in text generation, previous works have leveraged stylistic exemplars (Cao et al.. 2018a; 8a; Wei al., 2020; Wang al., 2022). More similar to IRP are models that closely adhere to the stylistic guidance, which have been explored
2.4 Wikipedia Generation
Another task that shares similarities with expository text generation Wikipedia generation (Sauper and Barzilay, 2009), which seeks to automatically create Wikipedia articles from article titles. Typi- cally, this is achieved by retrieving input text from the web followed by re-ranking and generation steps (Banerjee and Mitra, 2015, 2016; Liu et al., 2018; Pochampally et al., 2021). However, these models are often tailored for specific Wikipedia domains. Expository text generation is much broader task, encompassing more domains than just Wikipedia articles, with the unifying character-

---

istics of factuality and stylistic consisteney.
r), by first flattening Vi 1 (or r) into list of to- kens [1 82, .... Sm]- We initialize the causal language model with & and iteratively generate a content plan y/ = [ 8m+2; ..., < EOOS |>] un- til the end-of-sentence token is reached. By stop- ping at < EOS| >, we obtain a single sentence that outlines the content needed for the next sentence of the expository document. If GPT-2 generates the end-of-text token, the document is completed.
3 Method
The inputs for expository text generation include a topic for the expository text and 2) corpus of factual sentences C = {mz} related to topic L. We fix the corpus to compare models, but in practice, C can be acquired in real time for up-to-date facts. To guide the initial generation, IRP also uses 3) sequence of words r = {ru} to prefix the exposi- tory text. Using these inputs, IRP aims to produce sequence of sentences D {u.} comprise the expository text. The text D must contain accurate facts about from C, and must be presented in a consistent style dictated by expository texts in the training set 7 = (D,,..., D..).
3.2 Retriever
In order to effectively produce the information de- scribed in the content plan, we seek narrow the search space of where these facts could occur. Thus, given a stylistic content plan y/ produced by the Imitator, the Retriever p(zly)) searches for the top- candidate facts x S C that contain the content described in yt. We find that existing retrievers, such as DPR (Karpukhin et al., 2020) and BM25 (Robertson et al., 1995), may struggle to complete this task, the hallucinated factual entities in the content plan y; impair these models' search capabil- ities. For example, when generating an expository text for ML history, the content plan y/ may be the sentence "Machine learning was named by Noam Chomsky in 1984." The factual entities "Noam Chomsky" and "1984" should be ignored when searching for the correct facts, but DPR and BM25 still weigh these terms in their implementations.
As illustrated in Figure 2, IRP leverages three components: 1) style Imitator p(us|Uia 1) that generates stylistic content plan Yi for the next sentence in the expository document, based on the current state of the output V1:1-1 (or prefix in the first iteration); 2) a Retriever ((ry)ls)) that returns the top-k factual sentences x C most related to the content plan yı and 3) a Paraphraser p(z az yi) that combines the semantics of æ and the syntax of y; into a reworded sentence z. We will describe each of these modules, followed by how they are combined and trained for the full IRP model.
3.1 Imitator
To address this issue, we fine-tune DistilBERT (Sanh et al., 2019) with the task of classifying the index cach sentence of the expository texts in the training set (e.g. first sentence is labeled as 0), In doing so, DistilBERT gives lower token attribu- tion scores for factual entities, as sentences from different expository texts with the same index will not share these entities (Analyzed in $5.5). Dis- tIBERT performs fairly well on the classification task, given the consistent organization and style of expository texts in the training set.
To find facts for the next sentence of the expository text, we must first create a query to retrieve such facts. Hence, the Imitator (111)11111) generates a content plan yi in the style of the expository doc- ument domain for the next sentence in the output, conditioned the current sentences in the output Vi--- (or the prefix in the first iteration). To do so, we seek imitate the expert content planning of expository texts in the training set, achieved by minimizing the cross-entropy loss of token predic- tion (i.e. language modeling loss) for the exposi- tory texts in the training set T = (Di,..,,,,,) i.e.,
We compute the relevance of each sentence æj E C. to the content plan y by taking the dot product of æ and yıs both embedded by DistilBERT. To obtain these embeddings, through the classifier and take its representation in the last layer, averaged over all tokens:
Vwz C Da, YD2 C T:
feed each sentence
Dal logp(wy wi, w)-1)- (1)
Ximit
d=1==1
((z)) DistilBER((2;).
(2)
We leverage GPT-2 (Radford et al., 2019) mini- mize Ammit through causal language modeling. During each iteration of IRP, we create the stylis- tic content plan y, from sentences - (or prefix
( = DistilBERT(1))
(3)
p( (y:) ~ d( ((:)).
(4)

---

The top-k most relevant factual sentences C C Yi will have the k-highest values for p(ay) (3).. which can be obtained through Maximum Inner- Product Search (Shrivastava and Li, 2014).
Algorithm Imitate, Retrieve, Paraphrase
I: procedure 7,,
Initialize
output
3: Initialize 4-
p is the prefix
while true
3.3 Paraphraser
yc <- IMITATE(P) if y: enndofeext then
To ensure the expository document flows smoothly, we must reword the retrieved factual information in the style of the expository document domain. Thus, after obtaining stylistic content plan yi and factual sentences æ, the Paraphraser p(z[2r, ar, Va) must generate a single sentence z aligned the syntax y, and the semantics of æ. To achieve this goal, we formulate a variation on text generation with syntactic exemplars (Chen et al., 2019; Lin et al., 2020). We aim to minimize the cross-entropy loss of token prediction for 2, conditioned on Y/ and x:
7:
return
x RETRIEVE(M: C)
2 + PARAPHRASF(M,
10:
Append x
p p
to yi. Finally, the Paraphraser uses BART com- bine the syntax of y and the semantics of into a single sentence 2, which is appended the output D. The next prefix for the Imitator is set to D, and the process is repeated until the generated content plan is the < endoffext token.
Apara
(E x, 21, 2k-1). (5)
We minimize Apara with BART (Lewis et al.. 2020a), a seq2seq transformer-based language model. We modify the input so x and y, are sur- rounded by custom < fact >> and < <|stylel tokens, respectively. To add additional context to the Paraphraser, we include the topic of the expos- itory document t surrounded by a custom token < topic > in the input. Using the generated stylistic content plan yi, retrieved facts x, and the topic document of the document we train BART to generate the the next sentence the ground truth expository text.
3.5 Training
The Imitator, Retriever, and Paraphraser modules trained independently tackle expository text generation. We will now describe how we modify an expository text generation training set train each of these components. The training set con- tains triples of titles (input), factual corpora (input), and expository texts (output), i.e., (t,, C, D). The Imitator and Retriever are trained without modify- ing the training set, solely leveraging the expository texts. The Imitator performs causal language mod- eling with GPT-2 each document D, while the Retriever uses DistilBERT to classify the position of every sentence comprising each document D.
Our problem formulation differs from traditional text generation with syntactic exemplars, in that the input contains multiple sentences instead of one. This change is necessary, as the information outlined in the content plan yi may be distributed across multiple sentences. Thus, BART must learn to synthesize information from multiple sentences while adhering to the style of the content plan.
To train the Paraphraser, we require triplets of stylistic content plans ys sets of factual sentences x, and reworded sentences z. For given triplet, we can obtain the reworded sentence by selecting any of the sentences found an expository doc- ument D. Working backwards, we represent the stylistic content plan y/ as a sentence from a differ- ent expository document that has high similarity to 2, where similarity is calculated with Eq. 4. We obtain in similar manner, using top-k factual sentences x C C, also according Egq. 4. By using instead of yi to retrieve æ, we can more confident that x will contain the infor- mation needed to reconstruct 2, reducing the need for the Paraphraser to hallucinate during training.
3.4 The Iterative IRP Framework
The Imitator, Retriever, and Paraphraser are com- bined to generate expository documents, detailed in Algorithm 1. After the topic prefix r, and factual corpus C are provided, the Imitator first uses r as the initial context for GPT-2 generate stylistic content plan yi- Next, the Retriever embeds yi and each sentence x, E C with DistilBERT, in order to find the to factual sentences x C most similar
retrieve the

---

4 Experimental Setup
4.2 Baselines
We compare IRP with the following baselines: 1) LLaMA (Touvron et al., 2023) is an LLM shown to have competitive performance with GPT-3, We choose the 7B version of LLaMA and prompt with representative training examples. LLaMA pre- fixes its output with the same prefixes used by IRP. 2) LLaMA+Retr is LLaMA with an extra input of the top-5 retrieved sentences with DPR from the factual corpus using the document title as a query. 3) LED (Beltagy et al., 2020) is a seg2seq LM leveraging the Longformer model to encode and decode long documents. LED uses the topic and corpus as inputs to generate the expository text. 4) RAG (Lewis al., 2020b) uses DPR (Karpukhin et al., 2020) to retrieve the top-k facts from the in- put corpus using the document title as the query. Using the query and facts as inputs, we then train BART Large generate the expository document. 5) BART is trained generate the output using the topic the sole input. This model helps assess if other models use the factual corpus, or if they simply memorize the style of the expository text.
4.1 Datasets
We test the capabilities of IRP on three diverse, newly-collected datasets. 1) U.S. News is a cor- pus of 433 college descriptions from the top 500 ranked colleges on U.S. 2ews. We select the col- lege name as the topic of the document. 2) Med- line contains information for 844 medications from MedlinePlus, a medical library supported by the National Institute of Health. We select the medica- tion name as the topic of the document. 3) WikiCS is collection of the first paragraphs of history sections from 500 Wikipedia' articles in computer science. We select the Wikipedia article title as the topic of the document. For each dataset, we create a 70/10/20 train/validationntest split. No data from the test set used to train or validate any of the models or components of IRP. We provide full details for dataset collection in Appendix A.I.
As preliminary study for expository text gener- ation, we assume that the best corpus C has already been obtained for each document D. This is am approximation for the scenario where is acquired in real time. To obtain such ideal corpora, we col- lect documents from the web, reverse engineering with D. We web scrape sentences W from the top-5 web pages returned using the topic and each sentence of D as search queries. We exclude pages that contain the ground truth text D, such as any website with a URL containing "usnews" for U.S. News. In most cases, we find that the re- trieved sentences W provide the necessary facts for generating D ($5.4). But to guarantee a dataset that provides all facts, we create two versions of each dataset, one where C = W and one where C W U D, denoted by without doc and with doc, respectively. To introduce variation specifically for the with doc datasets, we perform back translation (Mallinson et al., 2017) on each expository text D and use it as the gold output, which we found not to affect its factual content and preserved its style (i.e. organization and phrasing). For the scope of this work, we assume that C contains accurate, consistent facts, and we believe future works could explore fact-checking (Rashkin al., 2017) for more robust expository text generation model. The corpora C are shuffled in each dataset.
4.3 Training Setup
IRP uses GPT-2 Large, DistilBERT Base, and BART Large for the Imitator, Retriever, and Para- phraser. We use "[topic] is." "[topic] is used to treat," and "[topic] was first created" as the prefixes for U.S. News, Medline, and WikiCS. These were selected by assessing common prefixes in the train- ing set. As a quality control check after generation, we filter sentences deemed repetitive by the Re- triever embeddings (cosine similarity above 0.98). We discuss more training details in Appendix A.2.
4.4 Quantitative Metries
We evaluate the quality of the generated documents with two sets of metrics. First, metrics. ROUGE-1 (R1) and ROUGE-2 (R2) (Lin, 2004), BLEU (Papineni al., 2002), and ME- TEOR (Denkowski and Lavie, 2014) measure the similarity between the predicted and true outputs.
traditional
However, as these metrics have low correlations with human judgements of factuality (Kryscinski et al., 2020: Fabbri et al., 2022), existing factuality metries. First, we calculate the average percentage of tokens in the generated text that are Hallucinated, meaning that they do not appear in the input corpus. Next, we use FaciCC, a classifier that is trained to detect factual errors
also adopt
/https://ww..uet www.ws wwwpe//ww..mbbbttcoooooloc $https: tttttps://meddineepple gov. /rugindu sittps://en.. htttts:://en.iikieeeee wikipedia.

---

Traditional Metrics
Factuality Metrics
Datasets
Models
BLEU
METEOR
Halluc
FactCC
NLI-Ent
NLI-Contr
Length
IRP (Ours)
0.911
0.828
0.802*
0.900
3,59
0.934
0.903
0.023
1.01
LLaMA
0.757
0.601
0.540
0.749
7.90
0.609
0.385
0.547
0.91
News
LeaM
0.759
0.613
0.548
0.753
7.56
0.608
0.387
0.562
0.91
with doc
LED
0.857
0.738
0.700
0.844
4.69
0.727
0.702
0.184
0.98
RAG
0.788
0.651
0.611
0.821
7.40
0.475
0.402
0.515
1.07
BART
0.774
0.621
0.586
0.807
9.94
0.341
0.255
0.682
1.04
IRP (Ours)
0,551
0.435
0.318
0,490
1.71
0.871
0,521
0.126
0.66
LLaMA
0.398
0.196
0.127
0.301
2.33
0,875
0.326
0,165
0.67
Medline
&Retr
0.484
0.282
0.206
0,403
2.89
0,853
0.368
0.181
0.77
with doc
LED
0,671
0,549*
0,453*
0,611
2.32
0,933
0,571
0.249
0.68
RAG
0.587
0.446
0.385
0,526
3.03
0.866
0.355
0.207
0.85
BART
0.400
0.205
0.179
0.355
12.12
0.805
0.098
0.379
0.94
IRP (Ours)
0.490*
0.372*
0.338
0.442
0.68
0.616
0.355
0.147
0.88
LLaMA
0.166
0.025
0.010
0,151
9.37
0.366
0.057
0.525
1.92
WikiCS
LeMA
0.169
0.025
0.010
0,153
4.51
0.740
0.172
0.250
1.93
with doc
LED
0.250
0.120
0.046
0.158
0.79
0,518
0,141
0.162
0.45
RAG
0.327
0.2011
0.076
0.228
1.11
0.567
0.326
0.170
0.42
BART
0.231
0.047
0.009
0.129
11.22
0.390
0,145
0,340
0,43
IRP (Ours)
0.807*
0.675*
0.649*
0,816
10.61
0.609
0.470
0.437
1.01
LLaMA
0.757
0.601
0.540
0.749
12.34
0.609
0.385
0.547
0.91
U.S. News
RetM
0.759
0.612
0.547
0.753
11.86
0.602
0.382
0.567
0.91
without doc
LED
0.792
0.624
0.813
0,776
10.71
0.539
0.468
0.544
1.09
RAG
0.793
0.653
0.613
0.824
11.84
0.449
0.351
0.593
1.05
BART
0.774
0.621
0.586
0.807
14.30
0.341
0.255
0.682
1,00
IRP (Ours)
0.512
0.347
0.257
0.446
2.76
0,883
0.448
0.171
0.69
LLaMA
0.398
0.196
0.127
0.301
3.11
0.875
0.326
0.165
0.67
Medline
Letr
0.388
0.182
0.111
0.287
2.82
0.869
0.319
0.174
0.65
without doc
LED
0.545
0,389
0.244
0.450
2.92
0.873
0,491
0.173
0.64
RAG
0.548
0.369
0.324*
0.504
4.12
0.819
0.356
0.213
0.96
BART
0.400
0..055
0.179
0.355
12.50
0.805
0.098
0.379
0.94
IRP (Ours)
0.380*
0.209*
0.159
0
0.491
0.263
0.193
0.82
LLaMA
0.166
0.025
0.010
0.151
0.525
WikiCS
LetMA+Retr
0.172
0.010
0.151
0.736
0.178
0.257
1.89
without doc
0.261
0.107
0.033
0.169
0.302
RAG
0,312
0.139
0.060
0,214
3.42
0.537
0,237
0.220
BART
0.231
0.047
0.009
0.129
11.48
0.390
0.145
0.340
0.43
Table 1: Comparison of traditional metrics (ROUGE-1, ROUGE-2, BLEU, METEOR) and factuality metrics (Hallucinations,, FactCC, Entailment, Contradictions) for expository text generation models. Models with values marked with significantly outperform all baselines (p 0.05, Wilcoxon signed-rank test (Woolson, 2007)).
between a source text and claim (Kryscinski et al., 2020). We use the true output as the source and each sentence of the generated text as the claim, and report the average proportion of source/claim pairs predicted as factually consistent. Finally, as research has suggested that natural language infer- ence has high correlation with human judgements of factuality (Maynez al., 2020b), we calculate if the generated text entailed (NLI-Ent) or con- tradicted (NLI-Contr) by the true output. We train a DiT classifier (accuracy of 0.82) on the MNLI dataset (Williams et al., 2018), and report the average proportion of generated sentences that are predicted to be entailed/contradieted by the true output. All metrics are reported from single run.
Results
5.1 Performance Comparison
In Table we observe that IRP obtains the highest factuality scores, achieving the strongest results for 19 out the 24 calculated factuality metrics. Further, we find that apart from Medline, IRP out- performs baselines on almost all traditional metrics. These findings confirm that the Paraphraser faith- fully rewords the retrieved facts in the style of the expository document domain. Previous works have shown that prioritizing factuality leads to a drop in ROUGE (Goodrich al., 2019; Maynez et al., 2020b). However, IRP adheres factual accuracy while maintaining the style of the expository docu- ment domain, suggesting the benefit of iteratively

---

with doc
without doc
Model
R1
Hallue
FactCC
NLI-EEnt
NLI-Contr
Dataset
Model
Style
Fact
Avg
Style
Fact
Avg
IRP Full
0.490
0.68
0.616
0.355
0.147
IRP (Ours)
4.92
4.57
4.75
4,90
3.80
4.35
Topic Query
0.165
2.58
0.329
0.201
0.229
ReaMA+Retr
4,82
3.30
4.06
4.77
3.25
4.01
Gen All
0.412
2.62
0.493
0.276
0.240
U.S.
LED
4.65
3.78
4.22
4.42
2.88
3.65
News
RAG
4.75
2.70
3.73
4.73
2.85
3.79
Table 3: RI and factuality of IRP versus Generating
ChatGPT
4.72
4.10
4.41
4.72
4.10
4.41
text All at once instead of iteratively, and using Topic
IRP (Ours)
4.53
4.78
4.66
4.12
4.88
4.50
Query over stylistic content plans on WikiCS with doc.
Ret
3,48
4.55
4.02
3.52
4.28
3.90
Medline
LED
4.32
4.08
4.20
4.28
4.30
4.29
RAG
4.28
4.15
4.22
4.10
4.13
4.12
ChaiGPT
3.53
4.83
4.18
3.53
4.83
4.18
put and encourage them to use external resources (Google Search). We observe high annotator agree- ment for Fact and Style, with Krippendorff's s a (Krippendorff, 201 around 0.70 on each dataset. IRP strikes the best balance between factuality and style (Avg) in 3/4 datasets and competes with ChatGPT on the fourth dataset (Table 2), despite having less parameters (1.2B vs 175B). Generally, note that the LLMs (ChatGPT, LLaMA+Rety perform better in factuality but worse in style, while the opposite true for seq2seq LMs (LED, RAG).
Table 2: Human evaluation of style and factuality of ex- pository documents on 5-Likert scale. Avg is the aver- age of style and factuality scores. ChatGPT is prompted with five representative training examples.
performing content planning, retrieval, and para- phrasing for expository text generation.
We also find that LLaMA+Retr does not consis- tently outperform LLaMA, implying that the LLM equipped with DPR cannot effectively search and use the factual corpus. Further, although RAG is typically effective in knowledge intensive tasks, the model obtains lower factuality scores than IRP in 23/24 metrics. Both findings suggest that the document title used by LLtMA+Retr and RAG is an insufficient query to retrieve the factual infor- mation needed to produce expository text. Hence, fine-grained queries, such the stylistic content plans used by IRP, are better strategy for obtain- ing all of the facts to include in the expository text. Finally, we note that most models achieved much better performance on the with doc datasets. How- ever, the with doc scenario is unrealistic, indicating that future models should prioritize the knowledge acquisition step of C, as it largely dictates the factu- ality of the output. We believe that studying models that search the web during inference (e.g. LLMs with search engines) is promising next step to- ward stronger expository text generation models.
5.3 Ablation Study
We conduct an ablation study (Table 3, full results Table 6) and find that using stylistic content plans over the topic as a query and generating text at the sentence level instead of all at once, both improve the performance of IRP. This suggests that an itera- tive RAG model that creates fine-grained queries, such as IRP, preferred strategy for our task.
5.4 Factual Error Investigation
To study the errors produced by IRP, we invite one computer science student to annotate 30 expository texts produced by IRP on the without doc datasets. First, we ask the annotator to identify errors, i.e., facts in the generated text that do not exist in the true output. We then ask if each error occurred because 1) an alternative fact exists in the retrieved sentences (i.e. no hallucination), 2) no suitable fact could have been found by the Retriever, as it does not exist in the corpus, 3) the fact exists in the corpus, but the Retriever could not locate it, or 4) the Retriever located the correct fact, but the Paraphraser hallucinated. We store each step of IRP so the annotator can answer this question.
5.2 Human Evaluation
To further test the quality of the generated expos- itory texts, we invite two computer science and engineering college students to evaluate 30 random expository documents in the test sets produced by each bascline U.S. News and Medline. Similar previous works (Hua and Wang, 2019; Balachan- dran et al., 2022), we ask annotators evaluate on Style adherence to the true output (i.e. organization and phrasing), and Factuality on a 5-Likert scale. For Fact, we provide annotators with the true out-
We find that many factual errors are due alter- natives existing in the retrieved facts, rather than the weaknesses of the Retriever, Paraphraser or data collection (Figure 4). For example, one source may report an outdated university ranking compared to
'tttps: //t thub. com/cdpl erse/ transformers interpret

---

Original: Emory University is a private institution founded 1836
U.S. News
emory university
private institution founded 1836
Original: Hydroxyzine belongs to class of medications called antihistamines
Medline
hydro ##xy
##zzine
belongs class of medications called anti #thiss ##taam #ilines
Original: The name Prologue was born around 1972 by Alain Colmerauer and Philippe Roussel
WikiCS
the name prologue was born around 1972 by alain col #ilmeer #ifauer and philippe ro #ituss #ifel
Figure 3: Visualized token attribution scores for the classification task performed by the Retriever on a sample of sentences from each test set. Darker shades of blue indicate higher token attribution scores. Scores calculated with transformers-imterrre"" a Python library that leverages Captum (Kokhlikyan et al., 2020).
Altemative
Paraphraser
accuracy while maintaining stylistic consistency, Our ablation studies confirm the importance of sentence-level generation and creating fine-grained search queries. Finally, we visualize the Retriever of TRP and study the factual errors produced by our model to suggest future research directions.
Medline
Wkics
0.50
0.75
1.00
Limitations
Figure 4: Distribution of IRP factual error types.
One drawback of IRP is that three separate com- ponents are trained for each expository document domain. For our initial exploration of expository text generation, train three separate components, as it made our model more interpretable and it thus simpler to detect errors in our components. The fact that IRP is not trained end-to-end sug- gests that there is still room for improvement and research in expository text generation.
U.S. News. This poses an interesting question for future lines work on expository text generation: how can we leverage fact verification to accurately select information when multiple options exist?
5.5 Retriever Embedding Analysis
The Retriever ignores hallucinated entities when creating embeddings, resulting in better search ca- pabilities compared to pretrained retrievers. We visualize this property in Figure and find that the Retriever puts less weight on factually specific en- tities (e.g. "Emory" and "Prologue"). Hence, the Retriever can focus its embeddings on the more im- portant terms for retrieving information (c.g. "be- longs," "class," and "medications"). We show the strength of the Retriever quantitatively in Table 5,
In addition, IRP is computationally more expen- sive than traditional RAG models, scales linearly with respect the number of sen- tences in the output. Although IRP shows improve- ments over baselines, we acknowledge that is im- portant to ensure IRP is computationally efficient. To overcome this issue, we believe that future it- erative RAG models could improve upon IRP by exploring efficient algorithms for maximum inner- product search, well as developing batched version of IRP that can generate multiple sentences in tandem.
runtime
5.6 Sample Expository Document Outputs
We provide examples of documents generated by models on our three datasets in Appendix A.3.
Lastly, we find that expository text generation frameworks have large performance gap between the with doc and without doc datasets ($5.1). As discussed in the paper, we believe this gap can be overcome by studying and developing models that can retrieve information from the web in real time during inference. For example, instead of using stylistic content plans to search a provided factual corpus, perhaps they could be reworded into search queries to retrieve up-to-date information
6 Conclusion
We introduce the task of expository text genera- tion and design IRP to overcome the limitations of existing models. IRP explicitly and iteratively performs content planning, fact selection, and para- phrasing to generate high quality expository texts. Automatic and human evaluations on three newly- collected datasets reveal that IRP preserves factual

---

from Google in real time, thus overcoming any limitations of a provided factual corpus. If future work this direction results in expository text generation models that can perform live retrieval during inference, they can also be compared and benchmarked with LLMs that are equipped with web search engine plugins.
Vidhisha Balachandran, Hannaneh Hajishirzi. William Cohen, and Yulia Tsvetkov. 2022. Correcting diverse factual errors in abstractive summarization via post- editing and language model infilling. In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Lunguage Processing, pages 981 $18 88 8-9830, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Siddhartha Banerjee and Prasenjit Mitra. 2015. Wikikreator: Improving wikipedia stubs automat- ically. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguisties and the 7th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 867-877.
8 Ethical Considerations
The fundamental goal of IRP is to generate factual content. However, as with all text generation frame- works, IRP may produce factual errors, as shown in $5.4. Future expository text generation models could ameliorate the harms of factual errors by per- forming fact verification, retrieving live informa- tion from the web during inference, incorporating more knowledge sources, or attributing the source of the generated facts for increased transparency.
Siddhartha Banerjee and Prasenjit Mitra. 2016. Wiki- write: Generating wikipedia articles automatically. In IJCAI, pages 2740-2746.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Further, the Paraphraser is the key component in IRP that ensures the generated text is faithful to the factual corpus. However, there is always the pos- sibility of someone creating their own Paraphraser aligned with the goal of producing misinformation or deceptive claims from true information and plug- ging this malicious component into IRP. We hope that future research will result in safeguards to de- tect and combat these potential risks of seq2seq language models.
Faeze Brahman, Baolin Peng. Michel Galley, Sudha Rao, Bill Dolan, Snigdha Chaturvedi, and Jianfeng Gao. 2022. Grounded keys-to-text generation: To- wards factual open-ended generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7397-7413. Associa- tion for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-190.
9 Acknowledgements
We thank the anonymous reviewers for their feed- back. This material is based upon work supported by the National Science Foundation IIS 16- 16-19302 and IIS 16-33755, Zhejiang University ZJU Re- search 083650, 1BM-Illinois Center for Cognitive Computing Systems Research (C3SR)- a research collaboration as part of the IBM Cognitive Horizon Network, grants from eBay and Microsoft Azure, UIUC OVCR CCIL Plamning Grant 434S34, UIUC CSBS Small Grant 434C8U, and UIUC New Fron- tiers Initiative. Any opinions, findings, and conclu- sions or recommendations expressed in this publi- cation are those of the author(s) and do not neces- sarily reflect the views of the funding agencies.
Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstrac- tive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Procexsing (EMNLP), pages 625 1-51-6 Online. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. 2021. CLIFF: contrastive learning for improving faithfulness and factuality in abstractive summarization. CORR, abs/2109.0920 0999099..
Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2018a. Retrieve, rerank and rewrite: Soft template based neural summarization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 152-161.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018b. Faithful to the original: Fact-aware neural abstrac- tive summarization. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial In- telligence Conference and Eighth AAAI Symposium on Educcational Advances in Artificial Intelligence. AAAI 18/IAAI 18/EAAT 18, AAAI Press.
References
Akari Asai, Matt Gardner, and Hannaneh Ha- jishirzi. 2021. Evidentialit--uudeed generation for knowledge-intensiee NLP tasks. CoRR, abs/2112 12.08688.

---

Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase gener- ation with syntactic exemplar. In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguistics, pages 5972-5984, Florence, Italy, Association for Computational Linguistics.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lu- cas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.0322
Susan J Davis and Janice Winek. 1989. Improving expository writing by increasing background knowl- edge. Journal of Reading. 33(3):178-18)
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang. Andrea Madotto, and Pascale Fung. 2022. Survey of halluci- nation in natural language generation. ACM Comput. Surv. Just Accepted.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings the Ninth Workshop Staristical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Ac- tive retrieval augmented generation. arXiv preprint
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung. and Jingjing Liu. 2020. Multi- fact correction in abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9320-9331, Online. Association for Computa- tional Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Fdduno, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Yue Dong, John Wieting, and Pat Verga. 2022. Faithful to the document or to the world? mitigating hal- lucinations via entity-linked knowledge in abstrac- tive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022. pages 1067-1082, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Daniel King, Zejiang Shen, Nishant Subramani, Daniel S. Weld, Iz Beltagy, and Doug Downey, 2022. Don't say what you don't otn't know: Improving the con- sistency of abstractive summarization by constraining beam search. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Met- rics (GEM), pages 555-571, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA- based factual consistency evaluation for summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2587-2601, Seattle, United States. Asso- ciation for Computational Linguistics.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Rebliz--Richardson. 2020. Captum: A unified and generic model interpretability library for pytorch. CoRR, abs/2009.07899
Martin Fajcik, Martin Docekal, Karel Ondrej. and Pavel Smrz. 2021. R2-D2: A modular bascline for open- domain question answering. In Findings of the As- sociation for Computational Linguistics: FMNLP 2021, pages 854-870, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Klaus Krippendorff 2011. Computing krippendorff's
reliabiiii.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332 2334,, Online. Association for Computa- tional Linguistics.
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of gener- ated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '1,, page 166-175, New York, NY, USA. Association for Computing Machinery.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising soweence-t--squenne pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871 1871-7880, Online, Association for Computa- tional Linguistics.
Xinyu Hua and Lu Wang. 2019. Sentence-level content planning and style specification for neural text gen- eration. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLA pages 191- Hong Kong, China. Association for Com- putational Linguistics.

---

Patrick Lewis. Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin. Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020b. Rerneval--ugmeetee generation for knowledge-intensivee nip tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474,
Computational Linguistics, pages 1906-1919, On- line. Association for Computational Linguistics.
Ani Nenkova and Kathleen McKeown. 2011. Auto- matic summarization. Foundations and Trends@ in Information Retrieval, 5(2-3))103-233..
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Anmual Meeting on Association for Computa- tional Linguistics, ACL '02. page 311-318, USA. Association for Computational Linguisties.
branches ouf, pages 74-81.
Shuai Lin, Wentao Wang, Zichao Yang. Xiaodan Liang, Frank F. Xu, Eric Xing, and Zhiting Hu. 2020. Data- to-text generation with style imitation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1589-1598, Online. Association for Computational I Linguistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: benchmark for knowledge intensive language tasks, In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pages 2523 2544, Online. Association for Computational Linguisties.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Compu- tational Linguistics: Volume I, Long Papers. pages 881-893, Valencia, Spain. Association for Computa- tional Linguistics.
Yashaswi Pochampally, Kamalakar Karlapalem, and Navya Yarrabelly. 2021. Semi-supervised automatic generation of wikipedia articles for named entities. Proceedings of the International AAAI Conference on Weh and Social Media, 10(2):22-79
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188::
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Ham, and Weizhu Chen. retrieval for open-
2021. inneeration-uugmentee domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I Itth International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 4089-4100, Online. As- sociation for Computational Linguistics.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and po- litical fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2931-2937, Copenhagen, Denmark. Association for Computational Linguis- tics.
Yuning Mao, Xiang Ren, Heng Ji, and Jiawei Han. 2020. Constrained abstractive summarization: Pre- serving factual consistency with constrained genera- tion. CORR, abs/2010.12723.
Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2027 Conference on Empiri- cal Methods in Natural Language Processing, pages 2825 2835, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Kazuki Matsumaru, Sho Takase, and Naoaki Okazaki. 2020. Improving truthfulness of headline genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1335-1346, Online. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020a. On faithfulness and factu- ality in abstractive summarization. arXiv preprint arXiv:2005.0066)
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp. 109:109.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020b. On faithfulness and factu- ality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108.

---

Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael valuable than you Leonor Santos and Sílvia Semana. 2015. Developing mathematics written communication through exposi- tory writing supported by assessment strategies. Ed- ucational studies in mathematics, 88(1))65-87.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. broad broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Christina Sauper and Regina Barzilay, 2009. Auto- matically generating wikipedia articles: A structure- aware approach. In Proceddings of the Joint Confer- ence of the 47th Annual Meeting of the ACI and the 4th International Joint Conference on Natural Lan- guage Processing of the AFNLP: Volume 1- Volume ACL '09. page 208-216, USA. Association for Computational Linguistics.
Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, pages 1-3.
Zheng Zhao, Shay Cohen, and Bonnie Webber. 2020. Reducing quantity hallucinations in abstractive sum- marization. In Findings of the A Association for Com- putational Linguistics: EMNLP 2020, pages 2237- 2249, Online. Association for Computational Lin- guistics.
Anshumali Shrivastava and Ping Li. 2014. Asymmetric Ish (alsh) for sublinear time maximum inner prod- search (mips). Advances in neural information processing systems, 27.
Carol Chase Thomas, Carol Sue Englert, and Stephanic Gregg. 1987. An analysis of errors and strategies in the expository writing of learning disabled students. Remedial and Special Education, 8(1):21-30.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302. 13971.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleav- ing retrieval with chain-of-thought reasoning for knowledge-intensiin multi-step questions. arXiv prepriat arxiv:22 105099.
Han Wang, Yang Liu, Chenguang 7hu, Linjun Shou, Ming Gong. Yichong Xu, and Michael Zeng. 2021. Retrieval enhanced model for commonsense gener- ation. In Findings of the Association for Computa- tional Linguistics: ACL-IJCNLP 2021, pages 3056- 3062, Online. Association for Computational Lin- guistics.
Zeng. 2022. Training data
think: simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume Long Papers), pages 3170-3179, Dublin, Ireland. Association for Computational Lin- guistics.
Ch A Weaver and Walter Kintsch. 1991. Expository text. Handbook of reading research, 2:230-245.
Bolin Wei. Yongmin Li. Ge Li. Xin Xia, and Zhi Jin. 2020. Retrieve and refine: exemplar-based neural comment generation. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 349-360.

---

A Appendix
A.2 Detailed Training Setup
The Imitator is trained with GPT-2 Large (774M parameters) through the aitextgen 10 Python pack- age. We choose a batch size of 1, a learning rate of le-3, and train the model for 3000 steps. All other parameters are set to the default value of the aitextgen implementation. The Retriever is trained with DistilBERT Base uncased (66M pa- rameters). We choose a batch size of 16, a learning rate of 2e-5, a weight decay of 0.01, and I train- ing epoch. The Paraphraser is trained with BART Large (406M parameters). We choose batch size of 32, learning rate of 2e-5, weight decay of 0.01, gradient accumulation step size of 8, and 5 training epochs. The Paraphraser takes ~2 hours to finish training. Each component of IRP is op- timized with the Adam optimizer and trained with a single NVIDIA A40 GPU. The parameters, resources, and models remain the same for each dataset, with only slight differences in training time. During prediction, GPT-2 uses temperature of 0.7 and generates text with maximum length of 512, DistilBERT retrieves k 15 factual sentences, and BART generates text with a maximum input and output length of 512.
A.1 Detailed Dataset Collection
To obtain the expository documents D on each dataset, we web scrape the respective websites with BeautifulSoup". We could not find specific research licenses for the three datasets, but note that they are free to access and publicly available online. Further, we found that each dataset has been analyzed in previous NLP research papers.
For given expository document D and its topic 6, we will now explain how we obtain the set of fac- tual sentences W, briefly described in $4.1. First, we break up D into a set of sentences {y:}. For each sentence y/s we obtain the URLs of the top 5 search results using the query "[t] [vi]". After repeating this for each sentence, flatten the list of URLS into a unique set, and filter the URLs that contain the ground truth expository document (c.g. for the U.S. News dataset, we filter URLs which contain the substring "usnews"). We then use Beau- tifulSoup to obtain the text of all of the <p> tags. Using the nltk sentence tokenizer', we extract all sentences and flatten them into unique set.
LED uses an input size of 16384 and is trained with a batch size of 8, learning rate of 5e-5, 8 gra- dient accumulation steps, 1500 warm-up steps, and trained for 8 epochs. The generator of RAG and the BART baseline are trained with the same pa- rameters as the Paraphraser. The retriever of RAG selects k 15 sentences. All unstated parameters are the default values of their respective implemen- tations. We ensure that each model is trained until the validation loss converges.
We clean sentences by keeping alpha-numeric symbols and punctuation with regex, as well as applying unidecode* to ensure sentences contain only ASCII characters. All information is in En- glish, and we studied a sample of sentences en- sure that there was no offensive language in the dataset. By analyzing a sample of the dataset, we did not find any personal identifiable information (PII), but to be cautious, we use the Presidio an- alyzer provided by Microsoft and remove all sen- tences with detected PII (prediction score > 0.3). encompassing the following entities: "PHONE NUMBER", "CRYPTO", "EMAIL ADDRESS". "IBAN CODE", "IP ADDRESS", "MEDICAL LI- CENSE", "US BANK NUMBER", "US DRIVER LICENSE", "US ITIN", "US PASSPORT", "US SSN". In Table 4, we display summary statistics of each dataset after this process.
For the LLaMA and LLaMA+Reir, perform 5-shot prompting using 5 manually selected, rep- resentative input/output training examples, an ex- ample of which is shown in Figure 5. For Chat- GPT, we use the web interface and perform 5- shot prompting. We assess the outputs of all base- lines and perform the same quality control check as IRP to filter semantically repetitive sentences, improving fluency.
A.3 Qualitative Analysis
"htrpps:/ppppt.oor/prooooe
In Tables 7, 8, and 9, we present examples of expos- itory documents generated by IRP on U.S. News, Medline, and WikiCS, respectively, on both ver-
beautifulsouph
hht.p/::/www...tt...rr//////// tokenize.btmi
hhttps://docs http://ddoccaattettee.e.ee aitextgen.iin. "nttps://chat cowpaaa...peaa.ccmmmmm
analyzer/

---

sions of the datasets (with doc and without doc). We also display the topic of the expository docu- ment and the true output. these examples, we can see that IRP produces text with high factual accuracy without sacrificing fluency or the style of the expository document domain.
Further, in Tables 10, 11, and 12, we di- rectly compare the expository document outputs of IRP and the baselines (LED, RAG, LLaMA, LLaMA+Retr) on U.S. News, Medline, and Wi- kiCS, respectively. On U.S. News, we find that the baselines tend to produce factual errors related to many of the key details, such as the institution's founding and tuition. On Medline, we find that the baselines struggle to generate accurate drug classes and explanations for how the medications affect the human body. Some generated documents also contain phrases that are repetitive and difficult to understand. On WikiCS, we find that the baselines are mostly factually accurate, but the documents lack overall structure and coherence. Compared to other models, LLaMA struggles the most with preserving the style of the expository document domain.
A.4 Human Evaluation
We display the set of instructions given to human annotators for evaluating the style and factual curacy of expository documents in Figure 6.

---

fillec { cacha line }
output: ( Earty examples CPU cachos are the Atlas 2 and the IBM System 360 Medel BS the L-1 forr instrcciinn).
level later Level caches, CPU, became mainstroam the late 1980s and entored the CPU markat with the ARMYSTE in 1997. in 2016, the L1 cache was even spiii sub-dollar SoCs acoiton, there are L2 caches there, and for larger processors there also torches. The 12 tocche
1980s. first GPUs
split
started
with
repository muls oore prosessor has dedicated L1 and
not shared and functions
already charec cache, nomeally shared between the CPUs The cores can be split between caches localed higher and higher.
titl:: excrppiio handlngg) output: The hardware excsplion handlngg found UNMAC 1961. Arthmmtic ovartoo eseouled two commands address that could tnmraerr control connell the reaul.))
fille: kandiionll progremmmigg }
The lambda calculation, develazed Alarzo Church the functional applicationa 1957, Turinn proved that the calouiation Turing machines are cquivallnt mathematcall models and showed that the lambda cal calcutttion complet. Lambca calcurtiion torms the bass of functional programming languages, A correspanding theorctical formutation, combinatorial Mozes Schönfinkel Haskell Curry
formal calculation
systam
developed 1920s 1930s
titer: ( variatiionl inqqusiites }
output: ( The first problem with varying invqually was the Signonnn problem niiedd by Antonio Signarini by works of thaory werw (Fiches 1983) and (Fichem 1964w), (Fichana 1984b) Later Gurdo Stampacohhs proved he generalicsion of the Mir-Migram theorem in (Stempacchi 1864) investiaale the reguler by problem for partill difirrenial equations and csired the name "Inequally of Varistion" protiens this
and
references (Antmen sp. 252-2204
Fichwns 1966)
Georges
encouraged
sludy
attanding conforence in study on the Signonni prob em, as reported by Antman 1983, p. 253: thus the theory becamo widely known throughout France.) }
expend Fichen''s
1985 Fichara presientd
quantum computing )
output: ( Quantum computing resoarch began 1990, when phaystast Poul Benioff proposed a quantum mechanical model of the "uring machine Richard Feymman and Yuri Manin later suggested that
computer
simulate things that
sould do. 1986,
Feyrmmn indroduced esrty
quentum
quantum algonthim find the primary factors of an integrator with the potentall to decrypt RSA ris--enryppedd communccaions. In 1895. hevic Chusng, Nei Censrenfeld and Mark Kubine creeted the lirst quantum computer
bits perform calculations, Despite ongaing experimenaal distard that
the
1990s, reweercherss dream" recont years, invastment quantum computer research the public and privato sectors incrused On Ockoer 23, 2019, Googhe All in partneshipp with MABA' Mational Squson Adminittaaionn (NASA) slll dissical topic.)
quantum compuling
Btle: locive Inteligennc)
output: ( Collective Intellgencee was first created
Figure 5: Prompt given for LLaMA (no retrieval) WikiCS. For LLaMA+ add "input: [informatio]]" to the each in-context learning example, where information is the concatenated facts retrieved from DPR.
Average Number of Facts with doc / without doc
Dataset
Train Valid/ / Test
Average Output Length (words)
U.S. News
315 / 39 79
52188 / 517.13 1002.89 / 999.49 .77 973.58 970.58
73.38
Medline
590 / 85 / 169
84.31
WikiCS
350 / 50 100
90.33
Table 4: Summary statistics of U.S. News, Medline, and WikiCS datasets for expository document generation.

---

with doe
without doc
Model
U.S. News
Medline
WikiCS
U.S. News
Medline
WikiCS
IRP-DistlBBERT
0.665
0.926
0.703
0.578
0.771
0.580
IRP-DPR
0.643
0.795
0.573
0.642
0.716
0.573
Table 5: IRP Retriever comparison using our DistilBERT setup versus DPR as the Retriever. We run both IRP variations on each dataset where the Retriever obtains the top-5 factual sentences each step. We store these factual sentences and calculate the average ROUGE-1 recall performance between all factual sentences and the ground truth output, which indicates the proportion tokens in the output that were covered by the retrieved factual sentences. On 5/6 datasets, DistilBERT retriever outperforms DPR (best results in bold).
Traditional Metries
Factuality Metrics
Dataset
Model
RI
R2
BLEU METEOR
Hallue
FactCC
NLI-Ent
NLI-Contr
IRP Full
0.911
0.828
0,802
0.900
3.59
0.934
0.903
0.023
U.S. News
Gen All
0.764
0.611
0.592
0.793
8.51
0.503
0.394
0.502
with doc
Topic Query
0,301
0.170
0.132
0.320
3.64
0.307
0.599
0.337
Full
0.551
0.435
0.318
1.71
0.871
0.521
0.126
Medline
Gen
0.534
0.368
0,318
4.84
0.145
0.378
0.168
with doc
Topic Query
0.281
0.141
0.137
0,244
3.92
0.066
0.330
0.156
IRP Full
0.490
0.372
0.338
0.442
0.68
0.616
0,355
0.147
WikiCS
Gen All
0.412
0.269
0.227
0.331
2.62
0.493
0.276
0.240
with doc
Topic Query
0.165
0.067
0.022
0.113
2.58
0.329
0.206
0.229
Table 6: Comparison traditional metrics (ROUGE-1, ROUGE-2, BLEU, METEOR) and factuality metrics (Hallueinations, FactCC, Entailment, Contradictions) for IRP ablations. Gen All generates the text once rather than pentence-by-sentens Topic Query uses the topic as the search query in the factual corpus instead of the stylistic content plans used by IRP. Best results are bold.

---

University (Topie)
True Output
IRP With Doc
IRP Without Doc
Stanford University
Stanford University is a private institution founded 1885. It has a total of 7.645 students (fall 2021), its location is sub- urban, and the campus is 8, 180 acres, uses a quarterly aca- demic calendar. Stanford Uni- versity ranks 3rd in the 2022- 2023 edition of the best col- leges. Its tuition and fees are $56,169.
Stanford University is private institution founded 1885. It has a total of 7.645 students (fall 2021), its location is sub- urban, and the campus is 8,180 acres. uses quarterly acu- demic calendar. Stanford Uni- versity ranks 3rd in the 2022- 2023 edition of the best col- leges. Its tuition and fees are $55,169.
Stanford University is a private institution founded 1885. It has a total 7,645 students (fall 2021), its location is sub- urban, and the campus is 8, 8,180 acres. uses quarterly aca- demic calendar, Stanford Uni- versity ranks 3rd in the 2022- 2023 edition of the best col- leges. Its tuition and fees are 156.169.
Grand Valley State University
Grand Valley State University is public institution founded in 1960. It has a total of 19,379 students (fall 2021), its location is suburban, and the size of the campus is 1,391 acres. It uses semester- based academic cal- endar. Grand Valley State Uni- versity runks 263rd among the top colleges in the 2022-2023 edition of the National Univer- sities. Its tuition and fees in the states are $13,952; out-of-statc tuition and fees are $19,856.
Grand Valley State University is public institution founded in 1960. It has total of 19,379 students (fall 2021), its loca- tion is suburban, and the cam- pus is 1,391 acres. It uses semester- based academic cal- endar. Grand Valley State Uni- versity ranks 263rd among the top colleges in the 2022-2023 edition of the National Univer- sities. Tuition and fees in the states are $13,952; out-of-state tuition and fees are $19,856.
Grand Valley State University is a public institution founded in 1960. It has total of 16,851 students (fall 2021), suburban location, and 1,338-acre cam- pus. It uses semester-based academic calendar. Grand Valley State University ranks 263rd among the top colleges in the 2022-2023 edition of the National University Ranking. Tuition and foos are $11,952 in the states and $19,856 outside states.
Southcastern University
Southcastern University is a private institution founded in 1935. It has a total of 8,118 stu- dents (fall 2021). a suburban location, and a campus of 88 acres. It uses a semestcr-bused academic calendar. Southeast- em University ranks # 331--40) in the 2022-2023 2023 edition of the Top Colleges of the National Universities, with tuition and fees totaling 130,148.
Southeastern University is a private institution founded in 1935. It has a total of 8,118 students (fall 2021). its loca- tion is suburban, and the cam- pus covers 88 acres. It uses a semester-based academic cal- endar. Southeastern University ranks # 331 1440 among the top colleges in the 2022-2023 edi- tion of the National University Ranking. Tuition and fees are $30,148.
Southeastern University a private institution founded in 1935. It has a total of 8,118 students (fall 2021). its loca- tion is suburban, and the cam- pus covers 88 acres. It uses semester-based academic cal- cndar. Southeastern University ranks 219th among the top col- leges in the 2022-2023 edition of the National Universities, Its tuition and fees are $31.150.
University of Florida
The University of Florida is a public institution founded in 1853. It has a total of 34,881 enrolled students (fall 2021), its location is suburban, and the campus is 2,000 acres. It uses a semester-based aca- demic calendar. The Univer- sity of Florida ranks 29th in the rankings of the best colleges for the years 2022-2023. Tu- ition and fees in the states are $6,380; outside the state, tu- ition and fees are $28.658.
The University of Florida is public institution founded in 1853. It has a total of 34,881 students (fall 2021). its location is suburban, and the campus is 2,000 acres. It uses a semester- bused academic calendar. The University of Florida ranks 29th among the top colleges in the 2022-2023 edition. Tu- ition and fees in the states are $6,380: out-of-state tuition and fees are $28,658.
The University of Florida is a public institution founded in 1853. It has a total of 34,881 students (fall 2021), its location is suburban, and the campus covers 2,000 acres. It uses a semester-based aca- demic calendar. The University of Florida ranks 29th among the top colleges in the 2022- 2023 edition. Tuition and fees in the states are $6,381; out- of-state tuition and fees are $28.658.
Table 7: Sample expository documents generated by IRP on U.S. News.

---

Medicine (Topic)
True Output
IRP With Doc
IRP Without Doc
Pentazocine
Pentazocine is used to relieve moderate to severe pain. Pen- class of
Pentazocine is used to relieve moderate to severe pain. Pen- tazocine belongs to class of drugs called opioid (narcotic) painkillers. It works by chang- ing the way the brain and ner- vous system react pain.
Pentazocine is used relieve moderate to severe pain. Pen- tazocine is only used to treat se- vere pain. Pentazocine belongs a class of drugs called opioid (narcotic) painkillers. It works by blocking pain signals in the brain.
tazocine belongs drugs called opioid (narcotic) painkillers. It works by chang- ing the way the brain and ner- vous system react pain.
Thiothixene
Thiothixen is used to treat symp- toms of schizophrenia (a men- tal illness that causes disturbed unusual thinking. loss of in- terest in life, and strong or in- appropriate emotions). Thioth- ixen belongs to group of med- ications called conventional an- tipsychotics. It works by ducing abnormal arousal in the brain.
Thiothixene is used to treat symptoms of schizophrenia (a mental illness that causes dis- turbed unusual thinking. loss of interest in life and strong inappropriate emotions) in adults and children 12 years of age and older. Thiothixen class of medica- tions called conventional an- tipsychotics. It works by ducing abnormal arousal in the brain.
Thiothixene is used to treat symptoms of schizophrenia (a mental illness that causes dis- turbed thinking, loss of interest in life and strong inappropri- ate emotions). Thiothixene be- longs a class of drugs called conventional antipsychotics. It works by reducing abnormal arousal in the brain.
belongs
Diltiazem
Diltiazem is used to treat high blood pressure and control angina pectoris (chest pain). Dil- tiazem belongs to a class of med- ications called caleium channel blockers. It relaxes blood ves- sels so the heart does not have pump hard. It also increases blood and oxygen supply the hcart.
Diltiazem is used to treat high blood pressure and angina pec- toris (a condition in which the heart unable to pump enough blood to all parts of the body). class
Diltiazem is used to treat cer- tain types of heart rhythm dis- orders such as atrial tibrillation (a condition in which the heart beats irregularly, causing short- ness of breath, dizziness and fa- tigue). Diltiazem belongs to class of medications called cal- cium channel blockers. It works by relaxing blood vessels that blood can flow the heart.
Diltiazem belongs of medications called calcium channel blockers. It works by re- laxing blood vessels in the body and heart and lowering the heart rate.
Midostaurin
Midostaurin is used with other chemotherapy drugs to treat cer- tain types of acute mycloid leukemia (AML; type of can- cer of the white blood cells). Mi- dostaurin is also used for cer- tain types of mastocytosis (a blood disease in which there are too many mast cells [a certain type of white blood cells]). Mi- dostaurin belongs to class of drugs called kinase inhibitors. It works by blocking the action of the abnormal protein that sig- nals the proliferation of cancer cells, helping stop the spread mast and cancer cells.
Midostaurin is used with other chemotherapy drugs to treat cer- tain types of acute mycloid leukemia (AML: type of can- cer of the white blood cells). Mi- dostaurin is also used to treat certain types of mastocytosis (a blood disease in which there are too many mast cells certain type of white blood cells]). Mi- dostaurin belongs a class of drugs called kinase inhibitors. It blocks the action of the abnor- mal protein that signals cancer cells to multiply, helping to stop the spread of mast and cancer cells.
Midostaurin is used in combi- nation with other medicines to treat certain type of acute myeloid leukemia (AML; can- cer of the white blond cells) that has spread to other parts of the body or cannot he treated surgi- cally. Midostaurin belongs to class of drugs called kinase in- hibitors. It works by blocking the action of the abnormal pro- tein that signals cancer cells to multiply. helping stop slow the spread of cancer cells.
Table 8: Sample expository documents generated by IRP on Medline.

---

Wiki Title (Topic)
True Output
IRP With Doe
IRP Without Doc
Social Network Analysis
The analysis of social networks has its thcoretical roots in the work of early sociologists such Georg Simmel and Emile Durkheim, who wrote about the importance of studying the rela- tionship patterns that bind so- cial actors. Social scientists have been using the concept of "social networks" since the be- ginning of the 20th century describe complex relationships between members of social sys- tems at all levels, from interper- sonal to international.
Social network analysis has its theoretical roots in the work of early sociologists such as Georg Simmel and Emile Durkheim, who wrote about the impor- tance of studying the relation- ships that bind social actors and the need to organize social net- works to ensure social stability.
Social network analysis has its theoretical roots the work of early sociologists such as Georg Simmel and mile Durkheim, w ho wrote about the importance of investigating patterns of re- lationships that connect social actors. Social network analy- sis is not only a methodological upproach to the unalysis of rela- tionships or structures, hut also theoretical method of struc- tural analysis, which is widely used.
Cryptography
Before modern times, cryptog- raphy focused on message se- crecy (i.c.. encryption) con- verting messages from un- derstandable form comprchensible one and back again the other end, render- ing them unreadable to inter- ceptors eavesdroppers with- out secret knowledge (namely. the key needed to decrypt that message). Encryption sought ensure secrecy communi- cations, for example, by spies, military leaders, and diplomats. recent decades, the field has expanded beyond concerns about confidentiality, including techniques for verifying the in- legrity of messages, authentical- ing the identities of senders and recipients, digital signatures, in- teractive evidence, and secure calculations.
Essentially, before the early twentieth century, cryptography primarily concerned with linguistie patterss. Other cryp- tographic primitives include the encryption algorithms them- selves, one-way permutations. trapdoor permutations etc. Valu- able information
Essentially. before the early twentieth century, cryptogra- phy was primarily concerned with the study of linguistic patterns. Other cryptographic primitives include the encryp- tion algorithms themselves, one- way permutations, trapdoor per- mutation. etc. The security of many modern cryptographic techniques is based on the diffi- culty of certain computational problems, such as the integer factorization problem the dis- erete logarithm problem.
in-
be ex- tracted from the traffic, such as the exchange of session tokens. which allows the attacker ac- cess application accounts that the attacker should not be able access.
Business Process Modeling
Business process modeling tech- niques such as flowchart, func- tional flowblock chart, control flowchart, Gantt chart. PERT chart, and IDEF have been around since the beginning of the 20th century. Gantt charts were among the first ap- pear around 1899, flowchart the 1920s, functional flow- block chart, and PERT in the 1950s, data flowchart, and IDEF the 1970s. Modern meth- ods include Unified Modeling Language and Business Process Model and Notation, but they are only fraction of the meth- ods used over the years to doc- ument business processes. Williams coined the term "busi- ness process modeling" in his 1967 article "Business Process Modelling Improves Adminis- trative Control."
S. Williams coined the term "business process modeling" in the 1960s in his 1967 ar- ticle "Business Process Mod- elling Improves Administrative Control." These process mod- cls help organizations document workflows, surface key met- rics, pinpoint potential prob- lems and intelligently automate processes. Business process modeling became the founda- tion of new methods. for ex- ample those that supported data collection, data flow analysis. process flow diagrams and re- porting facilities. Business process modeling techniques have been developed over the decades to support specific busi- ness needs.
The term "business process modeling" itself was coined in the 1960s in the field of systems engineering by S. Williams in his 1967 article "Business Pro- cess Modelling Improves Ad- ministrative Control." His idea was that techniques for obtain- ing better understanding of physical control systems could be used in similar way for business processes. It was not until the 1990s that the term be- came popular.
Table 9: Sample expository documents generated by IRP on WikiCS.

---

In following survey, you will read generated college descriptions the style U.S. News. Please rate each document on scale of 1-5 in Style Adherence, and Factual Accuracy. Please use the following guidelines for these two attributes using one sentence from the example of the fictitious "Moon University."
Style Adherence: How similar is the generated text compared to the output? Do they organize information concerned whether factual information correct, but rather factual information being described/outined appropriately.
order, generally using the same phrasing?
Example True Qutput:
Moon University public institution that was founded in 2022.
Example Ratings:
1 Moon University has total enrollment of 50 students.
3 Moon Universily was founded in 2022, is a public university, and enrolls students.
5 Moon University is a public institution that was founded in 2005.
Factual Accuracy: How accurate is the information conveyed in the document? Are there significant factual inconsistencies errors? We will provide the ground truth output along with the generated document. Please Google verify factual errors they not obvious.
Example True Output:
Moon University public institution that was founded in 2022. has a total enrollment of 50,
Example Ratings:
1 Earth University private institution that was founded in 1990. has 10,000 students.
3 Moon University is public institution founded in 2000. has enroliment of 75.
5 Moon University, founded 2022, public institution.
students are enrolled.
Figure 6: Evaluation instructions expository text generation on U.S. News.

---

Model
With Doc
Without Doc
True Output
Southeastern University is private institution founded in 1935. It has total of 8,118 students (fall 2021), suburban location, and a campus of 88 acres. It uses semester-based academic cal- endar. Southcastern University ranks 331 1440 in the 2022-2023 edition of the Top Colleges of National Universities, with tuition and fees totaling S30,148.
Southeastern University is private institution founded in 1935. It has a total of 8,118 students (fall 2021), a suburban location, and campus of 88 acres. It uses semester-based academic cal- endar. Southeastern University ranks 331-440 in the 2022-200 202-2003 edition of the Top Colleges the National Universities, with tuition and fees totaling $30,148.
IRP (Ours)
Southeastern University is private institution founded in 1935. It has total of 8,118 students (fall 2021), its location is suburban, and the campus covers 88 acres. uses semester-baet -baseerr---ddddeee academic calendar. Southeastern University ranks 331-440 among the top colleges in the 2022-2023 edition of the National University Ranking. Tuition and fees are 148.
Southeastem University is private institution founded in 1935. It has a total of 8,118 students (fall 2021), its location is suburban, and the campus cov- 88 acres. It uses semester-based academic cal- endar. Southeastern University ranks 219th among the top colleges in the 2022-2023 edition of the Na- tional Universities. Its tuition and fees are $31,150.
LLaMa
Southeastern University is private institution founded in 1935. It has a total of 2,500 students (fall 2021). its location is urban, and the campus is 100 acres in size. It uses semester-basee academie calendar. Southeastem University ranks 14th in the 2022-2023 edition of the best colleges. Its tuition and fees $22,900.
Southeastern University is private institution founded in 1935. It has a total of 2,200 students (fall 2021), its location is urban, and the campus is 100 acres in size. It uses calendar. Southeastern University ranks 14th in the 2022-2022 edition of the best colleges. Its tuition and fees are $22,900.
eemetter-bssed academic
LLaMa+Retr
Southeastern University is private institution founded in 1935. It has a total of 1,800 students (fall 2021), its location is suburban, and the campus is 100 acres in size. It uses semester-based aca- demic calendar. Southeastern University ranks 12th in the 2022-2023 edition of the best colleges. Its tuition and fecs are $21,000.
Southeastern University is a private institution founded in 1935. It has a total of 1,900 students (fall 2021), its location is suburban, and the campus is 100 acres in size. It uses semester-based aca- demic calendar. Southeastern University ranks 12th in the 2022-2023 edition of the best colleges. Its tuition and fees are $21,000.
LED
Founded in 1935, Nova Southeastern University is a private institution with a total of 8,118 students (fall 2021). a suburb of Lakeland, and a campus size of 330 acres. It uses semester-baat baseed academic calen- dar. Nova Southeastern University ranks #331 1-440 among the top colleges in the 2022-2023 edition of the National Universities. Its tuition and fees are 140,148.
Founded in 1935, Nova Southeastern University is a private institution with total of 8,118 students (fall 2021). suburban location, and a campus size of 88 acres. It uses a semester-based academic cal- endar: Nova Southeastern University ranks 234th among the top colleges in the 2022-2023 edition of the National Universitics. Its tuition and fees are $30,148.
RAG Southeastern University is private institution founded in 1935. It has a total of 8,118 students (fall 2021), is located in the city, and the campus covers 250 acres. It uses a cameetter-bsee academic calendar. Southeastern University ranks #331-440 among the top colleges in the 2022-2023 edition of the National Universities. Tuition and fees are 832,834.
Southeastem University is a private institution founded in 1935. It has a total of 8,118 students (fall 2021), is located in the city, and the campus covers 250 acres. It uses semester-basee academic calendar. Southeastern University ranks #331 -440 among the top colleges in the 2022-2023 edition of the National Universities. Tuition and fees are $82,180.
BART Southeastern University is private institution founded in 1891. It has a total of 2,856 students (fall 2021), its location suburban, and the campus covers 110 acres. It uses remester-based academic calendar. Southeastern University ranks #331-440 among the top colleges in the 2022-2023 edition of the National Universities. Its tuition and fees are $39,834.
Southeastem University is a private institution founded in 1891. It has a total of 2,856 students (fall 2021). its location is suburban, and the campus covers 110 acres. It uses semester-based academic calendar. Southeastern University ranks #331 -440 among the top colleges in the 2022-2023 edition of the National Universities. Its tuition and fees are $39,834.
Table 10: Baseline comparison of generated expository documents on U.S. News for Southeastern University. Red text indicates factual errors.

---

Model
With Doc
Without Doc
True Output
Daridorexant belongs to a class of drugs called orexin receptor antagonists. It works by blocking the action of a certain natural substance in the brain that causes alertness.
Daridorexant helongs to a class of drugs called orexin receptor antagonists. It works by blocking the action of certain natural substance in the brain that causes alertness.
IRP (Ours)
Daridorexant used treat symptoms of pulmonary embolism (PF; cancer in which the lungs are dam- aged) in people who have not been treated with other medications. Daridorexant helongs to a class of drugs called orexin receptor antagonists. It works by blocking the action of a certain natural substance in the brain that causes alertness.
Duridorexant belongs to class medications called dual orexin receptor antagonists (DORAs). blocks the binding of the wake- wake--ro neuropeptide orexin and can turn off overactive wakefulness.
LLaMa
Daridorexant used to treat insomnia (difficulty falling asleep or staying asleep). works by block- ing the action of histamine, a substance in the body that can cause drowsiness.
Daridorexant used treat insomnia (difficulty falling asleep or staying asleep). works by block- ing action of histamine, a substance in the body that can cause drowsiness.
LeaMa+Retr
Daridorexant is drug which acts antagonist of orexin receptors. It used for the treatment of insomnia.
Daridorexant is used treat insomnia (difficulty falling asleep or staying asleep). It works by block- ing the action of histamine, substance in the body that can cause drowsiness.
LED
Daridorexant is used to treat insomnia (insomnia; condition in which disturbed sleep is difficult to control and can lead to difficulty sleeping and poor daytime functioning). Daridorexant belongs to class of drugs called orexin receptor antagonists. It works by reducing the amount of gleep-promoting substances in the brain. Daridorexant does not cause insomnia improve daytime functioning.
Daridorexant is used to treat insomnia (insomnia; sleep disorder in which sleep is disturbed and dif- ficult control) in adults and children 12 years of age and older. Daridorexant belongs to a class of drugs called orexin receptor antagonists. It works by slowing activity in the brain to enable sleep.
RAG
Daridorexant is used alone or in combination with other medications to treat insomnia (insomnia or disturbed sleep) in adults and children 6 years of age and older. It works by slowing activity in the brain to enable sleep.
Daridorexant is used alone or in combination with other medications to treat insomnia (insomnia or disturbed sleep). It is also used to treat restless leg syndrome (RLS; restless legs syndrome; condition in which the legs do not function properly and the person can experience weakness, numbness, loss of muscle coordination and problems falling usleep) in adults and children 6 years of age and older who have not been successfully treated with benzo- diazepines (narcotic) or sedatives (sudden, unex- pected bouts of insomnia that can occur when the person is not treated with these medications).
BART
Daridorexant is used alone or in combination with
Daridorexant is used alone or in combination with other medications to treat high blood pressure. It is also used to treat agina pectoris (chest pain that occurs when the heart does not pump enough blood to all parts of the body) in adults and children 2 years of age and older. Daridorezant belongs to class of medications called angiotensin II receptor antagonists. It blocks the action of certain natu- ral substances that tighten blood vessels, allowing blood to flow more smoothly through the hody.
other medications to treat high blood pressure. It is also used to treat agina pectoris (chest pain that occurs when the heart does not pump enough blood to all parts of the body) in adults and children 2 years of age and older. Daridorezant belongs to class of medications called angiotensin II receptor antagonists. It blocks the action of certain natu- ral substances that tighten blood vessels, allowing blood to flow more smoothly through the body.
Table 11:
Baseline comparison of generated expository documents on Medline for Daridorexant, Red text indicates factual errors. Blue text indicates repetitive or difficult to understand phrasing.

---

Model
With Doc
Without Doc
True Output
Business process modeling techniques such flowchart, functional flowblock chart, control flowchart, Gantt chart, PERT chart, and IDEF have been around since the heginning of the 20th century. Gantt charts were among the first to appear around 1899, flowchart in the 1920s, functional flowblock chart, and PERT the 1950s, data flowehart, and IDEF the 1970s. Modern methods include Unified Modeling Language and Business Process Model and Notation, but they are only a fraction of the methods used over the years to document business processes. S. Williams coined the term "business process mod- eling" in the 1960s in his 1967 article "Business Pro- cess Modelling Improves Administrative Control."
Business process modeling techniques such flowchart, functional flowblock chart, control flowchart, Gantt chart. PERT chart, and IDEF have heen around since the beginning of the 20th century. Gantt charts were among the first appear around 1899, flowchart the 1920s, functional flowblock chart, and PERT in the 1950s, data flowehart, and IDEF in the 1970s. Modem methods include Unified Modeling Language and Business Process Model and Notation, but they are only fraction of the methods used over the years to document business processes, Williams coined the term "business process mod- eling" in the 1960s in his 1967 article "Business Pro- cess Modelling Improves Administrative Control."
IRP (Ours)
S. Williams coined the term "business process mod- cling" in the 1960s in his 1967 article "Business Pro- cess Modelling Improves Administrative Control.". ". These process models help organizations document workflows, surface key metrics, pinpoint potential problems and intelligently automate processes. Busi- ness process modeling hecame the foundation new methods, for example those that supported data col- lection, data flow analysis. process flow diagrams and reporting facilities. Business process modeling techniques have been developed over the decades to support specific business needs.
The term "business process modeling" itself was coined in the 1960s the field of systems engineer- ing by S. Williams in his 1967 article "Business Pro- cess Modelling Improves Administrative Control."" ". His idca was that techniques for obtaining better understanding of physical control systems could be used in similar way for business processes, was not until the 1990s that the term became popular.
LED
The term "business process modeling" coined by the systems engineer S. Williams in 1967.
Fractal patterns have been found in the paintings of American artist Jackson Pollock. Pollock's 's paintings were composed chaotic dripping and splattering, and Pollock's 's paintings were composed of chaotic dripping and splattering.
RAG
A business process model is graphical representa- tion of business process or workflow and its related sub processes. A process model can grow out of pro- cedural maps of day to day operations.
Business Process Management (BPM) was orig- inally developed in the 1980s by group of re- searchers at the University of Mlinoii at Urbana- Champaign under the name of Business Process Man- agement. Their goal was "build a business process model that would allow companies identify and manage their business processes in a way that would reduce costs and increase productivity.
BART
The concept of business process management dates back to the 1960s, with the introduction of the con- cept of "business process management" in the United States. Business process management was originally developed in the 1970s by the University of Cal- ifornia, Berkeley, and the National Institute of Standards and Technology (NIST).
The concept of business process management dates back to the 1960s, with the introduction of the con- cepl of "business process management" in the United States. Business process management was originally developed in the 1970s by the University of Cal- ifornia, Berkeley, and the National Institute of Standards and Technology (NIST).
Table 12:
Baseline comparison of generated expository documents on WikiCS for Business Process Modeling. Red text indicates factual errors. While LED is factually correct, it does not provide enough detail on the with doc dataset and fails to discuss the topic of Business Process Modeling adequately on the without doc dataset.


Reviews:
[{"review_id": "JMNhs791fK", "reviewer": "Reviewer_7sKR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We appreciate that the reviewer finds our experimental evaluation to be thorough, notes that our method has advantages in interpretability and factuality, and believes our datasets could be useful for future research.\n\n### RE: Error Propagation\n\nWe appreciate the reviewer’s concern regarding error propagation in our model. We agree that the three-step nature of our approach could leave some room for error propagation. However, we investigated this weakness of error propagation in Section 5.4 and did not find it significant. Specifically, we “find that many factual errors are due to alternatives existing in the retrieved facts, rather than the weaknesses of the Retriever, Paraphraser or data collection” (lines 573-576). \n\nWe also acknowledge this shortcoming in the limitation section. We decided not to train an end-to-end model because it made our approach more interpretable and thus easier to detect errors in each individual component (lines 613-616). Further, we believe the fact that IRP uses three separate components instead of an end-to-end framework suggests that there is still room for improvement and research in expository text generation (lines 611-619).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The evaluation seems thorough, and the technique proposed seems to have advantages for both interpretability and factuality. The datasets could be useful for future research.", "reasons_to_reject": "The three-step nature of the proposed architecture could result in error propagation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about an iterative three-step technique for expository text generation. It introduces the task and offers a framework and datasets. It also presents a thorough experimental comparison on automatic metrics and has a human evaluation component.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJyv634ZM3", "reviewer": "Reviewer_yjk6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and we appreciate your support of our work! We are delighted to hear that the reviewer found our design choices and composition of pretrained models to be very interesting and smart. Further, it's encouraging to hear that our techniques for generating content plans and retrieving facts were perceived to have applicability in other text generation tasks, which we also feel would be very interesting future work. We are grateful for the time and effort you've taken to thoroughly review our paper and provide insightful feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While the IRP method uses pretrained models (e.g. DistilBERT, GPT-2, etc), it composes them in a very interesting way in order to produce a framework for expository document generation.\n\nAlong the way, some very smart choices are made. For example, the authors propose an interesting method for generating content plans, by optimizing DistilBERT to predict the index of a sentence in a document (which thereby lowers token attribution for facts which vary widely between documents).\n\nExpository text generation is a practical task, and the methods proposed here for structuring content plans and retrieving facts can be applied to many other types of text generation as well.", "reasons_to_reject": "I can't see any. it's a really nice paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel method for generating more factually accurate and stylistically appropriate expository documents. It's called \"Imitate, Retrieve, Paraphrase\" -- which summarizes the steps:\n1. Sketch a template (\"content plan\") for the next sentence based on stylistic imitation of documents on similar topics\n2. Retrieve facts from a knowledge source to fill in factual statistics/statements\n3. Paraphrase the retrieved facts to match the style of the content plan.\n\nThe main contributions are:\n- Proposes the IRP framework as a general method for generating expository documents\n- Publishes three datasets to facilitate further research on factuality and style in text generation\n- Shows via experiments that IRP produces more factual and stylistically appropriate expository documents than competing models", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6y76OZGvOr", "reviewer": "Reviewer_Q3e3", "summary": "", "questions": "A) Do I understand it correctly in 3.3 that the training is done using the ground-truth document, i.e. the generated content plan + the facts -> gold sentence? If this is indeed the case, this could be made a bit more explicit. I guess the confusion came because the implicit assumption is that the facts in x need to align with the gold document already. Update: this is explained in 3.5 but should probably be referenced in 3.3 in some way.\n\nB) In the \"with doc\" scenario are only back-translated versions of the document included, i.e. is the exact original text excluded?\n\nC) Do you think larger models would be in any way beneficial for your approach? (I don't mean to imply that this should have been tested.)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review of our paper! We appreciate that the reviewer finds our work to be crucially important for the field of aligning modern LLMs with human expectations, believes that our experimental setup and evaluation are careful and thorough, and feels that our analysis of factual errors was particularly interesting.\n\n### RE: Applicability to Other Domains\n\nWe appreciate the reviewer’s concern regarding the applicability of our method. For the scope of this work, we focus on the task of expository text generation. We would first like to note that expository text generation alone has several diverse applications other than the ones we studied, such as generating biographies, synthesizing travel guides, or creating product descriptions. Further, we believe that IRP could be adapted for other NLP tasks, such as answering questions in a uniform style in long-form question answering or generating stylistically consistent summaries in multi-document summarization. We feel all of these tasks and applications could become meaningful research directions. We can add a discussion of these applications in the final version.\n\n### Question A\n\nYes, we train the paraphraser in section 3.3 to use the generated content plan and retrieved facts as inputs to generate the gold sentence as output. Thank you for noting our explanation in section 3.5. We agree with your suggestion to make section 3.3 clearer and can incorporate more details into section 3.3 in the final version.\n\n### Question B\n\nIn the 'with doc' scenario, we **do not include** the exact gold output as part of the input factual corpus. \n\n### Question C\n\nWe did not explore if larger components would be beneficial for our model. We believe that this may be the case, so in future work, it would be interesting to explore if these incremental quantitative benefits justify the increase in model size.\n\n### RE: Presentation Improvements\n\nWe agree with all of these points! In particular, we will be sure to modify line 83 so that it is more similar to how our content plan (i.e. Imitator) actually functions. Thank you again for taking the time to provide such thorough and insightful comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The field of guided generation is a path towards aligning modern LLMs with human expectations and therefore crucially important to the field as a whole.\n\nThe experiments seem to be conducted very carefully with a specific focus on not poisoning the retrieval dataset with the ground truth. The evaluation is conducted using a wide range of metrics on top of that even a human evaluation is performed which I would consider unusually thorough. The presented approach performs very well.\n\nThe analysis of factual errors was particularly interesting and something I wish I would see in more publications.", "reasons_to_reject": "Since the dataset is newly introduced the quality of the approach is hard to assess quantitatively, but a good range of comparisons are provided. The IRP approaches applicability to other domains is not explored.", "questions_to_authors": "A) Do I understand it correctly in 3.3 that the training is done using the ground-truth document, i.e. the generated content plan + the facts -> gold sentence? If this is indeed the case, this could be made a bit more explicit. I guess the confusion came because the implicit assumption is that the facts in x need to align with the gold document already. Update: this is explained in 3.5 but should probably be referenced in 3.3 in some way.\n\nB) In the \"with doc\" scenario are only back-translated versions of the document included, i.e. is the exact original text excluded?\n\nC) Do you think larger models would be in any way beneficial for your approach? (I don't mean to imply that this should have been tested.)", "paper_topic_and_main_contributions": "The paper introduces an architecture to generate stylistically consistent expository texts in a guided manner. The authors introduce a imitate retrieve paraphrase setup to guide the generation. Using a variety of methods (both n-gram overlap based and factualness-based ones) they evaluate their methodology against gold standard documents in  addition to using human evaluation. They test against a wide variety of other approaches, including 7B Llama models for the automated evaluation and ChatGPT for the human evaluation. Ultimatly there model outperforms the comparison in most scenarios.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "83: I found the quote to be confusing, after all this is not what your content plan would look like in practice (it would be some non-factual sentence with a specific ranking unless I misunderstand).\n\neq 1 and 5: maybe be more explicit about this just being a Language MOdeling loss?\n\n338: \"ensembled\" is a misleading choice of word in my opinion (due to the ensemble model implication), perhaps \"combined\"?"}}]

Author rebuttal:

Meta-review: This paper proposes the task of expository text generation and its evaluation dataset. Furthermore, the authors also propose the method named "Imitate, Retrieve, Paraphrase" (IRL) for generating more factually accurate and stylistically appropriate expository documents. This three-step approach has advantages for both interpretability and factuality in generating text. The experimental results on the newly created dataset by the authors show that IRP produces more factual and stylistically appropriate expository documents than competing models. The authors answered all questions by the reviewers, and all reviewers agreed with the acceptance of this paper. Thus, we should follow their agreement.
Decision: Accept-Main