Interactive Text Generation
Authors: Felix Faltings
Michel Galley
Kianté Brantley
Baolin Peng
Weixin Cai
Yizhe Zhang
Jianfeng Gao
Bill Dolan
Venue: EMNLP 2023 Main (2023)
Keywords: Interactivity
Text Generation
RL
IL
TL;DR: An interactive framework for training and evaluating text generation systems

Abstract: Users interact with text, image, code, or other editors on a daily basis. However, machine learning models are rarely trained in the settings that reflect the interactivity between users and their editor. This is understandable as training AI models with real users is not only slow and costly, but what these models learn may be specific to user interface design choices. Unfortunately, this means most of the research on text, code, and image generation has focused on non-interactive settings, whereby the model is expected to get everything right without accounting for any input from a user who may be willing to help.

We introduce a new Interactive Text Generation task that allows training generation models interactively without the costs of involving real users, by using user simulators that provide edits that guide the model towards a given target text. We train our interactive models using Imitation Learning, and our experiments against competitive non-interactive generation models show that models trained interactively are superior to their non-interactive counterparts, even when all models are given the same budget of user inputs or edits.

Interactive Text Generation
Felix Faltings
Michel Galley²
Kianté Brantley
Baolin Peng2 Bill Dolan
Weixin Cai Yizhe Zhang"
Jianfeng Gao
'MIT "Microsoft
CConnell University
Abstract
Text generation (e.g., keywords-to-text
NASA
N4SA 1s
Users interact with text, image, code, or other editors on daily basis. However, machine learning models are rarely trained in the set- tings that reflect the interactivity between users and their editor. This is understandable as train- ing models with real users is not only slow and costly, but what these models learn may be specific to user interface design choices. Un- fortunately, this means most of the research on text, code, and image generation has focused on non-inteen ion-interactive settings, whereby the model is expected to get everything right without ac- counting for any input from user who may be willing to help. We introduce new Interactive Text Generation task that allows training gener- ation models interactively without the costs of involving real users, by using user simulators that provide edits that guide the model towards a given target text. We train our interactive models using Imitation Learning, and our ex- periments against competitive non-interactive generation models show that models trained in- teractively are superior to their non-interactive counterparts, even when all models are given the same budget of user inputs or edits.
launching spacecraft
NIG model
Taunching a spacecraft to Mars 1n
final text
2070
user Input
Interactive Text generation (ITG)
NASA
NASA
MASA ix launchi spacecraft Monday
NUS
launching
NLS
2006-
user ingut
user Input
user Ingut
Figure I: The ITG framework of this paper allows gen- eration models to be trained and evaluated by directly interacting with users user simulators.
satisfy user needs without additional information. Distinguishing the real user need from the need as initially presented to the system has been the focus of decades of research (Taylor, 1968), and usually requires interacting with users to clarify their needs (Gao et al., 2022). Unfortunately, much of the research in generative models have been in "one-shot" settings, which don't allow any kind of iterative refinements to steer the generation towards what the user really wants.
1 Introduction
This paper introduces a new end-to-end genera- tion task, Interactive Text Generation, that accounts for interactivity without requiring real users during training. The framework is illustrated in Figure 1, where the model and a "user" take turns editing the text until a given stopping criterion is met. As this setup would be impractical to train with real users. we rely on a user simulator that provides a few high-quality edits that guide the model towards a given target text. This interspersion of user edits in the generation process allows text generation mod- cls to more efficiently use inferential capabilities of large language models (LLM). Contrast interactive text generation in Figure with conventional text generation, where both systems are given exactly three user input words. Interactive text generation leverages LLM's capability to infer, e.g., "space-
Advances in generative modeling have made it pos- sible to automatically generate high-quality texts (Brown et al., 2020), code (Chen et al., 2021), and images (Ramesh et al., 2021), but these creations can be unsatisfactory in many respects. For ex- ample, they often suffer from content errors erg. hallucinations (Ji al., 2022) may require help from the user. Even the generation is of good quality, may not be the kind of text, code, or image that the user was hoping to obtain. Indeed, open-ended and complex generation tasks are of- ten underspecified (c.g., from a simple prompt), which makes it almost impossible for the model to
Corresponding authors: faltings@mit.cdo {mgalley, jfgao| @microooft.comm Yizhe Zhang is currently at Apple.

---

craft" from "NASA" and allows it to focus on parts of the text (c.g., "Monday") that are more difficult to predict, and this allows the interactive approach to generate text that better meets user's needs.
generation models that can do most of the heavy lifting, this task gives more directional role to the user, while the bulk of the text is generated by the system. Therefore, our task is similar to the instructor-executor frameworks (Hu et al., 2019; Kiseleva et al., 2022) seen in other tasks. In the case of text generation, motivational examples of such instructor-execute interactions include a stu- dent writing a paper while getting occasional feed- back from an advisor, or freelance writer getting instructions from a client.
Our work makes the following contributions: We propose a task and framework for interactive text generation, and release models, dataset, and user simulators. Crucially, this task evaluates generation models with the same budget of user inputs or edits, to ensure the comparison between interactive and non-interactive models is fair. We present methods to train Transformer-basee based (Vaswani et al., 2017) interactive text editing models using imitation learning, where our mod- els learn to imitate expert that dynamically constructs a trajectory from the current docu- ment state to the goal state (target document). Our editing models include both autoregressive and non-autoreeressivve versions, with the non- autoregressive one achieving the best results. We show that interactivity indeed does help thanks to imitation learning when compared to their non-interactive counterparts, across differ- ent evaluation metries, model types, and model sizes and with the same amount of user inputs. This finding is consistent with prior work show- ing that user needs are often best handled interac- tively (Oddy, 1977; Belkin et al., 1995; Radlin- ski and Craswell, 2017; Gao et al., 2022), and confirms that our benchmark helps quantify the benefit of interactivity in text generation.
Our task models the interaction between a user and system, where the two parties successively take turns making changes to a draft document. As this interaction applies to both training and testing, it would be unrealistic to assume we have real users in both cases, and we therefore rely on user simu- lators. Although building effective user simulators is notoriously hard in tasks such as dialog (Schatz- mann et al., 2007: Li et al., 2016; Lin et al., 2022: Gao et al., 2019), it is less so in our work given how we frame the interactive generation task: We assume that there is goal text which one can think of as representing the user's s desire or needs. The system does not know the goal, but the user simu- lator does. The objective of the agent is to get close to the goal text as possible, while minimiz- ing the number of edits the user simulator needs to make.
This framing makes it much easier to design ef- fective user simulators, as a comparison between the current draft and goal text can help infer useful text edits. While agents in this setup are given an advantage by interacting with an oracle that has knowledge of the goal document, the number of oracle edits is generally set to be small, and we ensure comparisons between all models (includ- ing non-interactive models) are fair by giving each model the same budget of edits or inputs derived from the goal document.
As user simulation is a key component of this new task, contribute different user simula- tors. Our experiments show performance of our models remains consistent with different user simulators, which highlights the robustness of our new benchmark.
Another contribution is that text generation in this work is not merely aimed at producing well-formed text, but also at creating text that is tied to user needs. We release this framework with the hope it will help researchers in NLP, Imitation Learning, Reinforcement Learning, and AI in general as provides an environment to train AI agents that directly interact with users and user simulators.'
2.1 Task Formulation
We can formalize our task by the following proto- col. There two players in the game, the agent and user. Starting from a blank document, the play- ers take turns producing drafts over H rounds.? 2 The user has goal document G, and the objective of the game to get as close to the goal as possi- ble. Closeness is quantified by scoring function
2 Task: Interactive Text Generation
The task introduced in this paper considers sim- plc setting in which the system and user collabo- ratively write a document. As our goal is to train
EExcept for index ranges, e.g. t = 1,..,, T, we will use capital letters to denote random variables, and lower case letters to denote realizations.
Code and models for this paper will be made public.

---

Architecture
User simulator
Edit Model, either:
treggenerated document Ag
goal G
Taken Edit Model
(Repeat N)
Editing Model
Citrm
plaeer
Input draft
Oracle Model
choose a position
document Uhea, with
Insert
substhute
dekde
choose an op.
Oracle Model
Sonoc
Obamna
Inesih
predieet
ouiput draft
Align goal
Ag
Farock
Dr
famonk
Obama
ASB
pr preaiiint
Seq2Seq Transformer Model
Suruch
Oburma
44th
Input draft
Ani
Benuch
pbyer
Produce
Un
forock
Obaraa
44th
presifinee
output draft
Figure 2: Model architecture: the current draft repeatedly goes through two phases. First, it is edited with the edit model (either a con-autoregressive token edit model or standard seq2seq Transformer,, and then is annotated with edits coming from the user simulator. While the user simulator has access to the gold document, all our generation systems (interactive and non-interactive) are evaluated with the same budget of user inputs.
(t; G), and a tolerance 8 > O. We denote the agent draft at step h by As, and the user draft at step by Un- If we then fix goal document G, the protocol is follows. For h I..., H:
Where appropriate, we indicate references to the appendix in parentheses.
3.1 Edits
Both our simulated user described below, and our model described in Section 4, operate by making edits to a document. We will consider single-word edits of three types: inserting a word, deleting a word, substituting one word for another. If we denote edit by e, and document by z, then (x:) will denote the action of document x. producing new document y, Note that y will differ from by a single word. If have multiple single-word edits C1, CN, we can apply them one after another, as in - (e))..
1. User observes A4-1. If s ((An11: G) >1 5, stop. Otherwise, the user produces Un, accord- ing to G and Ap-1.
2. Agent observes Un and produces An-
Here Ag is blank document. The tolerance represents how close the agent needs to get to the target for the user to be satisfied. Let T H be the time at which the interaction stopped. We evaluate the task by looking at s(Ar; G) and T. The higher s( (Ar; G), the better the produced document, and the lower T, the more time the user saved.
3.2 Simulated User
Because the user has knowledge of the goal docu- ment G, we easily define sensible user simu- lator. Given a current draft produced by the agent Aд, we can compute sequences of edits e1, such that G = [et..en] as detailed in Sec- tion 3.3 below. The user will produce a draft U/41 = (e1...c](AAp) only by applying some small number of these edits < N, where the edits are chosen according to heuristic (App. B). An example heuristic would be to pick edits E1, that insert informative words first. This way, the user provides information the agent by produc- ing a draft Un i1 that is a few steps closer than A₁. This is like providing a gradient the direction G. Note that while this represents one
We assumed here that each instance of the task is parameterized by goal document G. However, could instead use multiple goal documents, Gi,.., G even replace s(; G') with some gen- eral score function s(-) that is independent of any goal document, which one could think of utility function for the user.
In this work, we assume that the user behavior is fixed and we will learn an agent policy.
3 User Simulator
This section describes the user simulator used in our work. Throughout this section and the next, we refer the reader to the Appendix for more details.

---

Expert policy Re provides trajectory from Sg to
tory of drafts (Ao, ........ (An-1, UR) exchanged so far. This information is necessary for the agent to make inferences about the goal However, this history can become very long. especially since each element is an entire document. Therefore, in practice we only give the policy Ta access to the last set of drafts. (A which we represent as diff (App. B), as illustrated in Figure 2.
$
current document
empty document
%
S2
goal document G
Markov Decision Pracess:
State space:
4.1
Left to Right Autoregressive Model
histories: ((Ag.UI.... U..
Current state encodes:
Action space:
History Ag U2.44 Uz.4z. 33
The first agent model that we consider is simple sequence to sequence (S2S) model that generates outputs in left-to-right autoregressive manner,
new drafts: (4)
Ag. Ag produced by agent
Ug U2. Uy produced by user
Figure 3: Interactive text generation as imitation Iearn- ing: The editing model (agent policy TT) is trained to imitate the expert policy T-
H
To (An =a|Sh) II
no Te(ailaci Sh),
where (a1,..., 01) and aci (a1, a4-1). Note that this model autoregressively generates the new draft from scratch.
way to build a user simulator, our task formulation is completely agnostic to the user behavior.
In our experiments we ensure that comparisons between non-interactive and interactive systems are fair by fixing the total number of single-word edits made by the user at N. In the non-interactive case, all N edits are made at the start. In the interactive setting, the N edits are spread out over M rounds of editing, where at each round the user only makes N/M edits.
4.2 Token Editing Model
While our S2S model forms reasonable base- line, we note that not particularly adapted for the task. Instead, we also propose a token editing model that directly edits the previous draft Un to produce its revised draft A,, by making a series of edits. This way, both the agent and user operate by making edits. Additionally, action to allow the model decide when to stop editing. Concretely, we parameterize the model a distribution over edits (and the stopping action), Te( (-[[(e (....ee]](((,,,) S,)). In other words, based on the previous edits made, and the current state, the model decides what edit to make next. The probability of producing particular draft is,
add stopping
3.3 Alignments
For any V.Y there may be many sequences el,,,,,, eN such that y [e)).. There is always such sequence since we can delete com- pletely and generate y. Practically, we can compute a set of such sequences of edits c1,...,NN cN by find- ing an alignment between x and y. as illustrated in Figure 2. The alignment defines which words in map to words in y, and which words were deleted inserted. Words that match to identical words are conserved, whereas mismatches correspond to substitutions. From there it is easy to get the edits C1,-.., Cy that transform 2 into y. We will denote this set by A(x, y). We use contextual word em- beddings from BERT to compute these alignments, so that words in & map to words in with similar meaning, as we found that this tends to produce more natural alignments (App. A.3).
M
Σ oo(soopl a, S..) II no co(elleet...e (Uh),
where the sum is over all sequences of edits T (c1, (.1,..., CM) such that (U) a. See Fig- ure for illustration.
4.3 Training
We train our model to copy an expert policy To that would perform well in our task. Because this ex- pert is only used training time, it is very simple: produces the goal G. See Figure for an illus- tration. We follow the DAgger (Ross et al., 201 1) algorithm as outlined Algorithm 1, which tries to minimize the following objective:
4 Imitation Learning Model
The agent policy ny will parameterize a distribution over documents, so that at each step of the interac- tion we draw A ~ S.).. Here Sh is the his-
L(A) = Eg-nEs,- [- log no (G( (G)Ss))

---

Algorithm 1: DAgger
upper bound (App. A.5),
Initialize mp; Initialize D <- 0; 6-
M
Bo,
Σ oki1
log To (e Sn))
repeat
π < STÆ + (1 (1- mg
where the two expectations are over uniform dis- tributions, and ny is the number of terms in the sum. The first expectation ranges over prefixes of edit sequences ef. ..... ex for SM, and the sum ranges of over distinct edits c 1- In words, this objective says that we select random number of edits, in random order Ø, that move us toward G, giving us the intermediate draft X We then try to predict the next edit that brings us closer to G. We evaluate this objective stochastically as described in A Algorithm (App. A.5).
Sample goal documents 91..... 9C ~ Vi
Sample trajectories from π with contexts .1, 90;;
Collect B states from sampled
trajectories into D;
Aggregate D + DU D;
Train policy Te on D; {- AB;
until convergence;
4.5 Decoding
where и the roll-in, or sampling policy, is a distribution over goals, and the second expectation is over histories produced by running our task pro- tocol with the policy π as the agent (App. A.4). Setting to Te would allow us to train on-policy, but since Ty will start off poor policy that visits many unnecessary states, DAgger uses mixture between the agent and expert policies, so ı f + (1 B) where [0, 1] is annealed to during training.
When decoding from the autoregressive model, we use standard algorithms such as beam search. For the token editing model, we sequentially sample edits from the model until the model's probability of stopping reaches given threshold or we exceed maximum number of decoding steps. If we hit the timeout, return the draft that had the high- est stopping probability according the model. This ensures that whatever draft the agent returns is highly confident that is a finished document. For example, if the model were editing "the man" into "the man and the dog". we would not want to stop and return the draft "the man the dog" where the model hasn't 't yet added the word "and". Algo- rithm 4 outlines this procedure (App. A.7).
4.4 Likelihood Estimation
The training objective from Section 4.3 requires the computation of the negative log-likelihood - llog re (G) For the autoregressive model, this can be directly computed as simple product de- scribed in Section 4. On the other hand, the like- lihood for the token editing model involves sum over sequences of edits which quickly becomes in- tractable. Instead, we optimize an upper bound on it. Using the alignments defined previously, we can specify a set of edit trajectories, denoted A(U,, G), SM, where
Experiments
Because of the novelty of our task, the main goal of our experiments is to assess the benefit of interac- tivity in text generation. We also provide example interactions between the learned agent and user simulator. Ablations and additional details on ex- periment settings are in the appendix.
that are indexed by permutations M is the length of the edit trajectories (App. A.3). So, an edit trajectory in A(U,, G) will be written G for all We can then assume that most alternative trajectories will have low probability, so we restrict the sum to all trajectories A(Uh, G). For convenience, we ..., (U,). This gives the will write X = [eq -ef,..,
5.1 Setup
......, e and leq ..., e5r]
Data We consider single sentences from CNN/DailyMail article summaries (Nallapati et al., 2016). While only consider single sentences for ease of implementation, we can extend our models full documents in straightforward way. Using news articles gives complex and factually grounded sentences, while restricting our attention to the summaries keeps the sentences
Forr example the infinite number of trajectories that in- volve inserting and deleting the same word.

---

0.55
0.55
0.55
0.55
0.04
EdRto
0.035
0.5
0.5
0.5
0.5
BARTScore
0.03
BLEURT
ROUGE
chrF
BLEU
0.45
0.025
0.02
0.4
0.4
0.015
0.35
0.35
0.35
0.35
0.01
of episodes
of episodes
# of episodes
of episodes
of episodes
Figure 4: Results showing the benefit of interactive generation, where #cpisodes=1 means the entire budget of 6 oracle edits is given the model all at (i.e., no interactivity). For feepisoes==2, the model receives imput 3 oracle edits per episode (2x3=6). For #episodes=3, the model receives 2 oracle edits per episode (3x2=6).
self-contained. This dataset forms the distribution over goal documents G.
6 Results
Automatic evaluation Figure 4 presents our main results on interactivity. We can see that for our main model, splitting the same number of user ed- its over more episodes leads to better scores across BLEU, BLEURT and BERTSCORE. For example, comparing the setting where the model receives all user edits at the start in episode, against the setting where the edits are given across episodes. we see improvements of about 7%, 4% and 5% (absolute gains) terms of BLEU, BLEURT, and BERTSCORE respectively. While these differ- ences are not large in absolute terms, we emphasize that this gain comes exclusively from interactiv- ity. The amount of information given to the model is the same in both settings. This suggests that, even in this narrow sense, interactivity helps. Note that there may also be many more benefits from human-computer interaction standpoint that we cannot quantify here.
Model We implement our models using pre- trained transformer models, with additional MLP heads to predict the edit actions in the case of the token editing model.
Metrics We evaluated generation using BLEU (Papineni al., 2002), CHRF (Popovié, 2015) BLEURT (Sellam al., 2020), BARTSCORE (Yuan et al., 2021), and ROUGE (Lin, 2004). As BARTSCORE returns a score interpretable as a log- probability, we report the natural exponent of that score. In the case of both BLEU and ROUGE, we evaluate with their unigram versions (BI.FU- I and ROUGE-1 1) as they provide interpretability of the results at the token level.
5.2 Interactivity
To probe whether interactivity helps, we compared the performance of our model given different levels of interactivity. Concretely, we take a fixed number of edits N provided by the user, and compare the performance of the model when those edits are pro- vided over varying number of episodes M. Thus, for given M, the user and agent interact over M episodes, with the user making N/M edits each episode. Note that the total amount of information provided by the user in each setting is thus the same. The only difference is that in interactive settings the agent is able to make changes in between receiving edits from the user. While this setup is not able to probe the ad advantages that interactivity provides still
We also note that our token editing model (Ed- itor) outperforms the left-to-right, autoregressive sequence to sequence (S2S) model While the difference is not staggering. it is notable given the success of standard S2S models across variety of text generation tasks. As motivated in Section 4, the token editing model more naturally suited to the task, and we argue that it constitutes the starting point for more interesting interactive mod- els. For example, one could foresee training the model using reinforcement learning, in which case
'In the case the S2S model, we added word reward feature (He et al., 2016), which is sometimes called "length penalty". We tuned this feature on a validation set of 500 instances. We added this feature due to the observation that S2S outputs in our task are often too short, which has been noted in other tasks such as machine translation (Murray and Chiang, 2018). This feature was only necessary with the $2S model, and resulted in the three models of Figure 4 having average lengths close to one another (within 6%)
in terms of human-computer interaction, expect to see better performance in the interactive case. For example, the model may be able to pre- empt some the edits the user makes, allowing the user to use its budget to make other edits.

---

Non-inte interactive
Interactive ++
Model
BLEU BLEURT BART
++
Bart Editor (baseline)
0.76
0.70
0.14
Mcaning 6.7%
30.2%
9.6%
42.6%
10.9%
Fluency 8.0%
36.0%
15.1%
33.4%
7.5%
noise level
0.0
0.74
0.69
0.14
0.1
0.78
0.73
0.16
Table 1: Human evaluation using Editor Large model, comparing interactive generation (3 episodes, 2 oracle edits) vs. non-interactive (1 episode, 6 oracle edits). Results indicate judges' preference: ++ (definitely supe- rior), + (somewhat superior), (equal quality).
0.2
0.73
0.67
0.12
oracle contiguous
0.71
0.66
0.12
adjacent
0.60
0.61
0.09
adj+config
0.66
0.64
0.11
Sampling annealing rate 0.85
the structure of the editing model based on editing actions better suited than a S2S model.
0.76
0.70
0.14
0.80
0.71
0.67
0.12
conducted human eval- uation (Table 1) to compare interactive and non- interactive generation, similar to the automatic eval- uation in Figure 4. We selected the best pest-performing model (BART Large) and compared 3-episode gen- eration against episode 1k test instances. Each instance was evaluated by 5 mechanical turkers. The judges were asked compare each pair of gen- erations based on semantic similarity to the gold text ("meaning") and linguistic quality ("fluency") using a 5-point Likert scale. Table 1 shows that interactivity improves performance, as the interac- tive system is generally semantically closer to the target (significance at p < le 7). The interactive system exhibits slightly lower fluency average, although the level of significance is weaker here (p = .037). We hypothesize this slight decrease in fluency is due to multiple rounds of generation.
Human evaluation
Table Ablation results with different training hy- perparameters. All ablations results are relative to the baseline. BART stands for BARTSCORE.
Model
BLEU
BLEURT
BART
adj+config (baseline)
0.76
0.70
0.14
contiguous adjacent
0.76
0.69
0.14
0.78
0.75
0.19
unrestricted
0.78
0.75
0.18
Table 3: Ablation results with different oracles changed at test time.
sampling annealing rate indicates how quickly we anneal from the expert to the trained model while sampling (lower faster). Table compares differ- ent user heuristics at fest time.
Ablations We provide extensive ablations of model variants. As a benchmark for comparison, we look at the quality the text produced by our models after interacting with the user over several episodes. For better comparisons we use a fixed number of episodes and fixed number of user edits per episode. We use 3 edits and 4 episodes. Tables 2 and 3 present our ablations. Note that these results use 4 episodes, with user hints per episode total of 12 user hints) compared to the 6 total user hints in Figure 4, so the overall results are higher.
We note that adding noise during training im- proves results (c.g. noise level 0.1 vs. 0.0). whilc annealing too fast can hurt performance (annealing rate 0.8 vs. baseline). Interestingly, training with user that better matches the inference-time user leads to worse performance (e.g. adj+contig baseline). It seems that using the most informative user (which simply returns the most informative edits, without restriction) leads to the best model (baseline). Comparing different user simulators at inference time, we see that adding restrictions to the user leads to decreased scores, as expected. Interestingly, we see that the most impactful re- striction seems to be enforcing contiguous edits. We suspect that this is because contiguous edits are highly predictable. For example, predicting "Obama" after "Barack" fairly obvious. Thus, if the user didn't 't provide contiguous edits, and only inserted the word "Barack", the model could score an easy win by predicting "Obama".
The baseline model is trained with noise level of a 0.3, an unrestricted user and sampling annealing rate of 0.9. All models in Table 2 were evaluated with the adjacent and contiguous user heuristics. Table 2 presents variations on train- ing parameters. The noise level the amount of noise injected during training, the user the (solc) heuristic used for the user during training, and the

---

Examples Table 4 provides two examples of in- teractions between the agent and user. We empha- size that these examples were not cherrypicked. Note how the agent is able to revise its previous version of the text based on the new information provided by the user at each step. Qualitatively, these interactions could be much more natural for a user than the one shot setting that is prevalent in the literature. However, systematic evaluation of this claim requires more comprehensive user study that lies out of the scope of this work.
hand, various works (Zhang et al., 2019; Du al.. 2022b) have studied the types of revisions made by humans. On the other hand, works have focused on modeling text edits (Guu et al., 2018; Yin et al., 2018; Faltings et al., 2021; Akoury et al., 2020), but they have generally been restricted to model- ing single episode of user edits at time. In our framework, model edits and user edits are inter- leaved. We note that (Du et al., 2022a) presented an interactive revision system, but their model was nevertheless trained on corpus of edits, rather than in an interactive environment as in our case. The recent versions of GPT-3 (Brown et al., 2020; Ouyang et al.. 2022) and CharGPT also offer text editing capabilities. For example, ChatGPT can handle prompt containing a piece of text and in- struction to improve the text, and ChatGPT's ability to execute that command often quite impressive. The ability of ChatGPT to interact with users was somewhat explored in (Bang et al., 2023), although not in the context of text editing. We think our work is complementary to GPT-3 and ChatGPT, as we provide framework for both modeling and evaluating edit models in more end-to-end setting. Our model of interaction between user and system, where the user and system both editing the text, may also be more natural than dialogue. Ultimately, we think will be beneficial fine-tune very large language models such as GPT-3 an environment that exposes them to interaction with a user or a user simulator (i.e., akin to user-in-the-low train- ing). This benefit is currently somewhat captured using reinforcement learning (RL) to tune large language models from human feedback (Ouyang et al., 2022), except that our approach features actual environment representative of the end-user experience while Ouyang et al. (2022) is more akin to offline RL.
7 Related Work
Text Generation Prior work on natural language generation (NLG) has largely focused non- interactive settings that have become increasingly more challenging and open-ended, e.g.. with gen- eration from prompts (Fan et al., 2019), out- lines (Rashkin et al.. 2020), topics or keywords (Ghazvininejad et al., 2016; Yan, 2016), plots (Riedl and Young, 2010), descriptions (Jain et al., 2017), events (Martin et al., 2017). This increase of difficulty can make NLG more prone to content quality issues, such as hallucinations (Wiseman al., 2017: Filippova, 2020; Çelikyilmaz al., 2020; et al., 2022), that can require post-editing from the user. Several works explored ways for LLMs to improve their outputs by iteration and self-critiquing (Huang et al., 2022; Gou et al., 2023; Welleck et al., 2022). In particular, Welleck et al. (2022) presented models for text generation and self-correction that also incorporated external feed- back. However, in their case the feedback is used training time learn corrector. In our task, the user feedback comes at inference time and the agent must use that feedback to guess what the user would like to generate.
Non Autoregressive Generation Several works considered non-autoregressive text generation (Gu et al.. 2019; Shen et al., 2020; Xu and Carpuat, 2021; Stern et al., 2019; Zhang et al., 2020: Welleck et al., 2019), but these models all focus on one-shot text generation. While some models are able to edit text (Gu al., 2019; Xu and Carpuat, 2021), it is primarily used as a means to refine the model's generations. On the other hand, we consider editing text into a completely different version conditioned on a given set of user-provided edits.
Interactivity Several works have explored inter- activity between humans and models to complete tasks collaboratively. Lee et al. (2022) presented dataset to reveal large language models' capa- bilities in assisting creative writing. Zhang et al. (2022); Li et al. (2022) explored pre-trained lan- guage models for code editing based on human- written comments. Closer to our work, Lahiri et al. (2022) created an interactive framework to refine user intents through test case generations and user feedback, and Kiseleva et al. (2022) studied an interactive agent that can interact with humans
Text Editing Text editing has previously been considered from two different angles. On the one
hhttps://chat openai. www.co///aaaoccccco..coo

---

Example
Example 2
target:
Progress 59 spacecraft will re-enter Earth's atmosphere in a week, Russia space agency says.
target: The Large Hadron Collider (LHC) begins again after two-year shutdown.
oracle: Progress 59 re-enter
oracle: (LHC)
agent: Progress in Europe means he has 59 per cent chance to re-enter the EU.
agent: Curators at the Mars Space Telescope (LHC) will launch in space the next month.
oracle: Progress in Europe means he has 59 per cent spacecraft will re-enter the EU.
oracle: The Large the Mars Space Telescope (LHC) will launch in space the next month
agent: Progress in will help the Europe meuns agent: he -has - 59 million -er -eent spacecraft will re-enter the Earth EU
The Large X-ray Telescope at the Mars Space Teleseope (LHC) is located wiill kuench in Sppee the Atlantic.. neex month
oracle: Progress will help the 59 million space- oracle: The Large X-ray Telescope (LHC) is lo- cated in two-year shutdown.
craft will re-enter Earth's says.
agent: Progress comes after Boeing will help agent: the 59 spacecraft will re-enter NASA Earth's 's spacecraft, it says.
The Large Mars Telescope-Hay Teleseope (LHC) will end two-year shutdown.
oracle: Progress comes after Boeing 59 spacecraft oracle: The Large Hadron Collider (LHC) will end a two-year shutdown.
will re-enter NASA's spacecraft, Russia space agency says
agent: Progress comes as Boeing's after Beeing agent: 59 spacecraft will re-enter Earth -NSAA spacecraft, Russia's space agency says
The Large Hadron Collider (LHC) will end a -nnd two-year shutdown.
Table 4: Example interactions between the oracle and agent. The oracle changes are shown red. Deleted words are crossed out. Note that these are randomly selected examples (i.e., without cherrypicking), with the only curation being the rejection of potentially offensive examples.
and follow natural language instructions to achieve goals in Minecraft. Finally, interactivity has also been studied from Human- Human-Computer Interaction viewpoint (Clark et al., 2018; Shen and Wu, 2023).
Limitations
A long-term goal this research is to enable in- teractive editing of full documents. For practical reasons and to facilitate adoption, we limited text length to 64 tokens, but we plan to extend our benchmark and released datasets to also include paragraph-level and multi-paragraph texts. Another limitation of our work is that training is done with simulated instead of real users, as training with users the loop can be extremely slow and costly. To make our approximation of user behavior realis- tic, our work relies on user inputs that worldw users perform routinely (i.e., word insertion, dele- tions, and substitutions) even settings that are not assisted with Al. However, we recognize that the behavior of real users in a Al-user collaborative setting may differ from that of our user simulators, and leave the study such behavior for future work. Finally, while the need for interactivity ap-
8 Conclusions
We presented a new task and benchmark for text generation that operationalizes interactivity be- tween an agent and user, without the need to involve real users during training. Our framework compares interactive and non-interactive systems that are given the same amount of user inputs, and shows that interactive text generation leads to text of higher quality according to multiple automated evaluation metrics and human evaluations. We also present a non-autoregressive editing model that out- performs a standard sequence-to-sed to-seueecce Trans- former in various settings. All baseline, data, and models of this paper will be made public.

---

plies to any kind of Al creation (e.g., also code and images), we leave the application other gen- eration tasks for future work. We note, however, that this paper treats text as simple sequences of symbols, and our work could readily be applied to other symbolic creations (e.g., code generation).
References
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng. and Mohit Tyyer. 2020. Storium: A dataset and evaluation platform for Machine-in-tth--ae story generation. arXiv preprint arXiv:2010.0171..
Yejin Bang. Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint vrXiv:2302.0402.
Ethics Statement
Text generation systems, including those relying on large language models, the risk of generat- ing text that is unsafe (Bender al., 2021; Bom- masani et al., 2021; Weidinger et al., 2021), and this also applies to generation models developed in this work. While we have not observed genera- tions that are overtly toxic or hateful, our models can generate texts that are biased and offensive to some readers. As our work focuses on generation of non-fictional texts (in contrast to prior work on story generation), our models also run the risk of generating text that is factually incorrect. However, the focus of our research to provide interactive capabilities to generation systems, and to make them more in control of the user. As illustrated between system and
Nicholas J. Belkin, Colleen Cool, Adelheit Stein, and UIrich Thiel. 1995. Cases, scripts, and information- seeking strategies: On the design of interactive in- formation retrieval systems. Expert Systems With Applications, 9:379-395.
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus. Stefano Ermon, John Etchemendy, Kawin Etha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lau- ren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti. Geoff Keeling, Fereshte Khani, O. Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi. Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Pa- padimitriou, Joon Sung Park. Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer. Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie. Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. ArXiv, abs/2 108.0725..
in Figure 1, a user can make the generated text more factual, and the same kind of interaction can also help increase its safety. Such sanitization of generated texts in our framework may still expose users with unsafe content, so it is still recommended to use current safeguards against hallucinations (Ji et al., 2022), biases (Dhamala et al., 2021; Weinberg. 2022), and other offensive content (Gehman et al., 2020; Welbi et al., 2021; Kiritchenko et al., 2021: Jones et al.. 2022; Xu et al., 2022) before displaying text to real users. In that sense, think our work is comple- mentary to current NL. research on hallucination and safety.
Acknowledgments
We thank Regina Barzilay, Faeze Brahman, Chris Brockett, Si-Qing Chen, Javier González Hernán- dez, Gerold Hintz, Qiuyuan Huang, Tommi Jaakkola, Zhang Li, Lars Liden, Kosh Narayanan, Hoifung Poon, Victor Quach, Chris Quirk, Sudha Rao, Chenglong Wang, Bin Yu, Zhou Yu, well members of the NLP and Deep Learning groups at Microsoft Research for helpful discussions. This work was partly supported by the Eric and Wendy Schmidt Center at the Broad Institute.

---

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Firie Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are shw-shot learners, arXiv.
Katja Filiippova. 2020.
Controlled hallucinations: Learning to generate faithfully from noisy data. In Findings of EMNLP.
William Gale. Kenneth Ward Church, al. 1994. A program for aligning sentences in bilingual corpora. Computational linguistics, 19(1):75-102.
Jianfeng Gao, Michel Galley, and Lihong 2019. Neu- ral approaches to conversational AI. Foundations and Trends in Information Retrieval, 13(2-3):127-298.
Jianfeng Gao, Chenyan Xiong, Paul Bennett, and Nick Craswell. 2022. Neural approaches to conversational information retrieval. arXiv.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Vossb,rt-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob MeGrew, Dario Amodei, Sam McCandlish, Nya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained on code. arXiv.
Samuel Gehman, Suchin Gururangan. Maarten Sap. Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompte Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356 3369.
Marjan Ghazvininejad, Xing Shi, Yejin Choi, and Kevin Knight. 2016. Generating topical poetry. In EMNLP.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen. 2023, Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305. 11738.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Lev- enshtein transformer. Advances in Neural Informa- tion Processing Systems, 32.
Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, pages 329-340.
Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018 Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
J. Dhamala, Tony Sun, Varun Kumar, Satyapriya Kr- ishna, Yada Pruksachatkun, Kai-Wei Chang, and Rabul Gupta. 2021. BOLD: Dataset and metrics for measuring biases in open-ended language genera- tion. Proceedings of the 2021 ACM Conference on Fairness, Accountabiliry, and Transparency.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. 2015. Contextual markov decision processes. arXiv preprint
Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with smt features. Proceedings of the AAAI Conference Artificial Intelligence, 30(1).
Wanyu Du, 7ae Myung Kim, Vipul Raheja, Dhruv Kurnar, and Dongyeop Kang. 2022a. Read, re- vise, repeat: A system demonstration for human- in-the-loop iterative text revision. arXiv preprint arXiv:2204.036855
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. 2019. Hierarchical decision making by generating and following natural language instructions. In NewrlPS.
Wanyu Du Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022b. Understanding iterative revision from human-written text. ArXiv, abs/2203.03802.
Jiaxin Huang. Shixiang Shane Gu. Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.1 166100
Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. 2021. Text editing by command, In NAACL-HLT.
Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mo- hak Sukhwani, Anirhan Laha, and Karthik Sankara- narayanan. 2017. Story generation from quence of independent short descriptions. ArXiv. abs/1707.05500
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In ACI.

---

Ziwei Ji. Nayeon Lee. Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hal- lucination in natural language generation. ArXiv, abs/2202.03629.
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al. 2022. CodeReviewer: Pre-training for automating code re- view activities. arXiv preprint arXiv:2203.09093
Keenan Jones, Enes Altuncu, Virginia N. L.. Franqueira, Yichao Wang, and Shujun Li. 2022. A comprehen- sive survey of natural language generation advances from the perspective of digital deception. arXiv.
Chin-Yew Lin. 2004, Rouge: A package for automatic
evaluation of summaries. In Text summarization branches out, pages 74-81.
Hsien-Chin Lin, Christian Geishauser, Shutong Feng, Nurul Lubis, Carel van Nickerk, Michael Heck, and Milica Gašié, 2022. GenTUS: Simulating behaviour and language in task-oriented dia- logues with generative transformers. arXiv preprint arXiv:2208. 10817.
Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C. Fraser. 2021. Confronting abusive language online: A survey from the ethical and human rights perspec- tive. ArXiv, abs/2012. 12305..
Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail S. Burtsev, Alexey Skrynnik, Artem Zholus, Alek- sandr I. Panov, Kavya Srinet, Arthur D. Szlam, Yuxuan Sun, Katja Hofmann, Michel Galley, and Ahmed Hassan Awadallah. 2022. Interactive grounded language understanding in a collaborative environment: IGLU 2021. In NewrlPS 2021 Com- petitions and Demonstrations Track, pages 146-161. PMLR.
Lara J. Martin, Prithviraj Ammanabrolu, William Han- cock. S. Singh, Brent Harrison, and Mark O. Riedl. 2017. Event representations for automated story gen- eration with deep neural nets. In AAAI.
Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceed- ings of the Third Conference on Machine Translation: Research Papers, pages 212-223, Brussels, Belgium.
Shuvendu K. Lahiri, Aaditya Naik, Georgios Sakkas, Piali Choudhury, Curtis von Veh, Madanlal Musu- vathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization.
Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning, pages 280-290.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018 Deterministic non-autoregressive neural se- quence modeling by iterative refinement. arXiv.
Saul B Needleman and Christian D Wunsch. 1970. A general method applicable to the search for simi- larities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443-453.
Mina Lee, Percy Liang, and Qian Yang. 2022. CoAu- thor: Designing human-ai collaborative writing dataset for exploring language model capabilities. CHI.
Robert N. Oddy. 1977. Information retrieval through man-machine dialogue. Journal of Documentation, 33:1-14.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gun- jan Chhablani. Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishinn, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman. Jacob Hilton, Fraser Kelton. Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU; a method for automatic eval- vation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 311-318.
Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016. A user simulator for completonn dialoges.. arXiv preprint arXiv: 1612:061...5688.
Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation. pages 392-395.

---

Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, & ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversational search. Proceedings of the 2017 Conference on Conference Human Informa- tion Interaction and Retrieval.
Laura Weidinger, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese. Borja Balle, Atoosa Kasirzadeh. Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hen- dricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. 2021. Ethical and social risks of harm from language models. ArXiv.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Nyy Sutskever. 2021. Zero-shot text-to-image gener- ation. arXiv.
Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline- conditioned generation with dynamic plot state track- ing. In P Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 4274-4295, Online.
Lindsay Weinberg. 2022.
Rethinking fairness: An in- terdisciplinary survey of critiques of hegemonic ML fairness approaches. Ariif. Intell. Res., 74:75-109.
Johannes Welbi, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John F. J. Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Cop- pin, and Po-Sen Huang. 2021. Challenges in detoxi- fying language models. ArXiv, 0as/2109.07444
Mark O Riedl and Robert Michael Young. 2010. Nar- rative planning: Balancing plot and character. In Journal of Artificial Intelligence Research.
Sean Welleck, Kianté Brantley, Hal Daumé Tii, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. In International Conference on Ma- chine Learning, pages 67 17-667. PMLR.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. 1. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proc. of AISTATS, pages 627 635.663517 PMLR.
Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. ArXiv, abs/2211.00053.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Yc, and Steve Young. 2007. Agenda-basse user simulation for bootstrapping a pomdp dialogue sys- tem. In Human Language Technologies 2007: The Conference of the North American Chapter of the As- sociation for Computational Linguistics; Companion Volume, Short Papers, pages 149-152.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253 2263,226632222 Copenhagen, Denmark.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing: System Demonstrations, pages 38 45, Online.
Hua Shen and Tongshuang Wu. 2023. Parachute: Evaluating interactive human-In Im co-writing systems. arXiv preprint arXiv:2303.06333.
T. Shen, Victor Quach, R. Barzilay, and T. Jaakkola. 2020. Blank language models. In EMNLP.
Temple Smith, Michael Waterman, et al. 1981, Iden- tification of common molecular subsequences. Jour- nal of molecular biology, 147(1)195-197,
Canwen Xu, Zexue He, Zhankui He, and Julian McAuley. 2022. Leashing the inner demons: Self- detoxification for language models. In AAAL.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible se- quence generation via insertion operations. In In- ternational Conference Machine Learning, pages 5976-5985. PMLR.
Weijia Xu and Marine Carpuat. 2021. Editor: an edit- based transformer with repositioning for neural ma- chine translation with soft lexical constraints. Trans- actions of the Association for Computational Linguis- tics, 9:311-328.
Richard S. Sutton and Andrew G. Barto. 1998. Intro- duction to Reinforcement Learning. The MIT Press.
Rui Yan. 2016. i, poet: Automatic poetry composi- tion through recurrent neural networks with iterative polishing schema. In IJCAL.
Robert S. Taylor. 1968. Question-negotiation and infor- mation seeking. Coll. Rex. Libr.

---

Pengcheng Yin, Graham Neubig, Miltiadis Allama- nis. Mare Brockschmidt, and Alexander Gaunt. 2018. Learning represent edits. arXiv preprint arXiv:1810.13337
A Methods Details
This section gives more formal and precise defini- tions of the notions of document, edits and align- ments introduced in the main text. We follow by more detailed description of the token edit- ing model and the derivation of the lig-likelihood lower bound.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScorr: Evaluating generated text as text gener- ation. ArXiv, abs/2106. 1.520.
Jiyang Zhang, Sheena Panthaplackel. Pengyu Nie, Junyi Jessy Li, and Milos Gligoric. 2022. CoditT5: Pretraining for source code and natural language edit- ing. arXiv preprint arXiv:2208.0544
A.1 Documents
We consider documents to be elements V the space of strings of length L formed by words in vocabulary V. We assume V contains blank to- ken so that VL corresponds to all documents of length up to L.
Xuchao Zhang, Dheeraj Rajagopal, Michael Gamon, Sujay Kumar Jauhar, and ChangTien Tien Lu. 2019. Mod- eling the relationship between user comments and edits in document revision. In Proceedings the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNL)), pages 5002 5011.
A.2 Edits
Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and Bill Dolan. 2020. Pointer: Con- strained text generation via insertion-basee genera- tive pre-training. ArXiv, abs/2005.000553
In this work we consider three types of singlc-word edits: insertions, deletions, and substitutions. In particular, we do not consider word movements, i.e. changing the position of a word in docu- ment. This will simplify the alignments we use to define the user simulator behavior. We for- mally define an edit as 3-tuple specifying: a location E [L] 1...., L, an operation o (ins, del, sub]. and word w E V. An edit e == (I, o, w) E [Z] x fins, del, sub} x V can then be applied to a document x E VL, which we de- note by [x), as defined by the following rules:
Asli Çelikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey, ArXiv, abs/2006.14799
If o ins: [e](x) F1...Ty-11 W
2. If o = del: [z](z) = x1....) 1 1...T
3. o sub: [c] = FI... T4 1807 1...0
When performing multiple edits C1, eN in sequence, will write [el...e]((a) [<x]()-[e)](==) shorthand. Note that the edits are not permutation invariant because of the loca- tion parameter For example, x is blank docu- ment, and e and e2 each correspond to inserting "the" and "dog" respectively at the start of the docu- ment, i.e. = (1, ins, the) and (1, ins, dog). then [e2e1](x) = "the dog" whereas [2))) "dog the".
"A more operator-like way to write this would have been [ew...r)) ). but we opt for the other order, where the interpreta- tion is that the brackets map the sequence of edits et,..., ex to an operator [e]...ex] which is equivalent to performing the edits in sequence.

---

A.3 Alignments
We use the cosine similarity between the BERT em- beddings of z3 and yi as their score. We found that using BERT embeddings gave more natural align- ments where words of similar meaning or function are matched together. For example, consider two single word documents "red" and "blue". Then it makes more sense to consider the change from "red" to "blue" substitution rather than dele- tion followed by insertion. On the other hand, "red" to "car" makes less sense as a substitution. This score can be easily optimized using dynamic programming (see for example Gale et al. (1994); Smith et al. (1981): Needleman and Wunsch (1970) for more details).
Given two documents v, y. an alignment will give us a convenient way to find a sequence of edits el,..., eN such that [e)...e]((x) = y. We use this to define the user simulator behavior, as well as for training our token edit model. Loosely speaking, alignment between and y is partial mapping between words in x and words in y,. More for- mally, it is an undirected hipartite graph with ver- tex sets V2 = [01,...,?] FL} and Vn = (v1...., }r and edges E. The edges then match words in x to words in y, where some words in x or may not be matched. A monotonic alignment is one where the edges in the graph do not cross.
Given alignment and y. the set of indices A = (i: के * ūe) naturally give set of edits. We can split A I D UUSSS into the union of indices corresponding respectively to insertions, deletions, and substitutions. These edits can be performed in any order, so the alignment gives a whole set of edit sequences, one for each permutation of A. For any E SM a permutation of A, where M = |A] is the size of A, there is sequence of corresponding edits eq. ......: e.-- The delicate part about getting these edits is determining their location parameters since these will depend on the order o. We will need the following two quantities:
For our purposes it will be convenient to repre- sent alignments in a partieular way. Rather than graph, an alignment between and y will be a pair E, 3/ E V2L, where æ, resp. y, a subsequence of x. resp y. Moreover, z and y only contain blank to- kens otherwise. See Figure 2 for an example. Then each pair (Z,, W:). i = 1,.., 2L is interpreted as an operation:
1. insertion if i; = and y: +
2. deletion if j = and i; +
3. substitution if + U: + F. +-
Pairs of blanks (._)) are ignored. Note that the the positions where neither 2; or are blank give monotonic alignment. The converse is not true because could construct many pairs (x, y) that correspond to the same alignment. This is because we can rearrange the order of the positions cor- responding to insertions and deletions since this won't it affect the alignment. We therefore require all insertions to come before deletions so that mono- tonic alignments between and y correspond one- to-one to pairs (z, y).
1. 1. = K6 <i: AnG) < Ao():: E ETH
2. D = Kij < i:: Art) < Aoll), Aoti) E DH
These correspond to the number of insertions and deletions that come before edit eg and which will affect its location parameter. Let B Kj Ao(i) σj = = be the number of blanks before position Ao(a) in x. Then, for i = 1,.... M, the location parameter will be 1 = Aa(s) BO D + L8. Note that this is just keeping track of where the edit needs to be made in the document [ef,... .... e,,,,(())))) after applying the first i - 1 edits.
Given x and y, there may be many possible align- ments. We therefore define a score on alignments and choose an alignment that maximizes the score. Given alignment (z, g). the score is
We then treat the three operations separately:
1. Aa(s) E I: e, = (1i, ins, TAoco)
2L
S(ā 7)
Σ s (T. To 12, v),
2. Ar(1) D: og del,
3. Ao(1) E S: ee = sub, (A,())
where s(Zr, Do |, v) scores the pair (Ti, Øt) given the contexts & and y. If both x; and y are blank, the score is O. If only one is blank, we assign baseline score b. If neither is blank, then o; will correspond to some n in x. Similarly Ti corresponds to yr in
In summary, for documents v, y we can compute a unique alignment (α,, 7) We then denote the set of edit sequences {eā,..., eMr o E SM} } as defined above by A(x, v)-

---

Algorithm 2: Token Edit Model Objective
where the sum is over all sequences of edits e1, EM such that [e]...er][(U,) a. We first lower bound the likelihood by discarding sequences outside of the set A(Un. a), so
Data: a, 8h
Compute edits A(un, a.):
Unifornly sample permutation a E SM:
Uniformly sample k E {1,2,... M + 1}; Compute oh = [c?,... (un)( return M+1 Eoz log No ( 1- 8h);
To TT(a(Sp) N
Σ OESM
(sttopp S.)) II
To mo(egIxE-1.
A.4 Markov Decision Process
where Xg- = [c....%-- This is the same type of sum as in (Shen al., 2020). Following their derivation,
As stated in Section 2, fix the user behavior and learn a policy for the agent. To do so, we model our task as a finite-horizon Contextual Markov Deci- sion Process (Sutton and Barto, 1998; Hallak et al., 2015) A,C, P. R, H, v) where the state space, AA the action space, C is the context space, P: S x A C (SS) is the transition func- tion, R S Ax [0, x)) is state-action- context dependent reward function that models the user's utility, H is the horizon, and p A(S) and E A(C) are the initial distribution over the state space and distribution over contexts. The action and context spaces correspond to VL, the sequences of length L over vocabulary V, where the context is the goal document G. The state space is a history of drafts produced so far. Because the agent is ignorant of G, having access to the his- tory of drafts allows the agent make inferences about the goal G. The reward models a tradeoff between minimizing T, the time it takes get to a satisfactory document, and s (Ar; G). the quality the produced document. The user is modeled by the transition function P. Given previous state (i.e. history of drafts) Sn-1, the agent's draft A₁-1, and the target G, the environment transitions to state [Sn-1l(AR-1, U.)), where denotes concatenation, and Un is the user's draft produced according to AІ-1 and G. This framework allows to use tools like imitation learning to train a policy Tg for this MDP, as done in Section 4.3.
log TO (a(aS))) N
M Il k=1
log Σ
Te(stopla, Sh))
OESM
M
C+ M!
log To ( X 1, SN):
0ESM
k=1
where = lolooo())) +log Te Sp). The last line comes from Jensen's 's incquality, where the sum over permutations became an expectation. Using the same trick of rearranging the sum over time steps and orderings as in Shen et al. (2020), we can rewrite the second term as,
Σ
log No ool
MI
oCSMr
M
Eq Eg
log TO volcg112211 Sp))
M k
where the sum is over the set {e og e, some a E SrI. We then fold the log @e(stop) a, S₁.) term into the expectation over k. This gives the objective,
A.5 Token Edit Model Likelihood Lower Bound
M +
Σ log no (e Sta)
E E&E.
ExEoo
Recall that at step h, the likelihood under the token editing model model of a document given the current history Sr. and latest draft Un will be,
T&k
where the expectation now over 1, ..., M M+ cM = stop for all a E SM, and
(A(,, = =a(Sp)=
M no To(eulle...ep-1(U.) St),
Σ To (stopla, Sh) II k=1
M M i M
74k

---

Algorithm 3: Noisy Edit Trajectory Sam-
Algorithm 4: Token Edit Model Sampling
pling
Data: 8, u, o,
Data: a, Ub, 0
Result: Generation fro ma{-|s)
Result: Edits , EM
20 f u;
I 4 uhi
8o + 0:
k 1; 1;
< 0:
repeat
repeat
Compute A(x, a):
8; <- t= 8);
Sample e from A(r, 1)
Sample non-stopping edit ej from a0(-|
< Bernoulliarr;
ifp p 1 then Ej - random edit;
ri+1 [e:](z;);
i i until > or N;
else
C/ < c;
f- argmax
end
return Øj
x f (x);
- k + 1;
until x = a;
A.7 Decoding
return e1,
To decode from the token editing model we use ancestral sampling with few modifications. First, when sampling an edit (or stopping action) from To. we only sample from the top k edits. Just as for autoregressive models, we found that this improves generation quality, since the nww-probabiligg edits are usually of poor quality and will figuratively speaking throw wrench in the decoding process. For long sequences of edits, the probability of sam- pling a bad edit also becomes non-negligible
A.6 Denoising
In practice, the lower bound derived above may be only considering edits
very loose because in AU a.). In other words, the learned policy Ty will inevitably find itself out of distribution at inference time by making mistakes.
In order to make the model robust, we leverage denoising training (Lee et al., 2018). To do so, we notice that in Algorithm 2 we are essentially sampling trajectory of edits eI...eM such that (U),) = a, and then randomly sampling point along this trajectory. Instead, we sample noisy trajectories of edits, where intersperse random edits, as in Algorithm 3. This simulates mistakes that the trained policy might make at in- ference time. Given this random prefix of edits e1..., we get a noised, intermediate docu- ment, - We compute the same loss as before over the set of edits {e: : = e, E A(āx-1,a) a)}. This is the set of first elements of edit trajectories from - to a, and we'll llll denote it by A( a)1.
The model also has a tendency to stop early. Again, this is because the probability of stopping erroneously increases as we sample more and more edits. In contrast other types of mistakes, stop- ping early is especially bad because there is no way recover, opposed to other mistakes that can be fixed with another edit later on. Therefore, we explicitly avoid sampling the stopping action, and instcad decode edits until either the model's stopping probability exceeds a threshold, or we reach a maximum number of edits. We then return the document that had the highest stopping proba- bility. The whole decoding procedure is given in Algorithm 4.
The objective becomes,
B Experiment Details
B.1 Data
M ++1
Σ eEA(0x-1.0)
log (e(e(Tx-1.
We version 3.0.0 of the CNN/DailyMail dataset from the HuggingFace dataset hub (Lhoest et al., 2021). We take the article summaries, which we split into sentences and then filter to sentences with less than 64 tokens, where the tokens are deter- mined by the BART tokenizer from HuggingFace
where now n4 = |A(# 1. 1. Intuitively, this is like using noisy roll-in policy to get to žg-1, and then matching an expert policy that can produce a.

---

(Wolf al., 2020). We then split the data into train, test, and validation splits, with (approximately) 1M, 55K, and 45K instances respectively.
cases, If 1, then
To(e|2,8) (o(e),,))=<((()) = ()(1),,)))
B.2 User Simulator Heuristics
If s = 0, = del, then
The environment of our contextual MDP is deter- mined by the behavior of the user simulator, We consider the following methods for generating user edits:
To(e|x,s) = (/0
Otherwise,
Ranking Given set edits, we rank them based on their informativeness. As a measure informa- tiveness, we use IDF scores.
To(c 8)
mg(0lar, s)ml((10x
())((,s))((())) s)m3(o)(, s)mă (wlo, x, s).
Adjacent Edits If the user simulator simply re- turns the most informative edits, it will have a ten- dency to make edits in disparate parts of the text. For example, it might return keywords from the end of the document, when the current draft only covers the start. In realistic setting, users make edits related to the current draft, they do pre- empt the end of the document. Thus, limit the user simulator to producing adjacent edits, where an edit is adjacent it is adjacent to a match in the alignment to the target.
Each of m.. Th. T8 m is implemented as a separate MLP head.
Model Inputs come prohibitively large, so keep track of only the last three drafts, denoted by Uh- A and Uh. This allows the agent to reason about the edits made at the last step and how the user responded. Additionally, we track all words inserted by the user along the entire history. This is easy to do by marking the relevant words, and allows the agent to know which words in the current draft came from the user. In practice, all these features repre- sented as a diff. See Figure for an illustration, where the different coloring and markings repre- sent which words were inserted or deleted by the agent or user. Practically, we implement this by adding labels to the tokens in the draft Un. We can easily track these labels for the user and token edit models because they operate by making edits. So if the model user deletes word, we can mark it deleted. For the Seq2Seq model this isn't possible because returns an entire new draft. For example, we couldn't 't tell a word was deleted or substituted for another. Instead, we compute the alignment be- tween U and A from which we can read off which tokens were inserted, deleted or substituted. Because words inserted by the user should likely never be deleted by the agent, we keep the labels on all words inserted by the user that they persist throughout the entire interaction.
The history of drafts S₁ could be-
Contiguous Edits While adjacency will keep the user simulator edits relevant the current draft, may still have tendency to produce a disconnected set of edits. Instead, we limit it to only produce contiguous edits.
Complete Words Finally, since the user simu- lator operates on tokenized text, and the tokens may break up words, we also constrain its edits to complete words.
B.3 Models and Training
Implementation Models were implemented with Transformer architectures with additional MLP layers on top. Specifically, used the BART base and BART large checkpoints made avail- able through the HuggingFace transformers library (Wolf et al., 2020). The models have (approxi- mately) 140M and 400M parameters respectively. For fixed document x and state s, the token edit- ing model parametrizes distribution over edits along with stopping action. We refer to the union edit actions, which are four tuples (s, o, w). where E {0, 1} indicates the stopping action, and 1, o, w, are as defined for edits. The probability of an edit action product of four probabilities, depending on three
Training Models were trained single 16GB V100 GPU for 600 iterations with sampling batch size of B 1 and sampling annealing rate of 0.9. We also used 300 warmup iterations (where was not annealed). For the token editing model we used a noise level of a = 0.3.
(s, I, o, w) is parameterized as a

---

Decoding We decode from the S2S autoregres- sive model using beam search with a beam size of 10. For the token editing model we used top 10 sampling actions, stopping probability thresh- old of a = 0.95 and a maximum number of edits of N = 64.
B.4 Metrics
We evaluated generation using BLEU (Papineni et al., 2002) and CHRI (Popović, 2015) with the SaCreBLEU implementation (Post, 2018). We also evaluate using BLEURT (Sellam et al., 2020) and BARTSCORE (Yuan et al., 2021), which are model- based metrics that have been shown to correlate well with human judgment on various text gener- ation tasks. As BARTSCORE returns a score inter- pretable as a pog-probability, we report the natural exponent of that score. We also use ROUGE (Lin, 2004) for evaluation its scores tend to be less sensitive length. In the case of both BLEU and ROUGE, we perform eval- uation with their unigram versions (BLEU-1 and ROUGE- 1) as they provide interpretability of the results at the token level.
alternative to BLEU, as


Reviews:
[{"review_id": "ekCvUmtAGI", "reviewer": "Reviewer_tSro", "summary": "", "questions": "- Given the expert policy always produces the goal G, what is the different between using the expert policy and simply using supervise objective to train the model for each turn?\n- How does the expert policy implemented for the token editing model?\n- For the seq2seq baseline, what is the size of the model? Given that there are two model variants for token editing model, i.e., editor and editor large, readers would expect the two variants are also incorporated for the baseline to get a better comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our work. To address your questions and comments:\n\n1. “Many important details of the paper are not incorporated in the main paper [...]”\n\nWe acknowledge that many technical points were left to the appendix. Because our paper is introducing a new framework for interactive generation, we opted to use most of the available space to discuss the framework, while moving more technical details to the appendix. Given the additional space provided for revisions, we will clarify these technical points directly in the main paper.\n\n2. “The paper claims that the work contributes different user simulators [...]”\n\nThe different user simulators come from the different user heuristics described in Appendix B.2. These can be combined or tuned in various ways to produce different user simulator behaviors. We evaluate with different user simulators in Appendix C, Table 4. However, as you note, this appendix is not referred to in the text. We will move these results back into the main paper, or make more explicit reference to them and clarify our claim.\n\n3. “Given the expert policy always produces the goal G, what is the different between using the expert policy and simply using supervise objective [...]”\n\nYou are correct that because the expert is quite simple, this is equivalent to training with a supervised objective. However, the main reason to use the expert is for the roll-in policy, which is a mixture between the expert and the current trained policy.\n\n4. “How does the expert policy implemented for the token editing model?”\n\nThe expert policy, as referred to in our answer in point 3, is the same for all models. However, the way that the token editing model is trained to produce drafts can itself be seen as a form of imitation learning, although we do not explicitly frame it that way in our paper. If you are referring to the expert in this context, it uses an alignment between the current draft and the goal document to determine a set of edit trajectories from the draft to the goal. This is akin to an expert policy in a driving task that dynamically finds a route from the vehicle's current position to a target position. This is explained in full detail in appendix A.3.\n\n5. “For the seq2seq baseline, what is the size of the model? [...]”\n\nThe seq2seq baseline (S2S) and the token editing model (Editor) are the same size. Both are based on the BART base checkpoint from Huggingface. Technically, the seq2seq baseline is actually larger since it uses both the encoder and decoder, while the token editing model only uses the encoder. Full implementation details are given in Appendix B.3, but we will clarify this point in the main text.\n\n6. Missing references\n\nWe will include the additional references you mentioned in the related work. We do however wish to note that, while these other works exploring interactivity are certainly important to discuss, we believe our work explores an orthogonal direction that has not yet been considered, since we formalize a concrete framework for interactivity that is used not just at inference time but also at training time.\n\n7. “Line 164--165: [...] unclear”\n\nWe will rewrite the sentence in lines 164-165 to make it clearer\n\n8. “In the oracle model of Figure 2, it will be easier to understand if it is stated which one is Ah and which one is Uh”\n\nWe will clarify in Figure 2, in the Oracle Model box, which document is A_h and which is U_{h+1}\n\n9. “Appendix C is never refered in the main paper.”\n\nAs noted in point 2, we will move appendix C back into the main paper, or make the reference more explicit.\n\nThank you again for your review, and we hope the answers and proposed changes above address your questions and concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Although many papers have showcased that interactivity can improve generation results, this paper introduces a formalized framework to train and evaluate model interactivity\n- The paper introduces a token editing model that generates revised documents via iterative token editing steps. The paper also provides a thorough evaluation compared to the seq2seq baseline", "reasons_to_reject": "- Although the paper mentions the scoring function s(·; G) can be generalized for multiple goal documents or even without any gold document, such generalization is not explored which makes the generalization of the framework itself questionable.\n- Many important details of the paper are not incorporated in the main paper, which makes it difficult to follow the paper without thoroughly reading the appendix.\n- The paper claims that the work contributes different user simulators and the performance of the model remains consistent with different user simulators. This contribution is nowhere to be found in the main paper.\n- The experiment is only conducted on smaller-scale LM which is not instruction-tuned, despite many prior works working on a similar problem exploring interactivity using much larger LMs.", "questions_to_authors": "- Given the expert policy always produces the goal G, what is the different between using the expert policy and simply using supervise objective to train the model for each turn?\n- How does the expert policy implemented for the token editing model?\n- For the seq2seq baseline, what is the size of the model? Given that there are two model variants for token editing model, i.e., editor and editor large, readers would expect the two variants are also incorporated for the baseline to get a better comparison.", "paper_topic_and_main_contributions": "The paper introduces a new task and framework that requires user-system interactivity namely interactive text generation. The task and framework requires the model to generates a document similar to the target document G, starting from a blank document A0. Instead of using a real user for interactivity, the work incorporate a user simulator (oracle model) which produces a revision list Uh given the generated draft document Ahand the gold document G. The paper showcases two imitation learning models to iteratively generate draft, i.e., Left to Right Autoregressive Model and Token Editing Model. The autoregressive model directly generates a new draft Ah using the draft and revision history Sh in an autoregressive manner, while the token editing model perform iterative word-to-word editing until it achieves a certain stopping criterion. The effectiveness of the framework is tested on single sentence generation retrieved from the article summaries of the CNN/DailyMail dataset. The results suggest that: 1) token editing model can be comparable to the seq2seq baselines while being more naturally suited to the task, 2) the interactive system shows better semantic similarity with the gold, while having a slightly lower fluency score.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "-"}, "extra_text": {"Missing_References": "Recent works exploring multi-turn interactivity using LLMs\n@inproceedings{\n    sun2023recitationaugmented,\n    title={Recitation-Augmented Language Models},\n    author={Zhiqing Sun and Xuezhi Wang and Yi Tay and Yiming Yang and Denny Zhou},\n    booktitle={The Eleventh International Conference on Learning Representations },\n    year={2023},\n    url={https://openreview.net/forum?id=-cqvvvb-NkI}\n}\n\n@misc{bang2023multitask,\n      title={A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity}, \n      author={Yejin Bang and Samuel Cahyawijaya and Nayeon Lee and Wenliang Dai and Dan Su and Bryan Wilie and Holy Lovenia and Ziwei Ji and Tiezheng Yu and Willy Chung and Quyet V. Do and Yan Xu and Pascale Fung},\n      year={2023},\n      eprint={2302.04023},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n\n@misc{huang2022large,\n      title={Large Language Models Can Self-Improve}, \n      author={Jiaxin Huang and Shixiang Shane Gu and Le Hou and Yuexin Wu and Xuezhi Wang and Hongkun Yu and Jiawei Han},\n      year={2022},\n      eprint={2210.11610},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n\n@misc{gou2023critic,\n      title={CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing}, \n      author={Zhibin Gou and Zhihong Shao and Yeyun Gong and Yelong Shen and Yujiu Yang and Nan Duan and Weizhu Chen},\n      year={2023},\n      eprint={2305.11738},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 164--165: If we denote the agent, resp. user, draft at step h by Ah, resp. Uh ... => incomprehensible sentence & the term \"resp.\" is unclear\n- In the oracle model of Figure 2, it will be easier to understand if it is stated which one is Ah and which one is Uh\n- Appendix C is never refered in the main paper."}}, {"review_id": "B9SjlU2sJb", "reviewer": "Reviewer_HjwP", "summary": "", "questions": "1. I'm curious about the choice of N being fixed for a given task because the relationship between the total budget of edits and the length of the document is likely non-linear. For instance, if you are creating a longer document, then you might see a much greater benefit from a larger editing budget than a shorter document (in the interactive case because you can guide the document more as opposed to providing a longer initial draft).\n\n2. Does increasing the number of episodes in Fig. 4 cause the lines to taper off i.e. is there a limit to the benefit of interactivity?\n\n3. The choice of summarization is apt for the interactive generation due to the subjective nature of relevance for the task. However, I think that the chosen datasets do not best fit this need. Perhaps query focused summarization datasets like Squality could be interesting to work with?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our work. To address your questions and comments:\n\n1. “[...] the authors make an assumption that all edits are equal regardless of when they are provided in the writing process. [...]”\n\nYou are correct that the quality of an edit depends on when it is given in the generation process. This is in fact one of the points that we would like to make. In our evaluation, we show that solely by letting a model receive edits from the user intermittently, rather than all at the start, the model is able to produce a better document. This should clearly demonstrate that getting edits later can be beneficial, which justifies the use of interactive generation models over one-shot models.\n\n2. “[...] it is important to discuss existing work in the collaborative writing space and the evaluation of interactive systems from the intersection of HCI/NLP.”\n\nWe agree that it would be beneficial to have a broader discussion of related work and we will add and discuss the additional references you suggest. However, we do wish to note that our goal in this work was to formulate a very concrete framework for interactive generation, so we believe our work to be orthogonal, but complementary, to work at the intersection of HCI/NLP.\n\n3. “I'm curious about the choice of N being fixed for a given task [...]”\n\nYou are correct that N should vary with respect to the task length. In our evaluations, we keep N fixed to ensure fair comparisons between interactive and non-interactive settings. However, the task does not otherwise assume a fixed number of edits (for technical reasons there is a fixed horizon H, but this can be very large). With a real user, the interaction stops whenever the user is satisfied with the document, so it may go on for as long as needed.\n\n4. “Does increasing the number of episodes in Fig. 4 cause the lines to taper off [...]”\n\nIn our evaluation setup we do expect to see a taper. We fixed the total number N of edits made by the user  to ensure a fair comparison. As the number of episodes M increases, the number of edits N/M made by the user in each episode decreases. Eventually, this may be so small that the model cannot gain any useful information from the user anymore. For example, if the user makes a single edit to a particle word (e.g. they change \"a\" to \"an\"), this would not convey much information.\n\n5. “[...] Perhaps query focused summarization datasets like Squality could be interesting to work with?”\n\nA summarization dataset such as Squality could be interesting to work with, but it would require an additional component of conditioning on the text to be summarized, which we believe falls out of scope for this initial work. However, this could make for an exciting future direction.\n\n6. “I found it confusing to say 'sequence to sequence autoregressive model' (l.263, 405) [...]”\n\nWe will change the terminology from sequence to sequence autoregressive model to left-to-right autoregressive model which should be more apt.\n\n7. “In l.165, what is the 'resp user.' meant to convey?”\n\nWe will rewrite line 165 to avoid the use of resp. (short for respectively).\n \nThank you again for your review, and we hope the answers and proposed changes above address your questions and concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Given some assumptions made in the formulation (discussed in Weakness 1), the paper provides a clear formulation for the interactive task setting and an accompanying user simulator/agent training algorithm. \n\nWhile I did mention some missing references, I think the novelty of providing a means of comparison between interactive and non-interactive generation (given some assumptions) is great. \n\nThe training of the editor models (section 4.3) more directly matches the desired interactive evaluation which results in clear performance improvements over the non-interactive baseline. ", "reasons_to_reject": "In order to compare interactive and non-interactive models, the authors make an assumption that all edits are equal regardless of when they are provided in the writing process. This is not always true as some key edits provided once some progress toward the final draft is made could influence a document more than just an initial draft to begin with.  \n\n\n(Minor) There's a mismatch between the motivating examples and the formulation in Section 2. In lines 121-126, the examples given are a student writing with an advisor/freelancer obtaining feedback from their client. In each of these cases, the latter is the 'user'. Then the formulation in lines 137-143 describes that the user knows the oracle document which is the target of the generation process. I think that this is not always realistic. Perhaps this is true only in a student/teacher setting and in each of the editing examples provided, the user is not necessarily an oracle but just an agent that exercises preferences (which are subjective and malleable based on the generated documents and not objective oracle) in the editing process. \n\n\n(Minor) While generally missing references are not a 'reason to reject', I do think that since this work is positioned as a new task setting, it is important to discuss existing work in the collaborative writing space and the evaluation of interactive systems from the intersection of HCI/NLP. \n\n\n\n", "questions_to_authors": "1. I'm curious about the choice of N being fixed for a given task because the relationship between the total budget of edits and the length of the document is likely non-linear. For instance, if you are creating a longer document, then you might see a much greater benefit from a larger editing budget than a shorter document (in the interactive case because you can guide the document more as opposed to providing a longer initial draft).\n\n2. Does increasing the number of episodes in Fig. 4 cause the lines to taper off i.e. is there a limit to the benefit of interactivity?\n\n3. The choice of summarization is apt for the interactive generation due to the subjective nature of relevance for the task. However, I think that the chosen datasets do not best fit this need. Perhaps query focused summarization datasets like Squality could be interesting to work with?", "paper_topic_and_main_contributions": "This paper makes the case that we often do not train/optimize user-centric machine learning models to perform well interactively. The authors' claim is that what we really want in this setting is a model that adapts well to user instructions to obtain the desired output (not necessarily one that gets as close as possible in a one-shot manner). \n\n(Contributions)\nHence the authors formulate an interactive text-generation task setting where the model also receives a fixed budget of edits from a 'user' along with the task input/prompt in order to guide the generation process. \nIn this regime, the authors are able to compare interactive and non-interactive models by varying when the 'user edits' are received - in the non-interactive case, all edits are received at the start. \nThe authors then present experiments evaluating various models in the interactive vs. non-interactive setting on the task of summarization. In order to do so, they also create a simple user simulator to provide these 'edits' which are essentially a form of oracle information during the generation. The findings show that interactivity helps across model architectures (autoregressive and seq2seq transformer models) and across various user simulators. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "While the work does discuss some interactive NLP work, there are a number of collaborative writing works which also fit the pairwise writing setup [1,2,3] and could be discussed in the context of the motivation for the work. These vary in the style of interaction but as this work aims for a broad task definition of interactive generation, it needs more discussion in this line. In addition, recent work in HCI has also made strides in discussing how we evaluate interactive generation tasks [4,5]. \n\n[1] Akoury, Nader, et al. \"Storium: A dataset and evaluation platform for machine-in-the-loop story generation.\" arXiv preprint arXiv:2010.01717 (2020). \n[2] Clark, Elizabeth, et al. \"Creative writing with a machine in the loop: Case studies on slogans and stories.\" 23rd International Conference on Intelligent User Interfaces. 2018.\n[3] Du, Wanyu, et al. \"Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision.\" arXiv preprint arXiv:2204.03685 (2022).\n[4] Lee, Mina, et al. \"Evaluating human-language model interaction.\" arXiv preprint arXiv:2212.09746 (2022).\n[5] Shen, Hua, and Tongshuang Wu. \"Parachute: Evaluating interactive human-lm co-writing systems.\" arXiv preprint arXiv:2303.06333 (2023).", "Typos_Grammar_Style_and_Presentation_Improvements": "1. I found it confusing to say 'sequence to sequence autoregressive model' (l.263, 405) when the term sequence-to-sequence is often used for non-autoregressive transformer models. The terminology in the paper is consistent, but a little off the commonly used one. \n\n2. In l.165, what is the 'resp user.' meant to convey?"}}, {"review_id": "WscPNpzSKu", "reviewer": "Reviewer_sfhR", "summary": "", "questions": "A. Does the current document limit the number of edits when aligned with the target document? If so, it is usually set to a few times. How is this set?\nB. How to determine what new words to insert during editing？\nC. Whether the problem of user input errors in the interaction process has been considered, and whether the model can fix the errors.\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our work. To address your questions and comments:\n\n1. “The description of the model is not clear enough [...]”\n\nWe acknowledge that many technical points needed to be relegated to the appendix in the interest of saving space, however we will use the additional space provided for revisions to clarify any ambiguous points in the main paper.\n\n2. “It does not take into account the error of real users in the input process and lacks the robustness study of the model.”\n\nWe discuss the point of user error below in response to your other question. Concerning robustness, we do include ablations in Appendix C, which may address your concern.  We will also move those results back into the main paper using the additional space provided for revisions.\n\n3. “For the part of the imitation learning model, the details such as ensuring accuracy are not clearly presented.”\n\nAs mentioned in our reply to your first point, we will move some of the discussion of more technical points from the appendix back into the main paper to make the presentation clearer and easier to follow.\n\n4. “Does the current document limit the number of edits when aligned with the target document? [...]”\n\nThe limit on the number of edits was only set for the purpose of a fair comparison between interactive and non-interactive settings. It was set based on the average length of the sentences in the dataset, and for the simple practical reason that N=6 is divisible by both 3 and 2. In a real user scenario, there would be no such limit and the user would be free to make as little or as many edits as they choose.\n\n5. “How to determine what new words to insert during editing?”\n\nIf you are referring to the training of the token editing model, this is inferred using an alignment of the current draft to the goal document, from which we can derive sequences of edits that transform the draft into the goal. This is explained in more detail in Appendix A.3.\n\n6. “Whether the problem of user input errors in the interaction process has been considered, and whether the model can fix the errors.”\n\nThis is an interesting point about user errors. The user simulators we use do not make mistakes, so the model learns to trust the edits made by the user and therefore rarely corrects the user. However, it would in principle be possible to simulate errors from the user, much in the same way that we simulate errors from the model so that it learns to correct itself (Appendix A.6).\n  \nThank you again for your review, and we hope the answers and proposed changes above address your questions and concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a new interactive text generation task and benchmark, constructs the interaction process between agent and simulated user, and designs a new idea for the interactive text generation task.\n2. The proposed method utilizes the idea of simulation learning to simulate the user editing process\n3. This paper shows the performance of text generated after each generation of interaction, and clearly shows that the effect of text generated gradually becomes better with the increase of the number of interactions", "reasons_to_reject": "1. The description of the model is not clear enough, and there are some confusing points.\n2. It does not take into account the error of real users in the input process and lacks the robustness study of the model.\n3. For the part of the imitation learning model, the details such as ensuring accuracy are not clearly presented.\n", "questions_to_authors": "A. Does the current document limit the number of edits when aligned with the target document? If so, it is usually set to a few times. How is this set?\nB. How to determine what new words to insert during editing？\nC. Whether the problem of user input errors in the interaction process has been considered, and whether the model can fix the errors.\n", "paper_topic_and_main_contributions": "This paper solves a new end-to-end generation task - interactive text generation. The main contributions are that the proposed model can be trained by interacting with user simulators. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel framework for interactive text generation and introduces a new task benchmark. The reviewers agree that this framework and benchmark are useful for interactive text generation. The paper also proposes an interactive text-editing model and provides a thorough analysis of the proposed method. One area for improvement raised by the reviewers is that some of the important technical details are buried in the appendix. It would be beneficial to include these in the main body of the paper, a suggestion with which the authors have agreed, given the extra page.
Decision: Accept-Main