SlowGuess/NAACL_2022_OCR huggingface dataset:
Vishakh Padmakumar
Machine-in-the-loop writing aims to build models that assist humans to accomplish their writing tasks more effectively. Prior work has found that providing users a machine-written draft or sentence-level continuations has limited success since the generated text tends to deviate from users' intention. To allow the user to retain control over the content, we train a rewriting model that, when prompted, modifies specified spans of text within the user's original draft to introduce descriptive and figurative elements in the text. We evaluate the model on its ability to collaborate with humans on the task of creative image captioning. On a user study through Amazon Mechanical Turk, our model is rated to be more helpful by users than a baseline infilling language model. In addition, third-party evaluation shows that users write more descriptive and figurative captions when collaborating with our model compared to completing the task alone. However, the improvement is not uniform across user groups: the model is more helpful to skilled users, which risks widening the gap between skilled and novice users, highlighting a need for careful, user-centric evaluation of interactive systems.
Creative writing tasks are challenging for humans because of their open-ended nature. Prior work shows that exposing authors to a collaborator that provides independent suggestions can spark new ideas (Garfield, 2008). This has motivated a line of work in machine-in-the-loop writing (Clark et al., 2018; Roemmele and Gordon, 2015; Samuel et al., 2016) where a human collaborates with a model to complete a writing task. However, recent work has shown that providing humans a draft generated by a machine is not very helpful because it may
diverge from the direction envisioned by the author (Clark et al., 2018). As a result, very little machine-generated text is ultimately retained (Akoury et al., 2020).
In this work, we aim to provide a form of interaction that gives human authors more control over the content while also assisting them to better express their own ideas (Roemmele and Gordon, 2015). We focus on the setting where authors have a clear writing outline but would benefit from suggestions on wording or framing. To allow authors to control the content, we develop a machine-in-the-loop system called Creative Rewriting Assistant (CRA) which either rewrites a span of text or infills between two pieces of text when requested (Figure 1). CRA is a sequence-to-sequence model, building upon recent advances in controllable text generation (Shih et al., 2019; Ma et al., 2020; Kumar et al., 2020) and text infilling (Donahue et al., 2020; Fedus et al., 2018; Joshi et al., 2019; Shen et al., 2020). We train the CRA model on a pseudo-parallel corpus of sentence pairs—a generic sentence and a more descriptive or figurative alternative (Section 3.1).
We evaluate the model on the task of creative image captioning (Section 2). Users that collaborate with CRA report that it is significantly more helpful than a baseline infilling language model (Section 5.1). Additionally, through a controlled experiment, we find that, on average, users writing with CRA produce more creative captions than those writing without assistance, highlighting the end-to-end benefit of our machine-in-the-loop setup (Section 5.3). In particular, users writing with the model produce captions with a more diverse vocabulary.
To understand how the system impacts different users, we analyze the user-model interaction logs (Section 6) and find that the machine-in-the-loop setup is more helpful to skilled writers because they tend to request targeted suggestions for shorter spans of text while giving the model sufficient con
1Our code and pretrained models are available at https://github.com/vishakhpk/mil-creative-captioning
Human Language Technologies, pages 588 - 601
Figure 1: Machine-in-the-loop rewriting for image captioning. The human is the central actor in the writing process and initiates interactions with the model by indicating what spans of text are to be rewritten. The model provides suggestions at these locations and the user chooses how to use them.
text. This highlights a need to study the impact of study of interactive systems on different user groups as these become more ubiquitous in assisting content generation given that this technology could result in an even wider gap in performance between different users.
2 System Overview
Creative Image Captioning To evaluate our system, ideally, we would use tasks like poem or story writing. However, it is challenging to control the content for a fair comparison of different systems on such open-ended tasks. Therefore, we evaluate on a proxy task, creative image captioning (Chen et al., 2015), where the user is asked to write an expressive caption (a figurative or descriptive one as opposed to a literal one) for a given image. The user is given access to a model that provides editing suggestions while they are working on the task. Our goal is to study if collaborating with the model makes them more effective at completing the task. Note that our model does not use the image when generating the suggestions, which is analogous to real use cases where the model does not have access to the author's global writing plan but instead provide improvements based on the local textual context.
Machine-in-the-Loop Rewriting An overview of our system is illustrated in Figure 1. The user
collaborates with the model to complete the writing task. We follow the user-initiative setup (Clark et al., 2018) where the model provides suggestions only when requested by the user. The system facilitates two types of editing: span rewriting and text infilling. Given a piece of text (written by the user), to request span rewriting, the user demarcates spans within the text that need to be rewritten. The model then edits the marked spans. For example, given "The iPhone was a [great piece of technology] that changed the world", the model suggests the rewrite "The iPhone was a revolution in technology that changed the world". To request text infilling, the user marks blanks to be filled. For example, given "The lion stalks the deer; a in its element", the model infills "The lion stalks the deer; a predator in its element".
By limiting the edits to local spans, we alleviate the issue of deviating from the input content or generating incoherent text (Holtzman et al., 2019; Wang and Sennrich, 2020). For both rewriting and infilling, multiple suggestions are provided for the user to consider. Then, they have the option to either accept a suggestion and continue writing, or reject them and retain their initial draft. This interactive process continues until the user is satisfied with the text and indicates the completion of the writing task.
Figure 2: Training data creation. The source sentence is created by masking out the annotated span and infilling it using BART-Large. The model is then trained to produce the creative sentence from the synthesized source sentence.
Source	Domain	Annotation	Example
Mohammad et al. (2016)	WordNet example sentences	Words that elicit emotion	I attacked the problem as soon as I was up.
Gordon et al. (2015)	Text collected by Mohler et al. (2015)	Metaphors in text	I will be out in the city today, feeling the vinous veinous thrust of blood, the apple-red circulation of democracy, its carnal knowledge without wisdom.
Bostan et al. (2020)	Headlines	Textual cues associated with emotion	Detention centers will shock the conscience of the nation.
Niculae and Danescu-Niculescu-Mizil (2014)	Product reviews	Figurative language	The stones appeared dull and almost opaque, like black onyx, with none of the sparkle you would expect from something called a diamond.
Steen et al. (2010)	News, fiction and academic text	Metaphors and personification	Like a buzzard in the eyrie, he would fly around.
Table 1: Sources of creative text and annotations used for creating training examples.
3.1 Learning from Creative Text
Our goal is to train a model capable of rewriting specific spans of an input sentence requested by a user, to assist them with the creative writing task. To this end, we need a dataset of paired sentences where the target sentence is produced by replacing or inserting text spans in the source sentence to make it more descriptive or figurative. While no such datasets exist, there are many resources that study creative text by annotating text spans with their corresponding literary devices (including metaphors, emotional cues, and figurative comparisons). We thus take the creative text from these datasets as the target sentences, and synthesize the source sentences by replacing the annotated creative spans with infills from a generic language model, which typically produces less creative text. An example is shown in Figure 2.
Specifically, we start with a creative sentence from one of the datasets listed in Table 1, mask the annotated creative spans in it, and infill them using the pre-trained BART model (Lewis et al., 2019) to generate the non-creative source sentence. For
each pair from this pseudo-parallel corpus, we create one rewriting example by inserting the rewrite markers, <replace> and </replace>, at the beginning and the end of the rewritten span, as well as one infilling example by replacing the span with a mask token, <mask>. We then train CRA to predict the creative sentences given the generic source sentence using cross-entropy loss.
3.2 Learning from Interactions
One important advantage of machine-in-the-loop systems is that they can be improved given user feedback. Once users interact with CRA, we obtain their reaction to the suggestions, i.e. acceptance and rejection. This feedback allows us to update the model, so that it adapts to the observed user preference over time. Specifically, we create an example pair whenever the user indicates a preference for one sentence over another when presented with model suggestions. When the user accepts a suggestion, we take the accepted suggestion as the target (creative) sentence and the user's initial input as the source (non-creative) sentence. Similarly, when the user rejects a suggestion, we take the rejected suggestion as the source and the user's
initial input as the target. Thus, the model always learns to improve the source sentence. We then add these new pairs to a similar-sized subset of the original training examples (to prevent forgetting) and fine-tune the rewriting model on the combined dataset.
We train the CRA model using the scheme laid out in Section 3.1 and deploy it in the machine-in-the-loop setup detailed in Section 2 in order to answer the following research questions.
User Experience When we study collaborative writing, the key stakeholder is the user so we evaluate if users writing in tandem with CRA find the model helpful. To answer this, we run a user study and compare suggestions obtained from CRA and a baseline BART model in the machine-in-the-loop setup (Section 5.1). Once CRA is deployed with real users, we would like to adapt it to users' preferences inferred from the observed interactions. Hence we also compare a user-adapted model (per Section 3.2) to the previously deployed CRA (Section 5.2).
Quality of the Writing We also want to study the outcome of the collaboration. In particular, does CRA help users' write higher quality captions for images? We compare captions collected from the machine-in-the-loop setup to those obtained from solo-writers using third-party annotators to see if users write more creative captions in a collaborative setup (Section 5.3).
Broader Impact of Collaboration In the machine-in-the-loop setup, we introduce CRA into the writing process. This intervention potentially impacts different users differently. In particular, we study how skilled and novice user groups interact with CRA to understand how such a model impacts the skill gap between users of different backgrounds (Section 6).
User Study We hire users on Amazon Mechanical Turk to perform the creative image captioning task. A screenshot of our user interface and the details about worker remuneration are provided in Appendix B. The plan for our user study was approved by the Institutional Review Board of our university. Each user is presented with an image
and asked to write a caption that is as figurative and/or descriptive as possible with at least 100 characters. The images were randomly sampled from the figurative subset of the Déjà Captions dataset (Chen et al., 2015), where the gold caption contains literary elements like metaphors and hyperbole. We ask users to request suggestions from the model at least twice while they are writing; however, they are free to ignore the suggestions. Users are instructed to use square brackets (as seen in Figure 1) to mark spans to be rewritten and underscore to indicate blanks to be filled. They can edit the text with the model iteratively until they are satisfied with the caption. Once users submit the final caption, they are asked to complete a survey to rate the helpfulness and grammaticality of the assistant as well as their satisfaction with the final caption. The full task instruction is provided in Appendix B.
Model Details To train the CRA model, we first create the pseudo-parallel corpus as detailed in Section 3.1. Using creative sentences from all the sources from Table 1, we obtain a corpus containing 42,000 training pairs, 2,000 validation pairs, and 1,626 test pairs. The CRA model is trained by fine-tuning the fairseq (Ott et al., 2019) implementation of BART on the training set of this corpus. We train the BART-Large pre-trained checkpoint from fairseq for 5 epochs with a learning rate of  $3 \times 10^{-5}$ . The learning rate was selected by held-out perplexity on the validation set. We use the recommended default values in fairseq for the hyperparameters of the Adam optimizer (Kingma and Ba, 2014), dropout rate, and learning rate scheduler.
To evaluate whether CRA provides helpful suggestions, we compare its performance to a pretrained infilling language model, BART (Lewis et al., 2019). When using BART for rewriting, we mask and then infill the spans of text demarcated by users (regardless of whether they are meant to be rewritten or infilled). To produce creative generations, a balance between diversity and fluency is desired during decoding. A small internal pilot shows a lack of diversity in beam search outputs. Thus, we use top- $k$  sampling for both models, with  $k$  set to 10.
2The beta values for the Adam optimizer are 0.9 and 0.999, the dropout rate is set to 0.1, and we use a polynomial decay learning rate scheduler with the weight decay set to 0.01.
5.1 User Evaluation of Suggestion Quality
To evaluate the quality of the suggestions provided by CRA vs. the pre-trained BART baseline, we conduct A/B testing on 50 images randomly sampled from the Déjà Captions dataset. Upon connecting to our server, each user is randomly assigned to work with either BART or CRA. We ensure that each image has one caption from each model. In addition, users working with both models are recruited from the same pool during the same time period, which minimizes the difference in performance due to individual users.
Once the task is completed, we ask the user to answer the following questions about the model on a Likert scale of 1 (worst) to 5 (best):
- How helpful were the model suggestions?
- How grammatically correct were the model suggestions?
How satisfied were you with the final caption?
In addition, to analyze the effect of users' initial writing ability, we ask them to assess their writing skills:
- How would you rate your own writing ability on a scale of 1 to 5? 1—I don't have much experience with writing or am not too confident with the language, to 5—I have writing experience and/or have considerable proficiency with the English language.
We also examine if this user-rated helpfulness tallies with automatic metrics that we compute on the observed interactions.
Results The results from the survey are presented in Table 2. Each reported value is an average of 50 user responses. We find that, on average, users find CRA to be more helpful than BART and report no significant difference between the two models in terms of grammaticality. By training CRA on the pseudo-parallel creative corpus, we align the model suggestions better to the creative writing task, resulting in a more helpful collaborator.
To see if the human evaluation tallies with automatic metrics, we calculate the fraction of model suggestions accepted by the users, across the 50 user responses, is reported in Table 3. CRA has a higher acceptance rate than BART, consistent with the helpfulness rating from users. The total number of suggestions requested from BART is slightly higher, perhaps explained by its lower acceptance rate—users may persist with variants upon receiving unsatisfactory suggestions.
Accepting a suggestion does not necessarily mean that it is useful since the user may further edit it. In fact, prior work has shown that a large fraction of the suggested text is not retained by the user (Akoury et al., 2020). To measure how much of the accepted suggestions are actually used, we calculated the Rouge-L recall score of accepted suggestions against the final caption submitted by the user. As shown in Table 3, larger fractions of CRA's suggestions were retained by users.
Question	BART	CRA
Helpfulness	2.23*	3.06*
Grammaticality	2.96	3.22
Satisfaction	3.69	3.65
Table 2: User evaluation of model performance for pretrained BART baseline vs. CRA. Each value is an average across 50 user scores per model. Bold values correspond to the higher average. Rows marked with an asterisk indicates statistically significant differences ( $p$ -value  $< 0.05$  according to a Mann-Whitney-Wilcoxon test). Users find the CRA model to be more helpful by a statistically significant margin.
# request	# accepted	% accepted	Rouge-L
BART	151	37	24.5	0.744
CRA	141	45	31.9	0.824
Table 3: Interaction statistics - How many suggestions were requested and accepted for the different models aggregated across 50 users for each model and the Rouge-L recall scores of accepted model generations against the final caption submitted by the user. The higher score of the two is bolded. Users accept more suggestions and retain more text from CRA.
5.2 Effect of Learning from User Interaction
From Section 5.1, we see that users find CRA to be more helpful than an infilling baseline model. In order to further align CRA to users' preferences, we fine-tune the model on paired examples created from their acceptance and rejection of the model suggestions (Section 3.2). The interactions with 50 users (collaborating with the CRA) result in a dataset of 474 pairs of sentences. To ensure that the model does not overfit to these examples and forget prior training on the pseudo-parallel creative corpus (Section 3.1), we also sampled 450 sentence pairs from it and added these to the interaction dataset. We then fine-tune the previously deployed CRA model for 5 epochs on this dataset. We choose the learning rate of  $3 \times 10^{-6}$  using five-fold cross
validation with the criteria of label smoothed cross-entropy loss  ${}^{3}$  . We evaluate this user-adapted CRA model against the initial CRA model on a fresh sample of 50 images, following the A/B testing setup from Section 5.1 .
Does user feedback improve the model? Our hypothesis is that adapting the model to user feedback should make it more helpful for subsequent users. From Table 4, we see that the users do find the updated model to be slightly more helpful than the initial model on average; however, a Mann-Whitney-Wilcoxon test shows that this difference is not statistically significant ( $p$ -value  $= 0.402$ ). A possible reason is that the different usage patterns of different users leads to the model getting noisy feedback and not significantly improving on the initial trained state. Thus, a potential future direction is to explore adapting the model separately to each user in a few-shot setting, possibly with longer interactions.
Question	Initial CRA	User-adapted CRA
Helpfulness	2.81	3.05
Grammaticality	2.87	3.26
Satisfaction	3.67	3.78
5.3 End-to-End System Evaluation
In Section 5.1, we observe that users find CRA more helpful than a baseline BART model. However, is the quality of the caption improved by collaborating with the model? To answer this question, we collect three captions for each of the 100 images using three systems: two machine-in-the-loop systems using CRA and BART respectively, and one solo writing setup. For solo writing, we recruit workers from the same pool as before (Amazon Mechanical Turk) and provide them the same instructions as in the machine-in-the-loop setup, except that all mentions of model assistance are removed. We then ask a 'third-party' human annotator (who did not participate in the writing task) to compare
the captions pairwise for each image. The annotator is presented with two captions for the same image and asked to pick the more creative caption of the two. In this manner, we collect 3 separate annotations for each pairwise comparison for each image and decide the winning caption based on a majority vote.
Does working with CRA improve the final caption? From Table 5, we observe that both collaborative setups (Human+CRA and Human+BART) outperformed the solo-writing setup according to the majority vote. While prior work in the creative domain was unable to match the performance of the human-only baseline using a less controllable assistant that provides full length drafts (Clark et al., 2018), here we show that collaborative setups are able to improve creative output of human users, inline with the expectations of literature on creativity support systems (Garfield, 2008).
Table 4: User evaluation of model performance for the initial model vs. the adapted model trained on user interactions averaged across 50 user scores. Bold values indicate the higher average. Users find the adapted model to be more helpful, although the difference is not statistically significant.
Majority Vote Wins
Human-Only	45	55	Human+BART
Human-Only	43	57	Human+CRA
Human+BART	48	52	Human+CRA
Table 5: Pairwise comparison of 100 captions from machine-in-the-loop writing with our model (Human+CRA) and the baseline (Human + BART) as well as a human writing without assistance (Human Only). Wins were decided by a majority vote amongst 3 crowd workers. Users write better captions in a collaborative setup.
How does CRA influence the captions? To analyze how model intervention affects the output text, we measure the count of unique trigrams in 100 captions produced from the Human+CRA setup and the Human-Only setup. Collaborative users are exposed to suggestions from an external model so we expect the generated text to contain more diverse vocabulary usage. From Figure 3, we see that, on average, captions generated from the collaborative setup do contain more unique trigrams.
The improvement in written captions because of the collaboration does not only come from direct model interventions the text. Some users also reported that considering different alternatives suggested by the model provided inspiration on how to improve the text (even though the suggestions are not accepted).
<sup>3</sup>We again use the released fairseq fine-tuning script retaining the recommended hyperparameters for the Adam optimizer, dropout rate and learning rate scheduler.