SlowGuess/NAACL_2022_OCR huggingface dataset:
ID	Demarcated Source Sentence	Accepted Suggestion	Edit
1	A solemn woman place her mother's diary on a stepping stone her late father laid in the garden. The [surrounding pale grass gently sway in the cold breeze] while the woman ponders times of the past. Reminiscence now taking over and winter's beginning, the woman braces herself for dreary time to come.	A solemn woman place her mother's diary on a stepping stone her late father laid in the garden. The pale grass gently danced and teased in the wind while the woman pondered times of the past. Reminiscence now taking over and winter's begin-ning peaks, the woman braces herself for dreary time to come.	Figurative lan-guage
2	A man walks along the seashore with the hori-zon looming in the background. The dark clouds — as the sun sets for the day.	A man walks along the seashore with the hori-zon looming in the background. The dark clouds slowly disperse as the sun sets for the day.	Precise word-ing
3	The image represents wisdom and profound intelligence. It is the face of a man who lead the nation with grace and honor. It is a statue that reflects the [moral aspect of American people today].	The image represents wisdom and profound intelli-gence. It is the face of a man who lead the nation with grace and honor. It is a statue that reflects the moral aspect of a great man who lived and breathed the ideals of freedom and democracy	Emboldishment
Table 6: Examples where the model was successful in providing assistance. Bold spans in the source and target sentences are what marked by users and rewritten by the model, respectively.
ID	Demarcated Source Sentence	Poor Suggestion	Error Type
1	In front of a wall, a girl with blonde hair is on her hands who seems to be [coming out of a magical door]	In front of a wall, a girl with blonde hair is on her hands who seems to be laughing out loud.	Content drift
2	A child stands tall in a [wave] on the beach.	A child stands tall in a motorized scooter on the beach.	Content drift
3	I am witnessing a field of golden grain with a tall flower is blooming. That flower is not yet fully grown, yet its shades of purple are there and plainly visible. [Overall, the image is nice. I do believe, however, that the quality of the image could be sharpened a bit.]	I am witnessing a field of golden grain with a tall flower is blooming. That flower is not yet fully grown, yet its shades of purple are there and plainly visible. Overall, the image is nice. I do believe, however, that the quality of the image could be sharpened a bit.	Repeated the source
4	A beautiful [sunset.A] beautiful sunset in the ocean lighting up the sky in exotic colors.	A beautiful sunset in the ocean lighting up the sky in exotic colors. A breathtaking view of nature at its best.	Excessive editing
Table 7: Examples of rejected model suggestions. Bold spans in the source and target sentences are marked by users and rewritten by the model, respectively.
Figure 3: Comparison of text generated from a collaborative setup (Human+CRA) and solo-writers (Human-Only). Collaborative users tend to write more diverse captions containing more unique trigrams  $(N = 3)$
To provide the full picture of CRA, we manually labeled 50 rejected suggestions to identify common error modes. Some illustrative examples of these
are listed in Table 7. The most common failure case (21 out of the 50) is content drift: when the model is asked to replace key content words, sometimes the rewritten text changes the meaning of the user draft. This is seen in Examples 1 and 2 in Table 7, where the model changes "wave" to "motorized scooter"; while the suggestion is coherent, it changes the original meaning of the sentence. This is likely an artifact of how we create the pseudo-parallel corpus of training data: When BART performs infilling, the text introduced is not guaranteed to preserve the original content.[5] The second common error type (14 out of the 50) is to copy the source text verbatim (example 3 in Table 7), especially when a long text span (e.g., a full sentence) is rewritten, which is rare in our training data. Lastly, there is a small fraction of cases (9 out of the 50) when
To validate the quality of the pseudo-parallel corpus, we randomly sampled 50 sentence pairs and manually checked them for hallucinations. We observed hallucination 4 times out of 50 possibly explaining the observed content drift errors.
4We include representative user feedback in Appendix C.
the model makes suggestions outside the desired demarcated region—this is often seen when the demarcated text spans two sentences and contains incoherent phrases (example 4 in Table 7).
6 How Does CRA Impact Different Users?
Which users find CRA more helpful? Our main hypothesis is that CRA benefits human authors by giving them more control over the global content and providing local wording suggestions (Roemmle and Gordon, 2015). Thus, its effectiveness relies on the assumption that the user has a coherent writing plan, which may or may not be true depending on the skill level of the writer. To analyze the influence of users' inherent writing skill on model effectiveness, we put users into two groups based on their self-assessed writing ability (1 is the least skilled and 5 is the most skilled). A user is considered a skilled writer if they rate themselves higher than 3 and otherwise a novice writer. Out of the 50 users who interacted with CRA, 22 fall into the novice group and 28 fall into the skilled group. As a sanity check, the self-reported skill level is consistent with the result from the third-party evaluation—more captions written by skilled writers were judged as the winning caption than the novice writers (72.72% vs. 46.42%).
We show the ratings of helpfulness of CRA and the acceptance rate of model suggestions by user group in Table 8. We observe that skilled writers find the model more helpful and accept a higher fraction of the provided suggestions, while novice writers tend to request more suggestions with a lower acceptance rate. This is consistent with the hypothesis that the skilled writers have a more clear plan thereby playing to the model's strengths.
To understand if the discrepancy in reported model helpfulness between the two groups is due to them requesting different kinds of suggestions, we identify the characteristics of edits that CRA is good at and compare them to the requests made by the two groups.
Why is CRA more helpful to skilled users? The model is more effective at editing longer sentences. A longer context allows the model to better infer the content and style of the requested suggestion, so we expect that the model would be more effective at editing longer sentences. In Figure 4a, we see that the accepted suggestions are indeed generated from longer source sentences compared to the rejected
Novice	Skilled
Helpfulness	2.27*	3.23*
# request	3.04	2.64
% accepted	29.8	33.7
Table 8: Breakdown of model performance grouped by self-assessed writing skill. The rows correspond to average ratings of model helpfulness from the user survey, the average number of requests made to the model and the acceptance rate of received suggestions for both user groups. Rows marked with an asterisk indicates statistically significant differences ( $p$ -value  $< 0.05$  on a Mann-Whitney-Wilcoxon test). Bold values correspond to the higher score. Skilled writers find the model significantly more helpful, request fewer suggestions but accept a higher percentage of them.
ones. From Figure 4c, we also see that skilled writers tend to write longer sentences (which CRA is good at); this partially explains why skilled users find the model to be more helpful. Figure 4d also shows us that though skilled writers tend to write longer sentences, they request smaller fractions of these sentences to be rewritten. Examples 1 and 2 in Table 6 are representative of this scenario where the model provides helpful suggestions.
(d)
Figure 4: Analysis of interactions in terms of length of source sentences provided to the model (a, c) and rewritten spans in the generated text (b, d). In each boxplot, the box indicates the interquartile range with the median values marked by the red line. Length is measured in terms of number of characters. We see that the model is more effective when given longer source context sentences (a) and generating smaller rewritten spans of text in the target sentences (b). Skilled writers find the model to be more effective (Table 8) because they play to the model's strengths by writing longer context sentences (c) and requesting shorter spans to be rewritten in them (d).
Takeaways This finding of disproportionate assistance highlights a need for careful, user-centric study of interactive systems as they become more ubiquitous. Given that the use of technology could widen the gap in performance between different users, future direction to explore include developing models that assist both sets of users equally and also developing evaluation metrics that capture how performance varies across users.
Collaborative writing. Our work builds on existing literature on collaborative writing. Early approaches (Swanson and Gordon, 2012; Roemmele and Gordon, 2015) that provide text suggestions to users in the creative domain were retrieved-based. Creative Help (Roemmele and Gordon, 2015) retrieved sentence-level suggestions at locations specified by a user from a large corpus of stories. A follow-up study (Roemmele and Gordon, 2018) found that grammaticality and the presence of noun phrases in the text were indicative of helpful suggestions. We observe similar trends in Section 6 and Appendix E.2. More recently, collaborative systems have incorporated text generation models for assistance. Clark et al. (2018) evaluated a machine-in-the-loop setting on the tasks of story and slogan writing. They tested one system that generates sentence-level continuations for story writing and another one that generates a slogan from a given set of keywords, and found that solo-writing was a very competitive baseline. Akoury et al. (2020) gave human writers a machine-generated draft for storytelling and observed that writers tend to retain only a fraction of the generated text. Coenen et al. (2021) frames collaborative writing as a conversation between a human and a dialog system leveraging large language models. Our work is closest to Ito et al. (2020), which demonstrated that a collaborative rewriting system helps non-native English speakers revise drafts of research papers. We focus on the more challenging domain of creative writing where users are more selective of the suggestions they accept. In addition, we study how the assistant helps with the creating writing process in an end-to-end manner, whereas Ito et al. (2020) focus on editing a given draft.
Editing models. Transformer models have shown to be good at editing text to change the style (Shih et al., 2019; Krishna et al., 2020), bias text (Ma et al., 2020), post-edit translations
(Grangier and Auli, 2018; Wang et al., 2020) and simplify text (Kumar et al., 2020). Chakrabarty et al. (2021) train a model to generate metaphors employing a pseudo-parallel corpus of metaphoric sentences and corresponding literal sentences similar to how we use the sources of creative text. Additionally, infilling literature (Donahue et al., 2020; Fedus et al., 2018; Joshi et al., 2019; Shen et al., 2020) has shown that we can train models to fill in blanks. We incorporate editing models to collaborative writing which adapts to human feedback.
In this paper, we develop a Creative Rewriting Assistant that is able to effectively assist users to complete the task of creative image captioning. Our machine-in-the-loop rewriting setup allows human users to control the content of their writing while utilizing the strengths of text generation models. Our model is found to be more useful for skilled users, so it remains to be explored how to better assist novice writers. One direction is to explore generating text from keywords because these users might need help with planning the global content and structure of their writing. Additionally, the most common error mode we see amongst the rejected suggestions is content drift, so another challenge is to balance faithfulness to the author's content with creativity in text generation.
We would like to thank the anonymous reviewers for their helpful comments. This work is supported by the National Science Foundation under Grant No. 1922658 and the Samsung Advanced Institute of Technology (Next Generation Deep Learning: From Pattern Recognition to AI).
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6470–6484, Online. Association for Computational Linguistics.
Laura Ana Maria Bostan, Evgeny Kim, and Roman Klinger. 2020. GoodNewsEveryone: A corpus of news headlines annotated with emotions, semantic roles, and reader perception. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1554-1566, Marseille, France. European Language Resources Association.
Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021. MERMAID: Metaphor generation with symbolism and discriminative decoding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4250-4261, Online. Association for Computational Linguistics.
Jianfu Chen, Polina Kuznetsova, David Warren, and Yejin Choi. 2015. Déjà image-captions: A corpus of expressive descriptions in repetition. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 504-514, Denver, Colorado. Association for Computational Linguistics.
Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. 2018. Creative writing with a machine in the loop: Case studies on slogans and stories. In 23rd International Conference on Intelligent User Interfaces, pages 329-340.
Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. 2021. Wordcraft: a humanai collaborative editor for story writing. CoRR, abs/2107.07430.
Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. In *Association for Computational Linguistics (ACL)*.
William Fedus, Ian Goodfellow, and Andrew M. Dai. 2018. Maskgan: Better text generation via filling in the. In International Conference on Learning Representations (ICLR).
Monica J Garfield. 2008. Creativity support systems. In Handbook on Decision Support Systems 2, pages 745-758. Springer.
Jonathan Gordon, Jerry Hobbs, Jonathan May, Michael Mohler, Fabrizio Morbini, Bryan Rink, Marc Tomlinson, and Suzanne Wertheim. 2015. A corpus of rich metaphor annotation. In Proceedings of the Third
Workshop on Metaphor in NLP, pages 56-66, Denver, Colorado. Association for Computational Linguistics.
David Grangier and Michael Auli. 2018. QuickEdit: Editing text & translations by crossing words out. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 272-282, New Orleans, Louisiana. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.
Takumi Ito, Tatsuki Kuribayashi, Masatoshi Hidaka, Jun Suzuki, and Kentaro Inui. 2020. Langsmith: An interactive academic text revision system. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 216-226, Online. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Kalpesh Krishna, Josh Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Empirical Methods in Natural Language Processing.
Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020. Iterative edit-based unsupervised sentence simplification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7918-7928, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. 2020. Powertransformer: Unsupervised controllable revision for biased language correction. In EMNLP.
Saif Mohammad, Ekaterina Shutova, and Peter Turney. 2016. Metaphor as a medium for emotion: An empirical study. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pages 23-33, Berlin, Germany. Association for Computational Linguistics.
Michael Mohler, Marc T Tomlinson, and Bryan Rink. 2015. Cross-lingual semantic generalization for the detection of metaphor. Computational Linguistics and Intelligent Text Processing.
Vlad Niculae and Cristian Danescu-Niculescu-Mizil. 2014. Brighter than gold: Figurative language in user generated comparisons. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2008-2018, Doha, Qatar. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of NAACL-HLT* 2019: Demonstrations.
Melissa Roemmele and Andrew Gordon. 2018. Linguistic features of helpfulness in automated support for creative writing. In Proceedings of the First Workshop on Storytelling, pages 14-19, New Orleans, Louisiana. Association for Computational Linguistics.
Melissa Roemmele and Andrew S Gordon. 2015. Creative help: A story writing assistant. In International Conference on Interactive Digital Storytelling, pages 81-92. Springer.
Ben Samuel, Michael Mateas, and Noah Wardrip-Fruin. 2016. The design of writing buddy: a mixed-initiative approach towards computational story collaboration. In International Conference on Interactive Digital Storytelling, pages 388-396. Springer.
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5186-5198.
Yong-Siang Shih, Wei-Cheng Chang, and Yiming Yang. 2019. XL-Editor: Post-editing sentences with xlnet. arXiv preprint arXiv:1910.10479.
G.J. Steen, A.G. Dorst, J.B. Herrmann, A.A. Kaal, T. Krennmayr, and T. Pasma. 2010. A method for linguistic metaphor identification. From MIP to MIPVU. Number 14 in Converging Evidence in Language and Communication Research. John Benjamins.
Reid Swanson and Andrew S. Gordon. 2012. Say anything: Using textual case-based reasoning to enable open-domain interactive storytelling. ACM Trans. Interact. Intell. Syst., 2(3).
Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544-3552, Online. Association for Computational Linguistics.
Qian Wang, Jiajun Zhang, Lemao Liu, Guoping Huang, and Chengqing Zong. 2020. Touch editing: A flexible one-time interaction approach for translation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 1-11, Suzhou, China. Association for Computational Linguistics.
Disproportionate assistance. One of the findings of our work was that the collaboration model discussed is more effective at assisting users who are already skilled at writing tasks. We noted in the paper that an important direction of future work is to develop systems that cater to the novice user group as well. An ethical consideration is that if such a system in its current state were deployed, it could lead to an increase in the disparity in performance between the two user groups. We believe that recording this observation is important as human-centered machine learning systems become more prevalent.
Appropriate remuneration for crowd workers. To complete the HIT on AMT, workers need to interact with the model a minimum of 2 times before submitting the caption—it is explicitly mentioned that they are free to reject the suggestions and accepting/rejecting suggestions has no bearing on the payment. From a small internal pilot (also confirmed with Mechanical Turk experiments) we estimate an average completion time to be 10 minutes with an additional 2 minutes to read the instructions, so the payment is set to  $3 for the HIT (prorated to an hourly wage of$ 15). The estimated completion time for third-party evaluation was 1 minute so the payment was set to $0.25 per annotation (prorated to an hourly wage of $15).
B HIT Instructions and Details
Figure 5 is a screenshot of the interface presented to the crowdworkers for the writing task.
B.1 Instructions for crowdworkers completing the writing task
- Along with the first question in the survey is a link to the image captioning task. Navigate there. You will see a panel on the top left that shows you an image that you need to describe.
- You're free to interpret the image as you please—be as descriptive/figurative as possible.
- To help you with the same, we have a feature where you can highlight a part of your text with square brackets ('[', ']') and request targeted suggestions in that area. Please look at the accompanying examples on how to use it effectively.
- While writing we find that we are often able to provide content but to make the text more interesting is difficult, hopefully the assistant helps there. You will always have the option to reject the suggestions of the assistant and switch back to your original text. Bear in mind that the assistant isn't really great at guessing content words.
- To complete the task, continue editing until you are happy with the description. We require that you at least request suggestions from the assistant for a minimum of two times, even if you choose to reject the suggestions.
B.2 Instructions for crowdworkers evaluating the captions
- Choose the better (more descriptive and/or figurative) caption for the image.
- A better caption is your subjective judgement, the rubrics to make the choice are that the caption is descriptive and/or figurative in its interpretation of the image (Refer the examples for further clarification).
- The explanation asked is supposed to be very brief. A single word of if you like it for being descriptive or interpretive will do.
- Relevance of the caption to the image is your subjective choice whether the caption appropriately represents what is in the image and is not just a catchy piece of text unrelated to the image.
- A caption that you deem irrelevant should never be the better caption, unless both are irrelevant.
C User Feedback from Mechanical Turk
We present some user feedback obtained from the task—these cover some of the positive and negative comments we received. The negative comments are representative of some of the issues we highlight in section 5.4
- I was impressed by how well this worked. I feel like my writing did improve by using the suggestions. At the very least it gave me good ideas.
Figure 5: User interface. The user demarcates the span they want suggestions for in a text box and the model offers three suggestions for the user to pick from. This continues iteratively till the human is satisfied and submits the caption to finish the task.