SlowGuess/NAACL_2022_OCR huggingface dataset:
- I got great suggestions that offered me words that I hadn't considered and fit even better than my own writing so I was pleased with the suggestions.
- I think everything was clear and straightforward and I enjoyed the interface.
- The suggestions were sometimes too far from the meaning of the original text so that they no longer made sense or were not grammatically correct.
- The instructions were fine, but the suggestions sure leave a lot to be desired. It replaced 'bright yellow' with red a couple of times.
D Details for Reproducibility Checklist
We use a pre-trained BART-Large (406M parameters) model as the starting point for our experiments, which was made available through the fairseq (Ott et al., 2019) implementation. Unless mentioned otherwise, the recommended values for the hyperparameters were taken from the
released fine tuning script in the library. We selected the learning rate for Section 5.1 using validation perplexity as a metric varying the value from  $1 \times 10^{-5}$  to  $1 \times 10^{-4}$ . The source code for our experiments, both to set up the interface and train the model, will be made available upon publication of this work. Model training was on a Titan Xp single GPU machine with 12GB of memory. The same machine was also used to host the server for the interactive experiments. A model inference is made for each request from the users.
D.2 Data Details
All the datasets from Table 1 are publicly available already. As highlighted in Section 6, one reason our model suffers from content drift is because the creation process does not guarantee that the content in the source and target is identical. So prior to making the pseudo-parallel corpus from Section 3 available, we aim to filter out those examples which suffer from content drift. The dataset of interactions from Section 3.2 cannot be directly shared.
Longer model rewrites get rejected more frequently. Our assumption is that users want to
control the content of the caption. When the model rewrites a longer span and adds more new text to the draft, it is likely to diverge from the original content given by the user. We compare the length of new text introduced into the draft by CRA in both the accepted and rejected suggestions. From Figure 4b, we see that longer revisions are more likely to be rejected.
E.1 Collaborative vs Human Writing
In Section 5.3, we saw that humans writing in a collaborative setup tend to produce better creative output. To analyze how model intervention affects the text, we collected some statistics on the 100 final captions produced from the Human+CRA setup and the Human-Only setup. We see that users in the collaborative setup write longer captions (Figure 6a) that tend to have more unique n-grams (Figure 3), indicative that users are incorporating more diverse elements into their text as a result of model interaction. We also calculate the perplexity of the final captions using a pre-trained GPT2 model. From Figure 6c, we see that the captions with CRA intervention have a lower average perplexity despite having higher lexical diversity. Also we see that the perplexity scores from the collaborative setup have significantly less variance than the human-only captions indicating that collaboration makes different people's writing more similar to each other.
E.2 POS Tags
To examine the kind of text that is helpful to users, we analyze the linguistic characteristics of accepted suggestions and rejected suggestions.
Accepted suggestions have more adjectives, adverbs and nouns. Figure 7 shows the fraction of different POS tags in the revised span of accepted suggestions and rejected suggestions. Accepted suggestions tend to have a larger fraction of adverbs, adjectives and nouns, whereas rejected suggestions have a large fraction of determiners. Prior work (Roemmele and Gordon, 2018) also observed that the presence of noun phrases in suggestions has a positive correlation with helpfulness.
Figure 6: Comparison of text generated from a collaborative setup (Human+CRA) and solo-writers (Human-Only). We see that collaborative users tend to write longer captions (Figure 6a), that contain more unique N-grams (Figure 6b,  $N = 3$ ), and on average have a lower perplexity (Figure 6c), as evaluated using a pre-trained GPT2 model. We use perplexity as a proxy for fluency in text. Collaborative users tend to consider more diverse options for text while retaining fluency in the text.
(a) POS tags of rewritten text for all accepted suggestions.
(b) POS tags of rewritten text for all rejected suggestions.
Figure 7: The 10 most common POS tags in accepted and rejected suggestions: Accepted suggestions tend to have more adjectives, adverbs and nouns and rejected suggestions tend to have higher fraction of determiners