SlowGuess/NAACL_2021_OCR huggingface dataset:
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China. Association for Computational Linguistics.
Naftali Tishby, Fernando C. Pereira, and William Bialek. 1999. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-377.
Antonio Torralba and Alexei A Efros. 2011. Unbiased look at dataset bias. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1521-1528. IEEE.
Alan M. Turing. 1950. Computing Machinery and Intelligence. Mind, LIX(236):433-460.
Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. Fill in the blanc: Human-free quality estimation of document summaries. arXiv preprint arXiv:2002.09836.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000-6010. Curran Associates Inc.
Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, et al. 2018. On evaluating and comparing open domain dialog systems. arXiv preprint arXiv:1801.03625.
Alex Wang and Kyunghyun Cho. 2019. Bert has a mouth, and it must speak: Bert as a markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3261-3275. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR.
Sida I Wang, Percy Liang, and Christopher D Manning. 2016. Learning language games through interaction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2368-2378.
Ludwig Wittgenstein. 1953. Philosophical Investigations. Wiley-Blackwell.
Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93-104, Brussels, Belgium. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019b. Defending against neural fake news. In Advances in Neural Information Processing Systems 32.
Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael Bernstein. 2019. Hype: A benchmark for human eye perceptual evaluation of generative models. In Advances in Neural Information Processing Systems, pages 3444-3456.
We provide the following items in the appendix:
- Dataset filtering criteria (Section A)
- Baseline model details (Section B)
- Computing statistical significance (Section C)
- Results from a different round of dynamic evaluation (Section D)
- Miscellaneous analysis (Section E)
Additional qualitative examples (Section F)
For more up-to-date information, visit the project page and dynamic leaderboard at rowanzellers.com/advice.
A Dataset Filtering Criteria
We discuss the criteria by which we extract situations and advice, both for our dynamic dataset REDDITADVICE, as well as for our static training dataset RedditAdvice2019.
A.1 Dynamic Filtering Criteria for RedditAdvice
We use the following selection criteria for retrieving situations, along with the top-scoring advice, from Reddit. Using the Reddit API, we will loop through Reddit posts, which might contain valid situations. We will perform several checks on the post, to ensure that we can reliably extract a situation from it, as well as a top-scoring piece of advice from the comments.
We do the following to retrieve situations:
a. We iterate through posts, which by sorting through the top posts, that were posted between 36 hours ago and two weeks ago, on the following advice subreddits: Relationships, Advice, NeedAdvice, Dating_Advice, Dating, Love, Marriage, InternetParents, TechSupport, and LegalAdvice.
b. We skip 'update' posts, in which a user refers to an older situation that they posted, and 'meta' posts, in which subreddit rules are discussed.
c. We skip any post that has an HTML link, since today's models (presumably) would not be able to visit such a link.
d. We skip any post with a score of less than 20.
e. We do our best to clean the text of the post. Many posts include valid situations, but are then edited to include updates that took place afterwards, in response to advice that was given. These are typically delimited by dashed lines, and the word EDIT or UPDATE.
Figure 7: Length distribution of REDDITADVICE, compared with other common NLU benchmarks benchmarks (HellaSWAG; Zellers et al. (2019a), GLUE; Wang et al. (2019b), SuperGlue; Wang et al. (2019a)). The examples in REDDITADVICE are significantly longer, representing highly complex situations.
f. Posts in some of the subreddits (Dating_Advice, Dating, Love, Marriage) is often in the form of tips and general suggestions, rather than situations. We skip any posts from these subreddits that do not include a question mark.
g. We filter out posts that contain sensitive topics, such as assault, suicide, and abuse.
h. Last, we skip any post that in total is fewer than 128 spaCy tokens, or, longer than 1280 spaCy tokens.
For a retrieved situation, we do the following to extract valid advice:
a. Given a post that contains a valid situation, we order the comments from highest-to-lowest scoring. We perform the following checks to determine if we can extract valid advice. Once we find valid advice, we will stop iterating.
b. We skip any comment that was posted by a moderator, the Reddit user who posted the original situation, or that was edited.
c. We skip any comment with a score of less than 20.
d. We skip any comment that contains fewer than 32 spaCy tokens.
e. One corner case is highly-scoring advice comments that refer implicitly to others. For instance, a comment might say 'You should listen to the other commenters and...' These references make sense inside a Reddit post, however, they are somewhat nonsensical when we pull the comment out of context. We thus skip any comment that seems to refer to others.
Once we retrieve a situation, that has at least one piece of valid advice, we are done - and we move on to the next situation. We loop over the top-scoring 1000 posts in total, and randomly select 200 valid situations from this pool.
A.2 Static Filtering Criteria for RedditAdvice2019
As mentioned in the main text of the paper, we used less stringent requirements to retrieve the static training dataset RedditAdvice2019. We did this because we hypothesize that today's neural generators are data-hungry: though we could retrieve the top-scoring situations and advice for each two-week span, this might not be enough to sufficiently train a model. Moreover, a single post (situation) on Reddit might have several comments that constitute reasonable advice.
We use the following static filtering criteria. For efficiency, we were able to retrieve all of the static training data from the PushShift Reddit dump that was posted before August 1, 2019.<sup>6</sup> We list the changes we make to the dynamic filtering criteria listed in Appendix A.1.
a. We use all posts that were posted to one of: Relationships, Advice, NeedAdvice, Dating_Advice, Dating, Love, Marriage, InternetParents, TechSupport, and LegalAdvice.
b. We skip 'meta' posts, but don't skip 'update' posts - since they perhaps might provide helpful signal to a model.
d. We only skip posts that have a score of less than 10, versus 20.
e. We don't bother to skip suggestion posts from the Dating_Advice, Dating, Love, and Marriage subreddits.
f. We don't filter out posts containing sensitive topics.
g. We skip overly short posts, but use a (less strict) minimum length of 64 characters. We do not skip overly long posts.
For RedditAdvice2019, we try to retrieve possibly multiple pieces of advice for each situation.
a. Again, we iterate through coments from highest-to-lowest scoring.
b. We allow for comments that were posted by anyone.
c. We skip any comment with a score of less than 10, or, any comment with a score of less than 1/10th that of the top-scoring advice comment. This ensures that we are retrieving advice that the community judged as almost as good as the reference advice.
d. We skip short comments using the (less strict) minimum length of 64 characters, versus 32
spaCy tokens.
e. We don't skip comments that refer to others.
By optimizing for recall, we are able to extract a large training dataset. In total, we retrieve 616k comments over 188k posts. The posts range from July 2009 to August 2019.
B Baseline model details
In this section, we provide details about how we set up our baseline models for advice generation.
B.1 Input format
A Reddit situation-advice pair is a collection of several fields:
i. The subreddit where the situation was posted, ii. The date on which it was posted,
iii. The title of the situation post,
iv. The body of the situation post,
v. The advice posted in response to the situation.
We adapt Grover in this setting by giving the model all of these fields in the given order (from i-v). Similar to how the model was pretrained, we include a field-specific start and end-token in each field, which allows the model to generate advice conditioned on the other fields.
In T5, the authors handle diverse tasks by preponding each field with its name (like Situation:) and concatenating the resulting fields. We do the same here. We place the context fields i-iv in the bidirectional encoder, and the target field (advice) is generated by the left-to-right decoder.
For the retrieval model, we combine the context fields (i-iv) into the same TF-IDF bag-of-words representation.
B.2 Length adaptation
As shown in Figure 7, our task contains lengths that are much longer than what has usually been explored in prior NLU work. For comparison, Grover (Zellers et al., 2019b) was trained on shorter texts (up to 1024 tokens) with absolute position embeddings. We thus pretrained Grover for  $20\mathrm{k}$  additional steps on three million news articles, using a new maximum length of 1536. We then finetuned Grover on REDiTADVICE using a sequence length of 1536. We hypothesized that this extra step might be unnecessary for T5, as it uses relative position embeddings (Shaw et al., 2018) and has separate Transformer stacks for the encoder and the decoder. We finetuned T5 on REDiTADVICE, using a context length of 1280 and a target length of 512.
Available at https://pushshift.io/.
Hyperparameter	Grover-Large	Grover-Mega	T5-3B	T5-11B
Learning Rate	{2.5e-6, 5e-6}	{5e-6,1e-5}	{1e-4, 2e-4, 4e-4}	{1e-4, 2e-4, 4e-4}
Epochs	{10, 20}	{10, 20}	{2, 4, 6, 8, 10}	{2, 4, 6, 8, 10}
Batch Size	512	512	128	128
TPU training hardware	v3-512	v3-512	v3-512	v3-1024
Training runtime	110 minutes	448 minutes	108 minutes	72 minutes
Perplexity	14.73	12.56	11.248	10.74
Table 1: Finetuning details for Grover and T5. We show both the largest models from each family (Grover-Mega and T5-11B) as well as their smaller variants; Grover-Large with 0.3 billion parameters and T5-3B with 3 billion parameters. The different values used for grid search are shown in curly braces. The best value, as measured by perplexity on the validation set (also shown) is bolded. We also show the runtime of training the best model.
Nevertheless, in  $6\%$  of cases, contexts are still too long. If this happens, we divide contexts into paragraphs and trim the middle ones, as often the first and last paragraphs contain important information (such as a summary or a question).
B.3 Training generative models
We finetune our learned models using a cross-entropy loss. We trained Grover to predict all fields, whereas we only trained T5 to predict the advice field (v), as the context is bidirectional.
We optimized our models using AdaFactor (Shazeer and Stern, 2018). We validated the number of epochs and the learning rate using a small grid search over the validation set. We kept other hyperparameters to be the same as how the models were originally pretrained. For Grover-Large, we finetuned for 20 epochs with a learning rate of 1e-5 and batch size 512; for Grover-Mega, we finetuned for 20 epochs with a learning rate of 5e-6 and batch size 512; for T5-3B, we finetuned for 10 epochs with a learning rate of 2e-3 and batch size 128; for T5-11B, we finetuned for 5 epochs with a learning rate of 1e-3 and batch size 128. We trained our models on v3-512 TPU pods, except for T5-11B, which was trained on a v3-1024 TPU pod. A full list of hyperparameters considered is shown in Table 1. Note that Grover and T5 use two slightly different implementations of AdaFactor, which we left unchanged from their public repositories; we thus found success using larger values for T5's learning rate (versus what might otherwise be expected for a larger model).
B.4 Generation through Nucleus Sampling
For open-ended generation tasks, such as ours, past work has shown that straightforward sampling - along with maximization approaches like beam search - tend to result in degenerate text (Holtzman et al., 2020). In our work, we use Nucleus Sampling (Holtzman et al., 2020) to limit the variance of generated text. We use a threshold of  $p = .95$ , meaning that at each timestep we only sample from the most probable  $95\%$  of the distribution.
B.5 Prompting GPT3
GPT3 is a 175-billion parameter transformer model that is only accessible through a web demo – it cannot be finetuned. Instead, the strategy used by the GPT3 authors (Brown et al., 2020) is to “prompt” it through a combination of natural language instructions and few-shot examples. The model is trained on contexts up to 2048 BPE tokens, so all of the few-shot training examples and the test example in question must fit within that window. For example, for the commonsense NLI dataset HellaSwag (whose length distribution is shown in Figure 7), the GPT3 authors fit 20 few-shot training examples within that window; few examples have more than 100 tokens.
However, as Figure 7 also shows, situation-advice pairs in REDDITAdvice are long and complex – so much so that few-shot or even one-shot learning is hardly an option (unless we wish to only seed it on the shortest situation-advice pairs). Instead, we focused on designing an effective prompt. Our prompt is the following:
On reddit.com/{FAKEID}/, [deleted] submitted:
{SITUATIONTITLE}
{SITUATIONBODY}
Chuckflowers22 commented (533 points):
For 'FAKEID' we created a realistic-looking
The finetuning over the context fields i-iv is not necessary, as we never must generate those fields at test time. However, we opted to finetune on them anyways in order to provide more signal during training. We scaled the loss on the context fields to be 1/10th as much, to encourage the model to primarily learn how to generate advice.
$\%$  Frequency that model advice is preferred over best Reddit advice
Figure 8: Evaluation results on advice from February 1st to 12th. This evaluation was done with a slightly different set of Turkers than in Figure 3, but with much of the same results - T5-11B outperforms smaller models but significantly underperforms human performance. We also compare against the smaller Grover-Large (0.3B parameter) and T5-3B models; they are outperformed by their larger counterparts.
reddit link, encoding the subreddit that the situation was posted on and a rough sketch of the situation title. For the example in Figure 5, this would look like r/legaladvice/comments/
j全流程的使用，包括：
We added a username so that the prompt would look more like it was quoted verbatim from Reddit (and with a high score of 533 points). We chose that username after performing a Google search for 'best advice giver on reddit' – the second Google result had the user winning the award for 'Best Advice Giver of the Month' (for fashion advice).
Given this prompt, we had GPT3 generate text until it generated  $\sim$ . Still, GPT3 would frequently add in web formatting artifacts (such as fake buttons like 'Submit Reply Delete'), of which we were able to automatically remove many thorough regular expressions. Even then, GPT3 would still often generate the empty string. We addressed this by manually inspecting all GPT3-written advice, having it regenerate advice where it had previously generated nothing.
C Measuring Statistical Significance
Here, we describe how we compute statistical significance for Figure 4. For measuring statistical significance, we use a continuous version of the advice preference. The machine advice gets 1.0 points from a worker if it is chosen as Definitely more helpful, and 0.5 points if it is Slightly more helpful. We use point values
Figure 9: Improvement (in absolute percentage %) between pairs of models for the February 1st to 12th evaluation, along with statistical significance as measured by a paired t-test. The improvement of very large models over much smaller ones is highly significant, such as T5-11B over Grover-Mega (5% gap,  $p < .01$ ).
of  $-1.0$  and  $-0.5$  for advice that is rated as Definitely less helpful and Slightly less helpful, respectively. For a single piece of advice, we average together the point values for all workers that agreed with the majority vote.
For example, suppose for a single pair that Worker 1 and 2 prefer human-written advice, and Worker 3 prefers the machine-written advice. We only use the responses from Worker 1 and 2, to agree with the majority vote. If Worker 1 rates the machine-written advice as Definitely less helpful, and Worker 2 as Slightly less helpful, then the score of the machine advice is
$$
\frac {(- 0 . 5) + (- 1 . 0)}{2} = - 0. 7 5.
$$
We can then use these scores to compare two different machines, using a paired t-test.
D A Different Round of Dynamic Evaluation
In this paper, we reported results from one round of dynamic evaluation, with advice from October 28 to November 7th 2020. During the earlier stages of this paper, we also conducted a round of evaluation with advice from February 1st to February 12th 2020. Results are in Figures 8 and 9.
The results show much of the same patterns, however there are some slightly different hypotheses being tested. During the February 2020 round, we explicitly compared smaller models (such as
Grover-Large with 0.3 billion parameters) with their larger counterparts (e.g. Grover-Mega). We found that larger models tended to be preferred more often, however, the gap between models like T5-11B and T5-3B is relatively small (and in this case not statistically significant). All of these models still underperform humans significantly.
One notable difference, however, is that during this round, T5-11B's performance is  $9.0\%$  as opposed to  $14.5\%$ . Though this might be seen as a large relative improvement, we note that both of the T5 results have overlapping error bars. In other words, while our evaluation with 200 shared situations shows great power in determining that models underperform humans at advice-giving, and that some baselines are stronger than others (e.g. T5 outperforms TF-IDF retrieval), its power might be more limited at discriminating between models in the 5 to  $15\%$  range.
One reason that might explain this difference is that during the October to November round, we used slightly looser Mechanical Turk filtering criteria. We used 31 workers during this round, as opposed to 22 in the previous round, and used a looser qualification score cutoff. We suspect that this resulted in more random error than before. Conversely, we suspect that had we increased the number of workers per advice-situation pair from 3 to 5 or 7, all models would drop in performance – the aggregate wisdom of the crowd might smooth out some of the random error.
E Miscellaneous analysis
E.1 Workers
We plot the number of annotations done per Mechanical Turk worker in Figure 10, for the October 28th to November 7th 2020 evaluation. Overall, 31 workers participated in our evaluation, though this number also includes workers who completed very few HITs. We also show the distribution for the February 1st to 12th evaluation in Figure 11; this shows a sharper distribution with two workers annotating over  $10\%$  of the dataset.
E.2 Are some domains harder than others?
One question might be whether some advice domains are inherently more challenging than others. We present results in Figure 12 that do not seem to suggest a clear pattern of this. Over all advice domains, we see the same trend of human performance being high, and machine performance being
low. Interestingly, models seem to perform best on 'Life' advice, however this is perhaps there are not many 'Life' situations in this evaluation round.
Though the error bars suggest some uncertainty, we find that models are especially poor at generating Legal advice – a phenomenon we also observed in the February 1st to 12th evaluation. This result might be somewhat surprising, as Mechanical Turk workers are (probably) not lawyers, but are still able to reliably spot model-written legal nonsense.
F Additional qualitative analysis
In this section, we provide additional qualitative examples. First, we show two examples with generations from all models in Figure 13 and Figure 14 respectively.
F.1 Natural language understanding errors
In Figures 15, 16, and 17, we categorize problems with T5-11B's machine-written advice under the framework of other core NLP tasks. Figure 15 is an unabridged version of the teaser figure.
The generated advice has key issues that fall under the purview of many language tasks, as seen broadly:
a. Natural Language Inference (e.g. Dagan et al., 2006; Bowman et al., 2015): whether a passage entails or contradicts another (or, neither). Generated advice often contradicts the provided situation, or even itself.
b. Reading Comprehension (e.g. Rajpurkar et al., 2016): Read and understand a passage (possibly, to be able to answer questions). Good advice requires us to first understand the situation at hand.
c. Coreference Resolution (e.g. Pradhan et al., 2012): Identify repeated entities in a document. Good advice requires us to identify who is who in a document, and not to mix people up.
d. Social Commonsense Reasoning (e.g. Sap et al., 2019): Identify people's intentions, feelings, and motivations in social interactions. Many of these situations are inherently social, so good advice often requires reasoning about social situations.
e. Physical Commonsense Reasoning (e.g. Zellers et al., 2018; Bisk et al., 2020): Have some notion of intuitive physics, and apply it to new situations. Many of these situations relate to physical situations, so writing good advice requires some physical commonsense
Figure 10: Distribution of the number of annotations for each worker in the Mechanical Turk evaluation from October 28th to November 7th 2020.