Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest
Year: 2023

Abstract:
Large neural networks can now generate jokes, but do they really “understand” humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest: matching a joke to a cartoon, identifying a winning caption, and explaining why a winning caption is funny. These tasks encapsulate progressively more sophisticated aspects of “understanding” a cartoon; key elements are the complex, often surprising relationships between images and captions and the frequent inclusion of indirect and playful allusions to human experience and culture. We investigate both multimodal and language-only models: the former are challenged with the cartoon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored explanations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the image’s locations/entities, what’s unusual in the scene, and an explanation of the joke.

heading: 1 Introduction
text: Humor can be dissected, as a frog can, but the thing dies in the process and the innards are discouraging to any but the pure scientific mind.
– White, E. B. (1941) Each week, The New Yorker publishes a uncaptioned cartoon image, inviting readers to submit their funniest English-language caption for it. Editors choose three finalists from sometimes thousands of submissions; then, readers vote to pick
the final winner. We develop a suite of three progressively harder tasks built around this contest to test how well AI models “understand” humor across vision and language: 1) matching jokes to cartoons, 2) identifying a winning caption, and 3) generating an explanation of why an image/caption combination is funny.
These tasks are difficult because the connection between a winning caption and image can be quite subtle, and the caption can make playful allusions to human experience, culture, and imagination. Consider the image and winning caption “Can you please pass the cow?” in Figure 1. Unlike literal image captions such as in MSCOCO (Lin et al., 2014), here, the caption’s relation to the image is indirect:1 the size of the mugs must first be recognized as unusual, and then, the caption invokes
1The (relatable) experience of “not getting” a New Yorker cartoon often results from inability to identify the image/text relationship.
688
an association between a large mug and a large amount of cream/milk — perhaps a whole cow’s worth. Further, matching a caption to an image is not sufficient: non-finalist entries (e.g., “...Insomniacs Anonymous” in Figure 1) also match the image, but something else makes one seem funnier than the other. Finally, even if a model can accurately identify winning submissions, we would like it to also be able to explain why a particular highly rated/relevant caption is funny.
We cover our three tasks in two settings: in the from pixels setting, models are given access only to the cartoon images at test time, and must perform computer vision; in the from description setting, we allow models access to a newly-collected, humanauthored corpus of cartoon descriptions, thus simulating access to a human-level computer-vision system — or, alternately, facilitating benchmarking of models that don’t have a built-in image-processing component. The annotations we collect and release are rich and multifaceted: they describe the image overall and its locations and entities, what’s unusual about the image, and an explanation of the joke. We view this effort as a significant contribution of our work.
Our results reveal a gap between AI and humanlevel humor “understanding.” In the from pixels setting, our best multimodal model (fine-tuned CLIP ViT-L/14 (Radford et al., 2021)) achieves 62% accuracy on a 5-way multiple choice task, but humans achieve 94% in the same setting. Even with significant manual annotation of the cartoons in the from description setting (and despite significant
improvements in language modeling performance since this work’s submission2) large language models still fall short: human explanations are still preferred in more than two-thirds of cases compared to our best explanation model, 5-shot GPT-4.
We release our challenging NLP/vision benchmarks,3 annotations, models, leaderboard, and code at https://capcon.dev/. Beyond AI research, we also hope that our work will spur progress in human-AI collaboration tools for cartoonists, contest entrants, and beyond (see Appendix G for AIgenerated captions).

heading: 2 Datasets and Task Setups
text: Our corpus compiles 14 years of weekly New Yorker caption contests. Each contest consists of: (1) a captionless cartoon; (2) that week’s entries; (3) the three finalists, selected by New Yorker editors; and (4) for some contests, quality estimates for each submission collected via crowdsourcing.4
The corpus was constructed from two sources. The first is Jain et al. (2020), from which we obtain roughly 250 contests (mean/median 6.1K/5.7K unique captions per contest; 1.5M total), starting from #508.5 Crowd ratings in this corpus are gath-
2GPT-3 (Brown et al., 2020) was the most performant in Jan. 2023 when this work was submitted, but we have since updated our results.
3Our data may contain offensive jokes. We manually removed a handful of cases we observed to target specific protected classes. We do not endorse the jokes in the corpus, but rather, view them as interesting objects of study.
4We regret that The New Yorker does not currently have an alliterative-paragraph contest.
5We manually corrected some errors in the corpus.
ered via the NEXT platform (Jamieson et al., 2015; Tanczos et al., 2017), where readers rate captions as “funny”, “somewhat funny”, or “unfunny”; we use the per-caption mean. There are over 114M ratings total (mean/median of 445K/471K per contest). We also sample three additional top captions that aren’t editorial picks to serve as additional “finalists.”
The second corpus, due to Shahaf et al. (2015); Radev et al. (2016) and derived from contests #1– #507, includes 2M unique captions (mean/median 5.2K/5.0K per contest), but no crowd ratings. We remove by hand 55 contests whose images’ resolutions are too low, and identify 80 low resolution (but usable) cases, taking special care when annotating this set (§2.2).

heading: 2.1 Task Setups
text: We pose three tasks. Matching and explanation are novel, whereas quality ranking extends the formulations introduced in Shahaf et al. (2015); Radev et al. (2016).
Matching. Can a model recognize when a caption is appropriate for a given cartoon? Five choices are given, only one of which truly corresponds. For the example in Figure 1, we supply the following possibilities: (a) O.K. I’m at the window. To the right? Your
right or my right? (b) I’d kill for some cream cheese. (c) Bob just came directly from work. (d) Can you please pass the cow? (e) They only allow one carry-on.
The correct caption is a finalist for the cartoon. Negative choices are randomly selected finalists from other contests, and as a result, are great captions for some other contest’s image.6 In some cases, matching depicted objects to their textual references may
6Distractors are balanced across instances so that a model that only examines the answer choices cannot achieve better than chance accuracy.
suffice, but in other cases, the relationship is more indirect. For example, Figure 2 (top) contains a subtle reference to Jane Goodall, thus requiring external knowledge; Figure 2 (bottom) relies on a stereotype of pharmaceutical companies being untrustworthy, hence requiring reasoning beyond the literal text.
Quality ranking. Can a model identify highly rated captions? For each finalist, we sample for comparison a caption that was not selected as a finalist, and ask models to identify which one (the real one or the distractor) was rated as higher quality. As preprocessing, we run one round of textonly filtering to discard submissions that are easily identifiable as low quality, and also perform semantic deduplication; more details in Appendix C. Here is the end result for Figure 1:
(a) Can you please pass the cow? (b) Welcome to Insomniacs Anonymous.
Which caption a particular individual prefers can be a matter of personal taste; but there is a general preference among our human annotators for the true finalist (see §3).
Explanation. Can a model generate as good an explanation as a human for why a caption-andimage combination is funny? Free-form explanations of why captions are funny/appropriate for their corresponding image were written by an author of this paper.7 The rough annotation guidance was: “In a few sentences, explain the joke as if to a friend who doesn’t ‘get it’ yet.” Starting from a random finalist for each contest, after filtering out cases where the author did not understand the joke, a corpus of 651 human-created joke explanations to serve as comparison points was formed (mean/median 60/59 words, 39.3K total). We consider a model to succeed at this task if human judges, presented with (unlabeled) pairs of author/machine-generated explanations, do not show a preference for the author-generated ones.
Evaluation metrics. For matching and quality ranking, we evaluate using accuracy. For quality ranking, we report NYAcc — the average accuracy over instances where the finalist was an official New Yorker finalist — and CrowdAcc, where the
7Several attempts to solicit explanations from crowdworkers were not satisfactory; similarly unsuccessful were prompting experiments with GPT-3 inspired by Wiegreffe et al. (2022); Marasović et al. (2022) — too few of the sampled explanations were correct to bootstrap a corpus.
“finalist” caption was selected by the crowd as high quality. These two measures allow us to account for different audience tastes. For explanation, we conduct pairwise human evaluations to test several hypotheses detailed in §3.2. To complement these human evaluations, we also report in Appendix E automatic metrics that take into account the human-written reference: (a) BLEU-4 (Papineni et al., 2002) using Post (2018)+ROUGE-L (Lin, 2004); and (b) word-level perplexity.
From Pixels + From Description. We consider two experimental settings. In From Pixels (FP), a vision+language model undertakes image processing, i.e., at test time, the only contest information available is the image itself. In the second setting, which we call From Description (FD), we factor out visual processing by providing the model with human written annotations, described in §2.2. FD models thus simulate access to a human-level computer-vision system.

heading: 2.2 Annotation of cartoons.
text: We collect several types of annotations about the 704 cartoons; these either serve as input to models in the from description setting, or as additional information available only at training time in the from pixels setting. For each cartoon, we gather:
(i) A phrase describing the setting of the scene, e.g., “an office” or “the park” (2 per cartoon) (ii) A literal 1-3 sentence description of the scene (3 per cartoon) (iii) A 1-3 sentence description or explanation of what makes the scene unusual (3 per cartoon)
(iv) 2-3 English Wikipedia links that an annotator identified as relevant, to serve as a proxy for world knowledge (2 per cartoon)
A random sample of annotations is shown in Figure 3. We used Amazon Mechanical Turk, and paid crowdworkers a minimum of $15/hr. Lowresolution images involved special treatment: 1) we offered additional pay to crowdworkers; and 2) at least one of the annotations is conducted by an author of this work using the same HIT interface. Details including qualification rounds, screenshots of the HITs, etc. are given in Appendix A.

heading: 3 Experiments
text: We split the 704 cartoons into 5 cross-validation splits such that entire contests are held out at test time. Task construction details are in Appendix C; modeling details (e.g., hyperparameter sweeps, task formatting) are in Appendix B.
From Pixels (FP) Models We explore two vision+language models.
CLIP. We fine-tune CLIP ViT-L/14@366px (Radford et al., 2021) (428M parameters), which consists of a text Transformer (Vaswani et al., 2017) and a vision Transformer (Dosovitskiy et al., 2021) pretrained to align images/captions in the WebImageText corpus (400M pairs). For multiple choice, we use InfoNCE (Oord et al., 2018) to encourage the cosine similarity of the cartoon/correct answer to be higher than the incorrect ones. For zero-shot classification, we use the prompt a new yorker cartoon with
winning caption. CLIP isn’t generative, so we can’t use it for explanation.
OFA→ LM. We use OFA Huge (930M parameters) (Wang et al., 2022), a seq2seq model that supports image/text inputs/outputs; it is pretrained on a variety of vision+language tasks. We finetune on the New Yorker corpus by training it to map from (cartoon, prompt)→ descriptions for the four types of annotations described in §2.2; see Figure 4 for example predictions. We organize the OFA-predicted outputs in the same format as the human-authored descriptions in our From Description (FD) models detailed below (except the inputs are the outputs of OFA), and pass the result to a language model:8 this composition can be considered a Socratic Model (Zeng et al., 2022).
From Description (FD) Models We formulate multiple-choice tasks as text-to-text by concatenating the human-authored cartoon descriptions with the choices as input: the target is simply the letter corresponding to the answer, e.g., E. For explanation, we autoregressively generate the explanations conditioned on the descriptions/captions.
T5. We fine-tune T5-Large and T5-11B (Raffel et al., 2020); these encoder-decoder transformer models have 770M and 11.3B parameters respectively. For explanation, we sample with tempera-
8We found that fine-tuning OFA directly was less effective.
ture 1.0 and nucleus sampling with p=.95 (Holtzman et al., 2020).
GPT-3, GPT-3.5, GPT-4. We use these three OpenAI models as both zero-shot and few-shot models. We provide the models with a description of the task, and, for the few-shot case, 5 random labelled in-context examples. Specifically, for GPT-3 we use text-davinci-002 (175B) (Brown et al., 2020), and for GPT-3.5/GPT-4, we use the May 12, 2023 versions (OpenAI, 2023). For GPT-3, we also consider a fine-tuned version (which is unavailable for GPT3.5/GPT-4).9 For zero-shot GPT-3.5/GPT-4, early experiments revealed that prompting models to “think” step-bystep with chain-of-thought (CoT) was helpful (Wei et al., 2022; Kojima et al., 2022). See §B.6 for GPT3 details, and §B.7 for GPT-3.5/GPT-4 details.
Baselines Caption Only. In addition to a Random-guess baseline, we fine-tune T5-11B given just the caption, i.e., without knowledge of the cartoon (Trichelair et al., 2019; Poliak et al., 2018).
Human performance estimates. Three people (two authors and one person familiar with the project) each attempted 100 randomly sampled instances from both the matching and quality ranking tasks.10 It is important to note that human performance is not an upper bound for model performance on matching and quality ranking because labels are not generated by a single human and tastes can vary; it can (and does, see §3.1) happen that a machine might be able to reconstruct New Yorker editor preferences more reliably than an untrained human. Annotators were given access to the images, but not the descriptions (akin to the FP setting).
Hardware+software details. T5, CLIP, and OFA were trained using 8 A100 GPUs in pytorch (Paszke et al., 2019). We use the Transformers (Wolf et al., 2020) implementation of T5: T5-11B was trained with deepspeed (Rasley
9https://beta.openai.com/docs/guides/fine-tuning; for explanation, we use the default settings; for multiple choice, we set prompt loss weight to zero. The validation set is not used by the API for early stopping, so we concatenate it with the training set and perform no validation.
10Matching instances were sampled such that there were no repeated options, i.e., annotators couldn’t use process of elimination across instances. 595 total responses were collected.
et al., 2020); T5-Large and CLIP were trained with Accelerate.11

heading: 3.1 Matching and quality ranking results
text: Table 2 contains the results. Among the from description models, GPT-4 (5-shot) generally performs best, e.g., achieving 84.5% accuracy on matching. It (and fine-tuned GPT-3) also perform better at predicting New Yorker editor selections than our three humans (column NYAcc: GPT-3 69.8 vs. Human estimate, 64.6), but underperform at predicting crowd selections (CrowdAcc column: GPT-4 73.3 vs. 83.7).12 We also see that our from pixels models leave significant headroom compared to the human performance estimates.
Other observations include: 1) both from pixels and from description models mostly outperform the Caption Only baseline (even for smaller model sizes), suggesting that the models are truly using feature interactions between cartoons/captions to improve their predictive accuracy; 2) fine-tuning CLIP tends to do best for matching in the from pixels setting, but OFA+T5-11B is competitive for quality ranking (and supports generation, see §3.2); and 3) the performance difference between T5 vs. OFA→T5 exemplifies the effect of subop-
11https://huggingface.co/docs/accelerate 12Also, crowd selectors greatly outnumber New Yorker editors, so crowd rankings may be a more dependable target, statistically speaking.
timal visual recognition when shifting from the from pixels setting to the from description setting. Finally, while performance drops are incurred universally for zero-shot models, pointing towards the utility of the new annotated corpus we are releasing (§2.2), GPT-4’s zero-shot chain-of-thought incurs a smaller performance drop compared to other zero-shot models; see §B.7 for a sample chain-ofthought.

heading: 3.2 Human evaluation of explanation.
text: We gather judgments from 3 crowd-workers per test instance by asking them which of a pair of explanations they prefer, and take a majority vote to determine a winner. Results and annotator agreement are in Table 3, and samples of GPT-3, GPT-4, and human joke explanations are in Figure 5. Our evaluations address seven questions:
Q1: Do models utilize the image context of the caption to generate better explanations? Test: T5-11B vs. Caption-only T5-11B. Answer: Yes. Compared to the same model trained with no access to image information, the model with image information wins in 84.7% of cases.
Q2: Is computer vision a bottleneck for topquality explanation generation? Test: T5-11B (in the FD setting) vs. OFA → T5-11B. Answer: Yes. Compared to the same model trained with access to human written descriptions available at test
time (i.e., the from description setting), the model trained with access only to OFA-predictions loses in 74.6% of cases.
Q3: Do bigger T5 models generate better explanations? Test: T5-11B vs. T5-Large. Answer: Yes. T5-11B with access to the same information at test time as T5-Large (770M) is preferred in 68.5% of cases.

heading: Q4: Does fine-tuning an LLM model help vs. in-context learning for explanation generation?
text: Test: FT-GPT3 vs. In context (=5-shot) GPT3. Answer: Not really. In contrast to the multiple choice tasks, we find that in-context explanation genera-
tions are comparable to fine-tuned ones according to pairwise human evaluations, even though the perplexity of the in-context model, reported in Appendix E, is much higher (107 vs. 21.8).13 We expect that the fine-tuned model more closely mirrors the style of the corpus, but that the in-context explanations also contain similar content, e.g., relevant entities.
Q5: Do supervised explanations help, even with GPT-4? Test: 5-shot GPT-4 vs. Zero-shot GPT-4. Answer: Yes. The zero-shot version of GPT-4 is missing access not only to the supervision of paired (caption, explanation) data, but also, explanations in the detailed style of our released corpus. Perhaps as a result, 5-shot GPT-4 (which also achieves significantly higher BLEU-4/Rouge-L) is preferred in 64% of cases.
Q6: Does GPT-4 outperform GPT-3? Test: 5- shot GPT-4 vs. 5-shot GPT-3. Answer: Yes, definitely. In our most definitive result, with equal amounts of supervision, GPT-4’s explanations are preferred nearly universally — specifically, in 93% of cases. Interestingly, GPT-3 performs slightly
13A disparity not mirrored in the word-overlap metrics BLEU-4 and Rouge-L, also reported in Appendix E.
better on automatic evaluation metrics for explanation like BLEU-4 and Rouge-L (see Appendix E), which suggest that the earlier family of may fit the surface features of the generation task more effectively, e.g., 5-shot GPT-3 achieves 5.07 BLEU-4 compared to 4.99 for 5-shot GPT-4. This suggests that mirroring the surface form of our explanation corpus is not sufficient to generate the highest quality explanations.
Q7: Does our best model, GPT-4, explain jokes as well as humans? Test: Human vs. Few-shot GPT-4. Answer: No. Human-written explanations are preferred by annotators in 68% of pairwise cases.14 We qualitatively examine the 39/130 cases where the human reference receives 3/3 annotator votes. In these cases, the machine-generated explanations usually incorrectly interpret the image, e.g., in one case, a caption jokes about two cavepeople in a hole looking at a caveman in a cave with the caption “Personally, I’m not a big fan of modern architecture.”; GPT-4 incorrectly interprets the hole as “modern architecture” instead of the cave. We also examine the 8/130 cases where the GPT-4 produced caption was unanimously preferred: a close reading of these cases is provided in Appendix F. In 3 of these 8 cases, the human explanations, while on the right track, had slight inaccuracies, and in the remaining 5 cases, the human and machine explanations both express the same idea, but with different styles (GPT-4’s sometimes arguably being more formal, detailed, or fluent).

heading: 3.3 Error Analysis for Matching
text: We conduct an error analysis of a performant from pixels model (CLIP ViT-L/14@336px finetuned), and a performant from description model (GPT3-175B finetuned). We concatenate the test set predictions over the 5 cross validation splits, and ask:
Q8: Are some contests more difficult than others? Answer: Yes. Details: We conduct a χ2 test by forming a contest-by-correctness (704-by-2) contingency table, aggregating over the 3-6 matching instances for each contest, and find that errors are clustered according to contest (p < .05 for both CLIP and GPT-3).15 There’s a moderate Spearman
14For a similar, earlier set of experiments with FT-GPT-3 vs. human, human was preferred in 87.8% of pairwise cases.
15Similar χ2 tests find no evidence of correlation between correctness and (a) cross-validation split (5-by-2 table; p=.84/.14 for GPT3/CLIP); or (b) which captions are randomly
correlation between the per-contest accuracy between the models (ρ = .28, p .001), but (as a null hypothesis) only a slight correlation between contest date and difficulty for either (later contests easier, GPT3/CLIP ρ = .07/.08, p = .08/.05). When the models’ predictions agree, they are correct 87% of the time. When GPT-3 is wrong, CLIP is right only 38% of the time; under the null hypothesis that their errors are uncorrelated, CLIP’s accuracy would be 62% (p .001 errors are uncorrelated, permutation test). However, when we attempt to identify consistent factors that predict contest difficulty using various visual/linguistic predictors, we find hard vs. easy difficult to predict a priori; our best classifiers perform only slightly above random. We will distribute the hard vs. easy contest lists as a resource for future work.

heading: 4 Related Work
text: Humor. Raskin (1979) and Attardo (2008) highlight three “great families” of theories of the roots of humor: 1) hostility, claims of superiority over someone or something (Gruner, 1978; Billig, 2005); 2) release of a constraint (Freud, 1905; Fry, 1963; Mindess, 1971) and 3) incongruity, (sometimes “incongruity-resolution”; Mulder and Nijholt, 2002) the introduction (and subsequent resolution) of generally incompatible contexts (Schopenhauer, 1818; Shultz, 1976). Shahaf et al. (2015) note that most New Yorker caption contest cartoons involve incongruous situations.
NLP + The Caption Contest. King et al. (2013), Shahaf et al. (2015), and Radev et al. (2016) analyze 5, 16, and 50 New Yorker Caption Contests, respectively. Best-performing features for identifying the funniest among a set of caption choices include: perplexity, match to image setting and uncanniness description, readability, proper nouns (Shahaf et al., 2015), overlap with WordNet’s (Fellbaum, 1998) “person” and “relative” synsets, lexical centrality among submissions (Radev et al., 2016, inspired by Mihalcea and Pulman (2009)), and sentiment (both papers). Our “location” and “uncanny description” annotations are direct analogs of the “context” and “anomaly” tags of Shahaf et al. (2015), and our data incorporates that generously released by the previous researchers. Our extensions are (a) the addition of two novel tasks; (b) using new data/resources/models to curate ranking pairs (see
assigned as negative choices (2646-by-2 table, p=.92/.79 for GPT3/CLIP).
§2); and (c) evaluating two distinct audience preferences: New Yorker editors vs. “the crowd”. Appendix H highlights efforts beyond the scope of peer reviewed AI venues, e.g., blog posts.
Measuring preferences over captions. While humor is ultimately subjective, work on the contest has studied modeling average preferences of raters. Tanczos et al. (2017) design quality ranking algorithms for the caption contest, framed as identifying the best “arm” in a multi-armed bandit setting; their crowdsourcing system NEXT (Jamieson et al., 2015) is used by The New Yorker. It does not directly use the content of the cartoons/contests. The result is Jain et al. (2020)’s continuously updated corpus, from which we draw some of our data.
Multimodal and computational humor. Chandrasekaran et al. (2016) explore humor recognition in images, and Castro et al. (2019); Hasan et al. (2019); Patro et al. (2021); Hasan et al. (2021) explore laughter prediction in TED-talks/sitcoms. Tsakona (2009); Fallianda et al. (2018) study political cartoons. Chakrabarty et al. (2022) recently proposed a version of NLI for figurative language, which can be humorous. Some work has tried to detect whether a sentence is humorous or not (Blinov et al., 2019; Annamoradnejad and Zoghi, 2020). More difficult to evaluate (Valitutti, 2011) are setups where the goal is to automatically generate humorous content in various contexts (Binsted and Ritchie, 1994; Stock and Strapparava, 2003; Mihalcea and Strapparava, 2005, 2006; Wang and Wen, 2015; Chandrasekaran et al., 2018; Yoshida et al., 2018; Sundaram, 2018; Shimomoto et al., 2019); a survey is provided by Amin and Burghardt (2020).
Explaining humor. In the taxonomy of Tan (2022), joke explanations are most related to proximal mechanisms: “This type of explanation attempts to provide the mechanism behind the predicted label, i.e., how to infer the label from the text”, or efficient cause a la Aristotle (Lombrozo, 2006). Chowdhery et al. (2022) undertake a qualitative exploration of (non-visual) joke explanations.

heading: 5 Conclusion
text: We demonstrate that today’s vision and language models still cannot recognize caption relevance, evaluate (at least in the sense of reproducing crowdsourced rankings), or explain The New Yorker Caption Contest as effectively as humans can. However,
the partial capacity of today’s AI is still substantial, and may be sufficient for models to serve as creative collaborators, e.g., as brainstorming assistants for humorists/cartoonists. Specifically: 1) our matching/quality ranking models could help entrants receive quantitative feedback on the relevance/predicted quality of their submissions, and 2) the annotated corpus+explanations we introduce could be repurposed for generation (we explore generation of novel cartoons/captions in Appendix G). Finally, a promising avenue for future work focused on generating humorous captions (c.f. our focus of humor “understanding” benchmarks) would be to operationalize the feedback provided by our matching/ranking models in an reinforcement learning from human feedback (RLHF) loop.
A last remark. We cannot claim to know whether the human-machine ‘humor understanding gap’ will be closed sooner or later.16 But we encourage other researchers to have as much fun with the topic as we did!

heading: 6 Limitations
text: The New Yorker Cartoon Caption Contest represents a narrow slice of humor, deriving from a particular language, region, history, culture, style, and set of conventions. Hence, the results of this study do not represent or cover all types of humor.
Our framing of the quality ranking task could be interpreted as seemingly prescriptive (i.e., that joke A is “objectively” better than joke B), but New Yorker editorial selections should not be taken as ground truth for funniness; disagreement about what is funny is expected and valid. Our tasks operationalize the prediction of only average preferences (rather than individual ones), and these preferences may include a partiality or bias towards items that conform to the characteristics of prior contest winners or published New Yorker cartoons.
Finally, the explanations in our annotated corpus were largely written by a single author of this paper. While a larger pool of the crowdworkers judged these explanations to be of higher quality in comparison to machine generations, future work would be well-suited to compare the person-toperson variance in explaining why particular jokes are funny.
16Or never. Is never good for you?

heading: 7 Acknowledgements
text: We thank the cartoonists and contest entrants for their wonderful efforts! We additionally thank our crowd annotators for their diligent work, Lisa Watkins for contributing to the human performance estimates, and the anonymous reviewers for their constructive comments. This work was funded in part by DARPA MCS through NIWC Pacific (N66001-19-2-4031), the Allen Institute for AI, and a Google Focused Research Award. Jack Hessel conducted initial work while at Cornell University. Ana Marasović conducted this work while at The Allen Institute for AI. Rowan Zellers conducted this work while at University of Washington.

heading: A Crowdworking Details
text: We use three Mechanical Turk interfaces to gather data. These are:
1. Cartoon description (Figure 6). We ran this HIT 3 times per cartoon. 2. Cartoon wikipedia links (Figure 7). We ran this HIT 2 times per cartoon. 3. Pairwise explanations (Figure 8). We ran this HIT 2.7K times to facilitate the comparisons in §3.2
Qualification+training rounds. To ensure our set of crowdworkers were properly trained for the annotations, we ran two types of qualification rounds: one for the description/link HITs, and one for the pairwise explanation HITs.
For the description/link HITs, our qualification round was based off an earlier and more involved HIT that involved a joint setup where, for 3 cartoons, users described cartoons, highlighted image regions, explained jokes, etc. We allowed users from {AU, CA, NZ, GB, US} with 10K prior approved HITs and a minimum acceptance rate of 97% on their previous HITs to participate. Some of the cartoons and captions contain mature themes; we provided the recommended disclaimer for this and other HITs: “WARNING: This HIT may contain adult content. Worker discretion is advised.” We manually graded the responses of 30 annotators in a qualification round, and qualified 21. Through a mix of the older, more involved HITs and the streamlined HIT in Figure 6, which is a pared-down version of the original HIT without captions, we gathered descriptions of the cartoons. We also gathered the locations/Wikipedia entity links from the qualified annotators. These annotations were gathered in mid-to-late 2021.
About 9 months later, we conducted a second set of Mechanical Turk studies for pairwise judgment evaluations for explanation. A second qualification round was run, in which we asked annotators to rate the quality of several joke explanations which we manually selected to be good/bad across various desirable axes. We qualified 29 out of 51 annotators who attempted the HIT via manual inspection of their responses. This set of annotators were given access to the final pairwise-judgment HITs.
Crowdworking studies of standard computer vision corpora (involving no personal disclosures) are not required by our IRB to be reviewed by them. While the authors of this work are not lawyers and
this is not legal advice, this opinion is based on United States federal regulation 45 CFR 46, under which this study qualifies and as exempt. We hashed crowdworker IDs in the public release so annotations cannot be back-traced to individual workers.

heading: B Additional Experimental Details
text: B.1 From Description details For each cartoon, we have multiple annotations of each type, as detailed in §2.2. During training, we utilize all location/description/uncanny description/sets of links, but at test time, we randomly sample a single set of these four annotation types such that inference requires only a single forward pass. For fair comparison, the randomly sampled description available at test time is held constant between all methods.
More detail about how we managed multiple annotations: because we have 2 locations× 3 descriptions × 3 uncanny descriptions × 2 entity links, there are potentially 36 possible combinations we could use to form a from description instance for each cartoon. However: tuples are constructed at the annotator level to account for potential dependencies between annotation types: because descriptions/uncanny descriptions were were collected in the same HIT, the uncanny description may reference entities from the description because they were authored at the same time by the same annotator in sequence. Similarly, the (locations, links) were collected in the same HIT. So, we instead consider all six possible tuples holding author constant between HITs, i.e., 3 (description, uncanny description) × 2 (location, link). For test time, select a single random valid tuple of annotations for evaluation, which is fixed for all comparisons.
B.2 CLIP For fine-tuning results, we do linear warmup for 200 steps and conduct a small learning rate search on the validation set for each cross-validation split independently between {5e-5, 1e-5, 5e-6}, keeping batch size fixed at 32. To keep the entire cartoon in the 336px square input, we resize and pad. At training time, we perform data augmentations on the image, including: random horizontal flipping, random color jittering, and random grayscaling.
B.3 OFA We use validation-set early stopping on crossentropy loss, and fine-tune OFA separately for each cross-validation split. After fine-tuning, we select the top-1 prediction according to beam search (n=5). We finetune OFA Huge with a learning rate of 5e-5, which was determined via a small grid search over the first cross-validation split. We use label-adjusted smoothed cross entropy loss as implemented by the OFA authors17 with smoothing of 0.1. We train for a maximum of 7 epochs with a warmup ratio of 6%. For each image, we query for the four different types of annotations shown in Figure 3. To facilitate this, in addition to providing OFA with the image, we also provide it with a per-annotation-type prompt:
1. for locations: “Where does this take place?” 2. for descriptions: “Describe this image.” 3. for uncanny: “What’s unusual about this im-
age?” 4. for entities: “What entities are there?”
In early experiments, instead of composing with a language model, we did attempt to fine-tune OFA directly for the explanation task. However, we found that the resulting perplexity (roughly 300) was significantly higher than for other fine-tuned models, with the errors difficult to diagnose.
B.4 T5-Large/T5-11B. For T5-Large, we conduct a small, per-crossvalidation split learning rate search between {1e-4, 1e-5, 5e-5} and keep batch size fixed at 64. For T5-11B we use a fixed learning rate of 1e-5 and a batch size of 64.
B.5 GPT-3 Zero Shot/In Context We use GPT-3’s davinci-text-002 model for our main zero shot and in-context learning experiments. Examples of zero-shot prompts for all tasks are given in Figure 9. The in-context prompts are similar, except they contain 5 random samples from the training set. A full, randomly selected in-context prompt for the explanation generation task is given in Figure 10.
B.6 GPT-3 Fine-tuning We use the OpenAI fine-tuning API to fine-tune davinci, a 175B parameter language model.18
17https://github.com/OFA-Sys/OFA 18https://beta.openai.com/docs/guides/fine-tuning
While the precise details of how the API works are not currently available (e.g., which parameters are updated, or which version of davinci is used), we use the same cross-validation setup as for the other models so that the results are comparable. The total fine-tuning cost is approximately (3 tasks) × (5 cross-val splits) × (40 dollars per fine-tune) = 600 dollars.
B.7 GPT 3.5/GPT-4 Details Between submitting this work and its acceptance, OpenAI released two new models, GPT-3.5 (sometimes called ChatGPT when accessed through the chat interface) and GPT-4; we updated our results to include these models. Figure 11 provides an example of a prompt/response in the new “Chat” API, which requires a more structured conversational prompt compared to the GPT-3 “Completion” API; this prompt includes a “system” prompt, which describes the desired behavior of the model, e.g., “You are CaptionContestGPT...” We sample with default hyperparameters in all cases. The cost of GPT 3.5 is an order of magnitude less than GPT-4. In total our GPT-4 queries cost on the order of $4K.

heading: C Task Construction Details
text: Identification of High Quality Captions. For each contest, our first step is to identify a set of high quality captions; these are involved in construction of instances for all three tasks. For cases where we have access to the three official New Yorker finalists, all are automatically added to the high quality set. Next, for cases where we have crowd ratings, we consider the top 5 crowd ranked captions according to the mean score provided by Jain et al. (2020). From these top 5, we select 3 diverse candidates among these using a semantic deduplication method: specifically, we compute the SBERT (Reimers and Gurevych, 2019) vector for each candidate using paraphrase-MiniLM-L6-v2, compute a hierarchical clustering of the candidates, and sample a single candidate from each cluster — the result is a set of candidates that is representative of all clusters. In total, there are 2.7K high quality captions across 704 contests. Each contest either has 3 high quality captions (coming from the official New Yorker finalists or, if those aren’t available, highly crowd-rated options), or 6 (if both official finalists and crowd rated are available).
Forming Matching Instances. For each high quality caption, we create a matching instance that serves as the correct answer. Next, we randomly assign captions to mismatched contests to form negative, mismatched sets to serve as false options. While the assignment is random, we have two constraints: 1) we assign within cross-validation splits only, to ensure that training/validation/testing captions are disjoint; and 2) we construct the corpus with no answer-only biases by performing the negative assignment such that each answer appears exactly once as a correct answer and exactly 4 times as an incorrect answer in other instances.
Forming Quality ranking Instances. For each high quality caption, we aim to sample from the larger set of all submissions for the contest captions that are just “okay.” First, we note that 25 contests from early on in the contest’s history were missing entries, so we are limited to sampling negatives for 679 contests. Next, because many entries are exact duplicates, we deduplicate on string matching, such that “okay” captions are not exact copies of 1) the
identified high quality captions; and 2) any other sampled “okay” captions.
Next, for later contests from Jain et al. (2020), we have estimated quality ratings based on crowd feedback for each entry already: in that case, we discard the top third and bottom third of captions according to mean crowd rating — the middle tertile form the “okay” set we sample from.
But, for earlier contests, we do not have direct ratings: we only have access to New Yorker finalists and a large pool of entries. For those cases, we aim to eliminate captions that are clearly likely to be low quality. To accomplish this, we train a quality ranking model (conditioned just on the caption text, rather than any information about the contest) using crowdlabelled data from 253 contests provided by Jain et al. (2020). We sample a good/bad set by selecting from each contest the top and bottom 1000 entries according to their mean crowdsource score: the resulting dataset forms a corpus of 506K captions. We form two sets of labelled data based on the parity of the contest
number (i.e., even vs. odd). We train/validate two T5-Large models based on this split for the binary classification task. While the average validation accuracy we achieve is 65%, we achieve higher precision in identifying the “bad” label: precisionat-10 is 83, precision-at-20 is 77, precision-at-30 is 72. It appears to be harder to identify very good captions than very low rated ones: precision-at-10 is 77, precision-at-20 is 73, precision-at-30 is 70. Upon training these models, we perform inference on all captions in contests without crowd ratings, and discard the 25% of entries with the lowest predicted score. Entries with very low scores have some common characteristics, e.g., they don’t have the gestalt of a New Yorker caption, they have many typos/formatting issues, they include the contact information of the submitter, etc. Examples of discarded captions (some are obfuscated for privacy reasons) are:
• THEY COULDN’T WAIT TO MARRY SO THEY CAME TO RECITE THEIR VOWS BETWEEN TAKES FROM “ PRIMITIVE LOVE LIFE” • You’re hurting me, will we ever break up?” (@ technology) • The stressed is so “Bad’ in the world. “you or me ” did not see(BIG )( “FOOT) • Too mammalian, needs reptile.” [NAME], [STATE] [EMAIL]@gmail.com
After identifying a set of captions that are not obviously bad, nor apparently among the top quality submissoins, our second step is to deduplicate entries. Because submitted captions for each contest are frequently identical to other submissions or play off the same core joke concept, we perform the same SBERT+hierarchical clustering semantic deduplication step as we did for sampling the diverse high quality set (described above). Specifically, we extract SentenceBERT embeddings (Reimers and Gurevych, 2019) for each of theN entries, and then compute a hierarchical clustering of the embeddings into .7 ·N clusters, sampling only a single representitive from each cluster to form a less-redundant set. This removes 30% of the data with close neighbors in the final set: for example, for a contest depicting two monsters eating buildings in New York City, this step downsamples 100 “tastes like chicken” jokes (which end up in a single cluster) to a single exemplar.
After filtering, for all contests, we are left with a (softly) deduplicated pool of candidate entries that
are likely to be at least okay, but unlikely to be as good as the verifiably high quality entries. For each high quality entry, we sample an “okay” caption with: 1) similar estimated quality according to the text-only models; 2) similar length in words; 3) similar length in characters; 4) similar amount of punctuation; 5) a dissimilar SBERT embedding.
Explanation corpus. After several attempts to solicit high-quality explanations from crowdworkers fell short, one of the authors of this paper decided to simply annotate a corpus of explanations themselves. For each contest, a high quality caption was sampled for annotation — this high quality caption was sampled arbitrarily from the set of New Yorker finalists if they were available, and, in the few cases where New Yorker finalists weren’t available, from the set of high quality crowd captions. Of the 704 sampled explanations, the author reported understanding 651 of them, and wrote an explanation for each. This was a substantial effort: the resulting corpus has a mean of 60 words of explanation per cartoon, and the total length, 39.3K words, is comparable in length to a novella.

heading: D Graphical version of matching and ranking results.
text: In Figure 12, we use vertically-stacked bars to illustrate the difference between zero-shot (small dots), five-shot (vertical stripes), and fine-tuned (solid) versions of various models. Human results are set off by dark green lines.
The scatter-plot in Figure 13 uses the same graphical conventions to display the qualityranking results. Recall our caveat that crowd accuracy may be more statistically reliable, in the sense that crowd selectors, whose tastes underlie the y-axis results, vastly outnumber New Yorker
editors, whose tastes underlie the x-axis results.

heading: E Automatic evaluation of explanations
text: For completeness, we provide the results for automatically-calculated explanation-evaluation metrics in Table 5. (Log probabilities are unavailable for GPT-3.5/GPT-4 so we cannot report perplexity for them.) However, we believe that the human evaluations reported in the main body of the text are better quality measures.

heading: F Machine explanations that were preferred over human ones
text: GPT-4 In 8/130 cases, for our human vs. GPT-4 5-shot experiments, the machine generation was
preferred to the human reference by 3/3 annotators. In Figure 14 we conduct a close reading of these 8 instances to understand where the human references fell short. In all cases, both were topical, but, for a handful of cases, the machine generation is arguably better because it’s more succinct, or offers a more meaningful detail.
GPT-3 We also include a close reading of several instances where a majority of annotators preferred GPT-3 annotations vs. our human ones. This occured in 16/131 cases for our human vs. GPT-3 experiments: in 15 cases, 2/3 annotators preferred the machine generation, and in 1 case, 3/3 annotators preferred the machine generation. We present a few examples of these cases with comments in Figure 15. Similar to GPT-4, most commonly, both the explanations were reasonable, in one case, the human-written explanation missed a pop culture reference to “The Attack of the 50 Foot Woman” (1958) but GPT-3 mentions it. In six cases, we disagree with annotator consensus: the machine caption makes some correct references, but we believe it (sometimes subtly) misses the point.

heading: G Aiding humor generation with system-assisted brainstorming
text: Our main experiments focus on three tasks that probe machine capacity for matching, ranking, and explaining caption contest entries. But, given prior interest in generating caption entries, to provide a pointer towards more creative use cases, we developed a curated prompt that re-frames the corpus in a manner that enables the production of cartoon descriptions, concept brainstorming, and, ultimately, caption generation. In contrast to our existing annotations, we are inspired by Wallace (2022)’s framing, where free associations are used to connect-thedots from cartoons to concepts to captions. So, for 9 finalist captions, we reverse-engineer a plausible brainstorming process that could have given rise to the caption by listing several potential associations from the scene, and, from that list, selecting the 1-3 that underpin the basis for the winning joke. In the end, our prompt consists of 9 examples formatted as:
1. cartoon description (from our annotations) 2. uncanny description (from our annotations) 3. list of entities in scene (from our annotations) 4. list of concepts from the cartoon that could
be used for a caption (hand-designed for the
prompt) 5. a selected set of 1-3 ideas (selected from (4)) 6. caption (a finalist) 7. explanation of the caption (from our annota-
tions)
A portion of our prompt is given in Figure 16, along with an unconditional generation (where the cartoon concept and caption are generated) and a conditional generation. Within 5 samples, GPT-3 invents a scene where a large chicken is playing croquet in a yard, and the caption: “I’m not sure this is what they meant by free range.” Also, when conditioned on a basic description of a real contest which depicts a large group of circus performers intruding on an unsuspecting person in their living room (Contest #818), it generates “I’m never buying a timeshare again.” Looking forward, we expect the matching/quality ranking models could be used in conjunction with this prompt to automatically filter for scene-specific generations with style similar to previous finalists.

heading: H Related work beyond peer reviewed
text: AI venues
Outside of peer-reviewed NLP venues, several projects have used computational techniques to analyze the contest, usually with the goal of generating AI-assisted entries: • The Pudding: Mishkin et al. (2022) collabo-
rated with GPT-3 (Brown et al., 2020) to generate entries. • coolposts: Wilson (2019) used topic models to condition an RNN caption generator. • LILY Lab @ Yale’s Spring 2017 projects include a number of caption contest efforts, including work by Prince, Friedman, Zucker, Anbarasu, and Dohrn. • The Verge: Zelenko and Bi (2015) trained a Markov language model on previous winning entries.

heading: I Some of our favorite New Yorker cartoons
text: We list our favorite captions below. The corresponding images can be seen by clicking on the cartoonist/author names.
YC: “The doctor said it might help me quit.” — Vince Conitzer/Jeffrey Adam Katzenstein
JD: “You are so smart. You look amazing. You inspire me. [Complimentary bread].”
— Seth Fleishman
JMH: “Thanks, I’ll write that down.” — Victoria Roberts
JDH: “They’re from Earth. I wonder if they know Dan.”
— Benjamin Schwartz
LL: “I want to be feared as a tyrant, loved as a father, and revered as a god, but I also want them to think I’m funny.”
— Zachary Kanin
AM: “I can’t believe I’d been carrying them in my mouth.”
— Amy Hwang
RZ: “Well, there’s your problem.” — Edward Koren
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section 6
3 A2. Did you discuss any potential risks of your work? Limitations section 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? The abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? yes, our new corpus/tasks. Section 2 describes them.
3 B1. Did you cite the creators of artifacts you used? Yes, section 2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Yes, we discussed the distribution of our dataset, which have made public under Creative Commons Attribution 4.0.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, section 2
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 2 and appendix C
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3 and appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3 and Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix A
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 2, Appendix A
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 2, Appendix A
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? appendix A
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We don’t know many specifics, other than country of IP: which we discuss in appendix A

heading: Future Work
text: While a larger pool of the crowdworkers judged these explanations to be of higher quality in comparison to machine generations, future work would be well-suited to compare the person-toperson variance in explaining why particular jokes are funny. 16Or never. Is never good for you?