Title: Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning
Benchmark: New Yorker Caption Ranking
Summary: The paper presents a New Yorker Caption Ranking Dataset, a novel multimodal human preference dataset for generating humorous cartoon captions. The paper presents additional novel evaluation methods to perform group comparisons between AI and human-generated cartoon captures, and leverages data from The New Yorker Caption Contest. The benchmark can be used to assess model-generated captions and support preference-based fine-tuning algorithms. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: humor, humorous captions, funny captions
Phenomenon defined by authors: No
Phenomenon definition: Generating a humorous caption is the task of writing funny captions on a literary piece, primarily cartoons.  
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The cartoon captioning task is defined as a model generating a funny caption given information about the cartoon. Both multimodal and language-only models are evaluated, where language-only models receive descriptions and object entities of the cartoons. The paper also compared zero-shot models against SFT, RLHF, and DPO finetuned models on certain contests within the dataset. 
Task item definition: A single item would have the cartoon, its language description (provided by GPT4o-vision), its caption, and its label (funny, somewhat funny, unfunny). 
Task source: Real task examples (e.g. GitHub issues), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: The dataset is crowdsourced from The New Yorker cartoon caption contest. 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 284183913
Dataset size notes: The fine-tuning experiments designated contests 530-890. The test set contains 47 contests, the validation set contains 44 contests, and the train set contains the remaining contests.
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: The paper presents a novel evaluation method for group comparison techniques, denoted by Group Overall and Group Best Pick. Human or LLM raters evaluate groups of 10 captions from different sources, and compare them against four groups of past human submissions in the buckets of ranks 1-10, 200-209, 1000-1009, and median. The evaluators then compare the overall funniness of the group against the contest-submitted captions, and pick the funniest caption overall between the funniest captions of the evaluation group and the contest group. GPT4-Turbo-vision, GPT4o-vision, GPT4-Turbo, and GPT4o were used as LLM evaluators. The ranking accuracy and caption win rates of the cartoons are then calculated from the evaluations. 
Metric aggregation: Simple Mean
Metric statistics: Simple mean and variance on accuracy are used to assess the overall and best pick comparisons for cartoons, and expectation adjusted distinct N-grams (EAD) and Sentence-BERT embedding cosine similarity (SBERT) are used to assess caption diversity. 
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Authors highlight that writing funny captions requires an understanding of how to appeal to a broad range and variability within humor and human judgements. Thus, a benchmark in funny caption writing requires a comparison to human performance, because the task is a domain where expert humans consistently outperform current AI system, leading to the creation of the introduced dataset. 
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding