Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench

## Abstract

Abstract We present HumorBench, a benchmark designed to evaluate large language models’ (LLMs) ability to reason about and explain sophisticated humor in cartoon captions. As reasoning models increasingly saturate existing benchmarks in mathematics and science, novel and challenging evaluations of model intelligence beyond STEM domains are essential.
Reasoning is fundamentally involved in text-based humor comprehension, requiring the identification of connections between concepts in cartoons/captions and external cultural references, wordplays, and other mechanisms.
HumorBench includes approximately 300 unique cartoon-caption pairs from the New Yorker Caption Contest and Cartoonstock.com, with expert-annotated evaluation rubrics identifying essential joke elements. LLMs are evaluated based on their explanations towards the humor and abilities in identifying the joke elements. To perform well on this task, models must form and test hypotheses about associations between concepts, potentially backtracking from initial interpretations to arrive at the most plausible explanation.
Our extensive benchmarking of current SOTA models reveals three key insights: (1) LLM progress on STEM reasoning transfers effectively to humor comprehension; (2) models trained exclusively on STEM reasoning data still perform well on HumorBench, demonstrating strong transferability of reasoning abilities; and (3) test-time scaling by increasing thinking token budgets yields mixed results across different models in humor reasoning.

## 1 Introduction

Recent advances in large language models and reasoning techniques have led to the saturation of many existing benchmarks, particularly in STEM domains such as mathematics and programming, where frontier models now approach or exceed human-level performance (Abdin et al., 2025a; Sun et al., 2025; Quan et al., 2025). This progression highlights the need for novel and challenging evaluations that can meaningfully differentiate model capabilities and provide insights into their reasoning processes. Non-STEM reasoning tasks, particularly those involving cultural understanding and implicit knowledge, represent underexplored territories for model evaluation.

Humor comprehension represents a particularly challenging frontier for artificial intelligence (Hessel et al., 2023; Zhang et al., 2024; Zhou et al., 2025; Kazemi et al., 2025; Liang et al., 2025). Although large language models (LLMs) excel across many domains, understanding humor still requires sophisticated reasoning that integrates context, cultural knowledge, and implicit connections. These challenges make humor an ideal testbed for evaluating advanced reasoning in AI systems.

Figure: Figure 1: Overview of our humor analysis approach. (a) We distinguish between objective and subjective components of a joke. To convert the open-ended task of humor explanation into a fair benchmark, we focus exclusively on objective elements. (b) Overview of the dataset, benchmark task, and grading scheme in HumorBench. Each cartoon-caption pair contains one or more “element” annotation. For the benchmark, an LLM is tasked with explaining the joke in the caption. An autograder evaluates if the explanation contains each element.
Refer to caption: x1.png

We present HumorBench, a benchmark that evaluates LLMs’ ability to explain sophisticated cartoon-caption humor by identifying the mental leaps connecting visuals, captions, and external knowledge (Figure [1](https://arxiv.org/html/2507.21476v1#S1.F1)). For each pair, we annotate the objective elements essential for comprehension, creating a ground truth focused on factual connections rather than subjective appreciation.

We benchmark both a standard set and a harder subset. On HumorBench-hard, which features more complex examples requiring multiple reasoning steps or obscure cultural knowledge, no current LLM exceeds 60% accuracy.
Our benchmarking of current state-of-the-art models reveals two key findings:

- 1.
We observe a high correlation between performance on HumorBench and existing STEM benchmarks, suggesting a significant transfer of general reasoning abilities to humor comprehension tasks.
- 2.
Even models trained exclusively on STEM reasoning tasks (e.g., mathematical problem-solving) perform well on HumorBench, indicating that abstract reasoning skills acquired in one domain can transfer effectively to humor comprehension.
- 3.
Test-time scaling measures for humor reasoning yield mixed results, indicating that simply increasing computational resources at inference time does not consistently improve performance on this challenging domain.

### 1.1 Why Another LLM Humor Benchmark?

Several benchmarks have already focused on measuring LLMs’ capabilities around humor. Specifically, Hessel et al. (2023) and Zhang et al. (2024) both build upon the New Yorker Caption Contest (NYCC) dataset—a weekly feature by the New Yorker magazine where readers submit funny captions for cartoons (see Figure [1](https://arxiv.org/html/2507.21476v1#S1.F1) for an example).
Hessel et al. (2023) created three benchmarks using this dataset: ranking the funniness of caption pairs, matching cartoons to valid captions, and explaining the humor behind captions. However, these benchmarks simultaneously measure two distinct capabilities: (1) understanding the intended jokes (objective elements) and (2) aligning with individual and subgroup humor preferences (subjective factors). As Zhou et al. (2025) points out, performance on these previous benchmarks is heavily influenced by an LLM’s ability to align with specific audience preferences rather than directly measuring its reasoning about the jokes themselves.

Our benchmark, HumorBench, addresses this limitation by focusing solely on the objective elements of humor comprehension, specifically measuring the *humor reasoning abilities* required to understand cartoons and their captions.
As validated by our experimental findings, LLMs’ performance on HumorBench correlates well with their performance on other reasoning benchmarks.

## 2 Related Work

Reasoning-focused language models.
Recent advances in large language models (LLMs) have seen the emergence of specialized reasoning models that excel at logical deduction, mathematical problem-solving, and multi-step reasoning while maintaining strong general language capabilities. These reasoning-enhanced models employ various approaches: training-focused methods like those used by *Minerva* (Lewkowycz et al., 2022), *WizardMath* (Luo et al., 2023), and *Phi-4-Reasoning* (Abdin et al., 2025b) leverage carefully curated STEM-heavy corpora; inference-time techniques boost reasoning without changing model weights, including *Self-consistency* (Wang et al., 2023), *Tree-of-Thought* methods in *DeepSeek-Math* (Shao et al., 2024), *Least-to-most* prompting (Zhou et al., 2023), and *Process supervision* (Lightman et al., 2023); while hybrid approaches like *MAmmoTH* (Yue et al., 2023) combine diverse training data with structured inference protocols, *ToRA* (Gou et al., 2023) integrates formal verification systems, and *MathGLM* (Yang et al., 2024) combines symbolic computation with natural language reasoning. Models like *Gemini Ultra* (Google, 2024) and *Claude 3.5* (Anthropic, 2024) achieve strong reasoning through both architectural innovations and sophisticated training, suggesting that advances in machine reasoning now follow multiple complementary paths rather than relying solely on parameter count (Wei et al., 2024).

Humour benchmarks.
Beyond simple joke generation, several resources now probe LLM humour competence. Hessel et al. (2023) introduces three New Yorker cartoon-caption subtasks that test multimodal humour understanding and explanation. For word-play, the *ExPUNations* corpus augments classic pun datasets with human-written explanations and funniness ratings (Sun et al., 2022), while Xu et al. (2024) systematically benchmarks pun recognition, explanation and creation. Complementing these datasets, (Ermakova et al., 2025) Lab provides reusable test collections for humour-aware information retrieval.

Open-ended evaluation frameworks.
Automatic grading of creative, unconstrained outputs increasingly relies on the *LLM-as-Judge* paradigm. G-Eval couples chain-of-thought GPT-4 judging with a form-filling rubric, achieving human-level reliability on summarisation and dialogue (Liu et al., 2023). MT-Bench and its crowdsourced *Chatbot Arena* show that GPT-4 judges agree with human preferences on multi-turn instruction following in $\sim$80% of cases (Zheng et al., 2023). Going further, PaperBench grades agents on reproducing ICML-level research papers with hierarchical GPT-4 rubrics and expert audits (Starace et al., 2025). We adopt a similar rubric-guided judging scheme but focus specifically on humour reasoning, enabling systematic comparison of explanation quality across models.

## 3 HumorBench

### 3.1 Main Benchmark Task

HumorBench frames humor understanding as an open-ended task: given a textual description of a cartoon and its caption, a model must articulate in its own words the underlying joke. We deliberately avoid the multiple-choice or ranking formats common in existing humor benchmarks because, for creative tasks, fixed answer sets can (i) inadvertently hint at the punchline and (ii) fail to accommodate the diverse range of valid explanations a competent reader might produce.

To make this free-form setting automatically gradable, we distill each cartoon into a concise rubric of 1–3 objective “elements." An element represents a single, easily verifiable fact that any correct explanation must include (e.g., in NYCC Contest #665, the observation that “the shark interprets the swimmer as groceries"), as shown in Figure [8](https://arxiv.org/html/2507.21476v1#S4.F8). This approach allows for creative expression while maintaining consistent evaluation standards (see Appendix [A](https://arxiv.org/html/2507.21476v1#A1) for the complete prompt).

### 3.2 Dataset: Cartoon and Caption Sources

Our dataset comprises cartoons and captions from two primary sources: the New Yorker Caption Contest (NYCC) and Cartoonstock.com. We sourced NYCC captions from publicly available datasets (Hessel et al., 2023; Jain et al., 2020; Zhang et al., 2024), selecting only those ranked among the top 3 finalists to ensure each cartoon features a coherent, high-quality joke. For Cartoonstock cartoons, we utilized their original accompanying captions. Both sources specialize in dry, witty humor that demands sophisticated reasoning—often requiring multiple mental leaps to fully comprehend, as illustrated in Figure [1](https://arxiv.org/html/2507.21476v1#S1.F1).

While cartoons inherently include visual elements, our benchmark focuses on testing humor comprehension rather than visual interpretation capabilities. Therefore, we created detailed textual descriptions of each cartoon, carefully capturing all information necessary to understand the caption while maintaining neutrality. These descriptions include essential details about the setting, characters, visible emotions, and speaker identification, while deliberately omitting artistic style unless directly relevant to the joke. For researchers interested in extending this to a multimodal benchmark, we provide source links to the original images—NYCC images are available through (Hessel et al., 2023; Jain et al., 2020), while Cartoonstock images require licensing.

### 3.3 Dataset: Element Annotation

The core labels in our dataset are the *element* annotations assigned to each cartoon–caption pair. For every pair, we hand-annotated one to three elements—concise, direct statements that capture the objective components essential to understanding the joke.
As discussed in Section [1.1](https://arxiv.org/html/2507.21476v1#S1.SS1), comprehending cartoon humor requires two distinct capabilities: understanding the objective content of the joke and recognizing the subjective aspects that influence audience reception. Figure [1](https://arxiv.org/html/2507.21476v1#S1.F1) illustrates this distinction—subjective explanations focus on audience reactions (which vary between individuals), while objective elements center on content comprehension. Our benchmark specifically targets these objective elements, which require identifying the mental leaps necessary to "get" the joke through recognizing references, wordplay, implications, or similar mechanisms.
The autograder then evaluates LLM explanations against these elements, verifying that each explanation adequately covers the fundamental objective components of the humor, ensuring a fair and consistent assessment across different models.

In summary, to make this task easily gradable, annotations follow a set of deliberate guidelines:
(1) Elements must be short, direct, and easily verifiable from the description and caption;
(2) An element addresses exactly one concept. Bundling multiple ideas may add noise by forcing the autograder to guess about partial correctness;
(3) Elements deliberately avoid adding bias from subjective opinion about humor.

Figure: Figure 2: Example HumorBench annotation. The cartoon (left) is paired with its description, caption, and two hand-labeled joke elements (right).
Refer to caption: figures/weasel.jpeg

### 3.4 Dataset Refinement

For an LLM evaluation to provide trustworthy results, the underlying dataset must be both accurate and internally consistent. We initially collected 655 unique element annotations, but despite careful guidelines, some entries proved vague or imprecise. To systematically improve quality, we implemented an iterative refinement process.

First, we generated sample explanations for each cartoon–caption pair, alternating randomly between GPT-4o and Claude 3.7 Sonnet. Each explanation was evaluated ten times by our autograder, with elements showing verdict disagreement exceeding 30% flagged for review. These problematic cases were either refined or removed entirely. We repeated this quality control cycle until fewer than 5% of annotations triggered inconsistency flags, ultimately resulting in 499 high-quality unique element annotations forming the foundation of HumorBench.

As an additional validation step, we invited a former chief cartoon editor of the New Yorker to review a random subset of 30 annotations. The editor confirmed that all elements were fair and accurately captured the essential components of each joke. Together, these atomic, objectively verifiable criteria create a robust rubric that enables our autograder to provide consistent and reliable evaluation at scale.

### 3.5 Autograder and Evaluation

During evaluation, an LLM judge assesses each model’s explanation against individual elements to determine whether they adequately cover the essential components of the joke. This approach allows us to efficiently evaluate open-ended text generation at scale.

However, ensuring autograder consistency presents challenges, particularly for tasks that are inherently difficult for LLMs to understand (Min et al., 2020; Starace et al., 2025). To address this, we created a separate benchmark of 300 human expert judgments on explanations from three distinct LLMs: GPT-4o, Gemini 2.5 Pro, and Claude 3.7 Sonnet. Using GPT-4o as the autograder, we achieved 92% accuracy overall:

Table: Table 1: Autograder performance (GPT-4o judge) on 300 human-labeled explanations.

This validation provides two key insights. First, across all models, the autograder’s false positive rate (FPR) substantially exceeded its false negative rate (FNR), indicating a leniency bias. This suggests that *HumorBench scores should be interpreted as an upper bound on model performance*. Second, despite using GPT-4o as the autograder, we observed no significant advantage for GPT-4o-generated explanations compared to those from other models. Together, these findings confirm that our autograder provides a valid, albeit slightly optimistic, mechanism for large-scale evaluation.

Length Control. While models were instructed to keep responses under 200 words, some models exceeded this limit, particularly when reasoning traces were included in the final output. To ensure fair comparison, we truncated all model outputs to the last 1000 tokens.

## 4 Experiments

Along with creating the HumorBench evaluation, we extensively benchmarked current frontier models. For consistency, all models are given the same prompt and scaffolding describing the task (see Appendix  [A](https://arxiv.org/html/2507.21476v1#A1)).
We arrived at this prompt after validating across several different LLMs (Claude 3.7 Sonnet Anthropic (2025), GPT-4o OpenAI (2024a), Gemini 2.5 Pro DeepMind (2025)). While many LLMs had different API endpoints, we tried to maintain consistent parameters where possible. For example, all models had temperature set to 1 and external tool calling deactivated. Note, for all evaluations, autograders, and benchmarks, “GPT-4o" refers to the gpt-4o-2024-08-06 release.

### 4.1 Main Results

Figure: Figure 3: Benchmarking results on frontier models
Refer to caption: figures/full_bench_cost.png

In general, the results from the main benchmarking effort were unsurprising. As shown in Figure  [3](https://arxiv.org/html/2507.21476v1#S4.F3), OpenAI o3 OpenAI (2025a) leads the pack at 87.5% accuracy, dramatically ahead of other SOTA models (Gemini 2.5 Pro, Claude 3.7 Sonnet, and Deepseek R1 DeepSeek (2025)), all achieving approximately 80%.
In general, smaller models (like Llama 4 Maverick Meta (2025), Qwen 2.5 Alibaba (2025), and o3-mini OpenAI (2025b)) performed worse.
We also found that newer versions of models generally dominate older versions of the same model, with o3 outperforming o1 OpenAI (2024b) and Gemini 2.5 pro outperforming Gemini 1.5 pro.
In general, "reasoning" versions of models seemed to outperform the base versions of the same model. For example, DeepSeek R1 (79.8%) strongly outperformed Deepseek V3 DeepSeek (2024) (72.2%), despite being based on the same 671B parameter architecture. Similarly, Claude 3.7 Sonnet with a thinking budget of 1024 tokens (83.6%) clearly outperformed the base Claude 3.7 Sonnet (80.4%).
When compared with total cost of running the benchmark, we see that more expensive models tend to outperform less expensive models, either due to a larger underlying model or using more reasoning tokens in the output.

### 4.2 Transferability of Reasoning Skills

Figure: Figure 4: HumorBench performance compared to several common benchmarks. We see positive correlation with GPQA, ARC-AGI, and LMArena. In particular, ranking compared to ARC-AGI is nearly identical to that of HumorBench, indicating a strong reasoning component to the HumorBench task.
Refer to caption: figures/full_bench_gpqa.png

**Table 2: Spearman’s rank correlations between HumorBench and other benchmarks. Asterisks indicate significance: *$p<0.05$, **$p<0.01$. ARC-AGI correlation shown separately for results with and without o-series models, which were fine-tuned for ARC-AGI**
| Benchmark | Corr. | p-value |
| --- | --- | --- |
| GPQA Diamond | 0.736* | 0.024 |
| ARC-AGI (with o-series) | 0.650 | 0.058 |
| ARC-AGI (w/o o-series) | 0.943** | 0.005 |
| LM Arena ELO | 0.714 | 0.071 |

To gauge how well other model skills transfer to humor comprehension, we correlate HumorBench accuracy with three widely used LLM benchmarks: GPQA-Diamond Rein et al. (2023), ARC-AGI Chollet et al. (2025), and LM Arena ELO Chiang et al. (2024). HumorBench scores are positively associated with all three (see table [2](https://arxiv.org/html/2507.21476v1#S4.T2)). In particular, after removing o-series models (whose reported scores come from ARC-tuned variants) the correlation with ARC-AGI rises to $\rho=0.943$ ($p=0.005$), underscoring the shared reasoning demands of the two tasks. The LM Arena correlation ($\rho=0.714$) is solid but notably lower. Overall, this also suggests that LLM progress on STEM domains translates to Non-STEM reasoning as well.

#### STEM-only reasoning improves HumorBench

Our comparison between reasoning models trained on STEM tasks via Reinforcement Learning and their base counterparts yielded particularly revealing results. As illustrated in Figure [6](https://arxiv.org/html/2507.21476v1#S4.F6), R1-Zero, which developed reasoning capabilities exclusively through self-play on STEM problems, demonstrated significant improvements over its base V3 model. Remarkably, it performed nearly on par with DeepSeek R1, despite the latter being trained on non-STEM data such as reading comprehension. Similarly, Phi-4 Reasoning Plus exhibited superior performance compared to its base model (Figure [6](https://arxiv.org/html/2507.21476v1#S4.F6)), although its training was limited to math and coding data (Abdin et al., 2025a). These findings suggest that abstract reasoning capabilities are transferable to humor comprehension, indicating that the reasoning skills required for STEM domains may be fundamentally similar to those needed for understanding humor.

We also note that both R1-Zero and Phi-4 Reasoning Plus include their reasoning traces in their final outputs. Therefore, we evaluated their performances using the length control measure described above to ensure fair comparison across models.

Figure: (a) OpenAI o- series models, varying "reasoning effort"
Refer to caption: figures/openai_reasoning_exp.png

Figure: Figure 6: Deepseek R1 Zero and Phi-4 Reasoning Plus, both exclusively reasoning-trained on STEM tasks, outperform their base versions on HumorBench
Refer to caption: figures/deepseek_comparison_plot.png

### 4.3 Test-Time Scaling

As seen in figure  [5](https://arxiv.org/html/2507.21476v1#S4.F5), while including some reasoning clearly helped model performance, the effect of continuing to increase test-time compute varied significantly between models.
For Qwen plus and the o- series models, increasing the reasoning parameter (reasoning budget and "effort", respectively) generally improved performance.
However, for Claude 3.7 Sonnet, increasing the thinking budget beyond the minimum 1024 tokens clearly hurt performance.

A closer look at the reasoning trace lengths highlights that for most captions, the models did not fully exhaust their reasoning budget (see  [7](https://arxiv.org/html/2507.21476v1#S4.F7)).
Qwen Plus, for example, rarely used more than 400 tokens, even when budgeted 2000, a trend we saw for all test-time experiments. This suggests the LLMs are providing final answers based on completed thinking traces, which makes the inverse test-time scaling effect more puzzling. While a few studies have looked related problems (Su et al., 2025; Shi et al., 2023; Yang et al., 2025; McKenzie et al., 2023), we defer thorough investigation of our observation to future work.

Figure: Figure 7: Token usage with different ‘thinking budget‘ parameters on Claude 3.7 sonnet. Most completions (in the hundreds of tokens) were far below their budgets.
Refer to caption: figures/claude_tokens_overlay.png

### 4.4 Analysis of HumorBench Hard Subset

Figure: (a) Caption: “How about some help carrying the groceries?” Element: To the sharks, the person is the groceries
Refer to caption: figures/hard_cartoons/sharks.jpg

While frontier models like o3 demonstrate impressive performance on HumorBench, certain elements are persistently challenging for all models. To better understand the specific types of humor that remain challenging, we conducted a targeted analysis on the 100 unique elements that were most often missed during the benchmarking, which we call HumorBench Hard. Cartoons in this subset range from pass rates of 60% (6 in 10 models get correct) to 0% (No model gets correct).
See  [3](https://arxiv.org/html/2507.21476v1#S4.T3) for examples of the hard subset.

To get a more granular view of the elements that constitute the hard subset, we analyzed three predefined humor categories: wordplay, cultural references, and toxic or shocking humor elements. These categories were annotated by an LLM categorization pipeline built on o3, which individually categorized each element as in or out of each category. Most elements did not fit into these categories, while some fit into multiple. Our analysis examined the relative representation of each category within the hard subset compared to their representation in the overall HumorBench dataset.

**Table 3: Representation of humor categories in the full dataset compared to the hard subset.**
| Humor Category | Entire Set | Hard Subset | Diff (%) |
| --- | --- | --- | --- |
| Wordplay | 24.4% | 19.0% | -5.4 |
| Cultural Reference | 19.0% | 17.7% | -1.3 |
| Toxic or Shocking | 25.7% | 26.6% | +0.9 |

We summarize the main findings of this analysis in Table  [3](https://arxiv.org/html/2507.21476v1#S4.T3). Overall, these relatively minor deviations indicate that humor category alone is not the primary determinant of difficulty for models. Challenging examples likely hinge on subtler factors, such as the implicit conceptual leaps required or the obscurity or references. We observed that wordplay was slightly under-represented among the hard subset ($-5.4\%$), suggesting current LLMs handle puns or jokes that rely on linguistic manipulation somewhat better than other elements. *Cultural references* and *toxic or shocking* humor, meanwhile, were essentially evenly represented, indicating that these styles do not disproportionately increase difficulty. These nuanced insights encourage further qualitative investigation into the underlying reasons why particular humor instances remain difficult, even for state-of-the-art LLMs trained explicitly with reasoning capabilities.

Lastly, in Figure [8](https://arxiv.org/html/2507.21476v1#S4.F8), we highlight a few concrete examples that are particularly challenging in the HumorBench hard subset. Detailed descriptions are also provided in Table [4](https://arxiv.org/html/2507.21476v1#A3.T4). To our surprise, although several of them are quite intuitive and easy to recognize for humans, LLMs usually struggle on these examples. This represents a fundamental difference in reasoning abilities between humans and LLMs.

## 5 Conclusion and Future Work

In this work, we introduced HumorBench, the first large-scale evaluation that isolates *humor comprehension*, as opposed to subjective funniness, by grading model explanations against concise, expert-annotated *objective elements*. Our experiments with more than a dozen frontier and open-source LLMs revealed that (i) progress on STEM reasoning benchmarks translates strongly to non-STEM humor reasoning, (ii) specialized "reasoning" variants consistently outperform base models even when they were trained only on STEM corpora, and (iii) test-time compute helps, but only up to the point where the relevant background knowledge is actually present in the model. Together, these findings position HumorBench as a sensitive probe of higher-level reasoning that remains comfortably unsolved: the best model still misses over 40% of elements in our hard subset.

Looking forward, there are a number of promising avenues for extending this work. For example, a natural next step is to develop a *multimodal* version of HumorBench by reintroducing the original cartoon images. This would allow evaluation of both visual recognition and reasoning, better reflecting the complete humor comprehension task. Alternatively, the use of HumorBench’s element rubrics as intermediate supervision signals offers an exciting opportunity to explore fine-tuning or reinforcement learning. Finally, improving the reliability of LLM-as-judge evaluation remains an important open challenge, particularly for creative tasks with high output variability.

We hope HumorBench spurs progress on reasoning that bridges the gap between logical deduction and nuanced human culture, and serves as a springboard for genuinely funny, culturally aware AI systems.

## Limitations

HumorBench focuses on textual explanations derived from detailed image descriptions. While this controls for pure vision problems, it inevitably removes part of the challenge: recognizing visual cues. The autograder, despite a 92% agreement with humans, is mildly lenient, so reported scores should be seen as upper bounds. Additionally, while extensive validation and refinement were applied, some element annotations may still contain errors or reflect subjective interpretations that slipped through, introducing noise into model evaluation. Finally, the dataset size (499 unique elements) limits the statistical resolution of fine-grained analyses.

## References

- Abdin et al. (2025a)
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio C. T. Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, and 4 others. 2025a.
[Phi-4-reasoning technical report](https://arxiv.org/abs/2504.21318).
*arXiv preprint arXiv:2504.21318*.
- Abdin et al. (2025b)
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, and 4 others. 2025b.
Phi-4-reasoning technical report.
*arXiv preprint arXiv:2504.21318*.
- Alibaba (2025)
Alibaba. 2025.
Tongyi qianwen (qwen) 3 series and qwen 2.5 models.
[https://qwenlm.github.io/blog/qwen2.5-max/](https://qwenlm.github.io/blog/qwen2.5-max/).
Accessed 19 May 2025.
- Anthropic (2024)
Anthropic. 2024.
Claude 3.5 sonnet.
[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet).
- Anthropic (2025)
Anthropic. 2025.
Claude 3.7 sonnet and claude code.
[https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet).
Accessed 19 May 2025.
- Chiang et al. (2024)
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024.
Chatbot arena: An open platform for evaluating llms by human preference.
*arXiv preprint arXiv:2403.04132*.
- Chollet et al. (2025)
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. 2025.
Arc prize 2024: Technical report.
*arXiv preprint arXiv:2412.04604*.
- DeepMind (2025)
Google DeepMind. 2025.
Gemini 2.5 pro: Our most intelligent ai model.
[https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/).
Accessed 19 May 2025.
- DeepSeek (2024)
DeepSeek. 2024.
Introducing deepseek v3: 671b-parameter mixture-of-experts model.
[https://api-docs.deepseek.com/news/news1226](https://api-docs.deepseek.com/news/news1226).
Accessed 19 May 2025.
- DeepSeek (2025)
DeepSeek. 2025.
Deepseek r1 release.
[https://api-docs.deepseek.com/news/news250120](https://api-docs.deepseek.com/news/news250120).
Accessed 19 May 2025.
- Ermakova et al. (2025)
Liana Ermakova, Tristan Miller, Fabio Regattin, Antoine Bosselut, Saurabh Chaudhuri, Benoît Jeanjean, Sébastien Lefait, Stan Matwin, Véronique Moriceau, Patrick Saint-Dizier, and Lucas Vial. 2025.
Overview of the clef 2025 joker lab: Automatic humour processing–recognition, retrieval, and generation.
In *Working Notes of CLEF 2025 - Conference and Labs of the Evaluation Forum*.
- Google (2024)
Google. 2024.
Gemini 2.0 flash thinkingg.
[https://deepmind.google/technologies/gemini/flash-thinking](https://deepmind.google/technologies/gemini/flash-thinking).
- Gou et al. (2023)
Zhibin Gou, Zhihong He, Jian Wang, Xingyao Gao, Tong Wu, Jingze Bai, Juanzi Chen, Oriol Vinyals, Luke Perkins, Bing Sun, , and 1 others. 2023.
Tora: A tool-integrated reasoning agent for mathematical problem solving.
*arXiv preprint arXiv:2309.17452*.
- Hessel et al. (2023)
Jack Hessel, Ana Marasović, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023.
[Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest](https://doi.org/10.18653/v1/2023.acl-long.41).
In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 688–714, Toronto, Canada. Association for Computational Linguistics.
- Jain et al. (2020)
Lalit Jain, Kevin Jamieson, Robert Mankoff, Robert Nowak, and Scott Sievert. 2020.
The new yorker cartoon caption contest dataset.
- Kazemi et al. (2025)
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K Jain, Virginia Aglietti, Disha Jindal, Peter Chen, and 1 others. 2025.
Big-bench extra hard.
*arXiv preprint arXiv:2502.19187*.
- Lewkowycz et al. (2022)
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022.
Solving quantitative reasoning problems with language models.
*arXiv preprint arXiv:2206.14858*.
- Liang et al. (2025)
Tuo Liang, Zhe Hu, Jing Li, Hao Zhang, Yiren Lu, Yunlai Zhou, Yiran Qiao, Disheng Liu, Jeirui Peng, Jing Ma, and Yu Yin. 2025.
[When ’yes’ meets ’but’: Can large models comprehend contradictory humor through comparative reasoning?](https://arxiv.org/abs/2503.23137)
*Preprint*, arXiv:2503.23137.
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Tom Brown, Ben Mann, Sheer Agnihotri, Casey Möller, Nick Kalaitzis, and 1 others. 2023.
Let’s verify step by step.
*arXiv preprint arXiv:2305.20050*.
- Liu et al. (2023)
Yang Liu, Dan Iter, Yichong Xu, and Shuohang Wang Yelong Peng. 2023.
G-eval: Nlg evaluation using gpt-4 with better human alignment.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 11069–11081. Association for Computational Linguistics.
- Luo et al. (2023)
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2023.
Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.
*arXiv preprint arXiv:2308.09583*.
- McKenzie et al. (2023)
Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, and 1 others. 2023.
Inverse scaling: When bigger isn’t better.
*arXiv preprint arXiv:2306.09479*.
- Meta (2025)
Meta. 2025.
The llama 4 herd: The beginning of a new era of natively multimodal intelligence.
[https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
Accessed 19 May 2025.
- Min et al. (2020)
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.
[Ambigqa: Answering ambiguous open-domain questions](https://doi.org/10.48550/arXiv.2004.10645).
*arXiv preprint arXiv:2004.10645*.
Published as a long paper at EMNLP 2020.
- OpenAI (2024a)
OpenAI. 2024a.
Hello gpt-4o.
[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/).
Accessed 19 May 2025.
- OpenAI (2024b)
OpenAI. 2024b.
Introducing openai o1-preview.
[https://openai.com/index/introducing-openai-o1-preview/](https://openai.com/index/introducing-openai-o1-preview/).
- OpenAI (2025a)
OpenAI. 2025a.
Introducing openai o3 and o4-mini.
[https://openai.com/index/introducing-o3-and-o4-mini/](https://openai.com/index/introducing-o3-and-o4-mini/).
Accessed 19 May 2025.
- OpenAI (2025b)
OpenAI. 2025b.
Openai o3-mini.
[https://openai.com/index/openai-o3-mini/](https://openai.com/index/openai-o3-mini/).
- Quan et al. (2025)
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2025.
[Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings](https://arxiv.org/abs/2501.01257).
*arXiv preprint arXiv:2501.01257*.
- Rein et al. (2023)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023.
Gpqa: A graduate-level google-proof q&a benchmark.
*arXiv preprint arXiv:2311.12022*.
- Shao et al. (2024)
Zhihong Shao, Ganqu Lu, Jishuai Yu, Hang Wang, Cong Zheng, Yu Wang, Tao Zhao, Zherui Yang, Xipeng Chen, Xuetao Mao, Dan Su, and Zhilin Xu. 2024.
Deepseek-math: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*.
- Shi et al. (2023)
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023.
Large language models can be easily distracted by irrelevant context.
In *International Conference on Machine Learning*, pages 31210–31227. PMLR.
- Starace et al. (2025)
Matteo Starace, Adam Fisch, Nora Kassner, Jason Wei, Johannes Welbl, Marjan Ghazvininejad, Sebastian Riedel, and Jack Merullo. 2025.
Paperbench: Benchmarking large language models for scientific paper understanding and knowledge production.
*arXiv preprint arXiv:2501.12077*.
- Su et al. (2025)
Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. 2025.
Between underthinking and overthinking: An empirical study of reasoning length and correctness in llms.
*arXiv preprint arXiv:2505.00127*.
- Sun et al. (2025)
Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Lei Fang, and Ji-Rong Wen. 2025.
[Challenging the boundaries of reasoning: An olympiad-level math benchmark for large language models](https://arxiv.org/abs/2503.21380).
*arXiv preprint arXiv:2503.21380*.
- Sun et al. (2022)
Jiao Sun, Ajay Nagesh, and Zachary C. Lipton. 2022.
Expunations: Augmenting puns with keywords and explanations.
In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 2866–2879. Association for Computational Linguistics.
- Wang et al. (2023)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2023.
Self-consistency improves chain of thought reasoning in language models.
*International Conference on Learning Representations*.
- Wei et al. (2024)
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, William Fedus, Percy Liang, and Denny Zhou. 2024.
Emergent reasoning in large language models.
*arXiv preprint arXiv:2401.02642*.
- Xu et al. (2024)
Zixuan Xu, He He, Emmanuele Chersoni, Yi-Lin Tuan, and Qin Lu. 2024.
A good pun is its own reword: Evaluating pun recognition, explanation, and generation in llms.
In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4213–4228. Association for Computational Linguistics.
- Yang et al. (2024)
Liangyi Yang, Yihuai Wang, Xinyan Du, Zhekai Zhang, Haotian Shi, Zhengying Zhu, and 1 others. 2024.
Mathglm: Towards generalizable mathematical reasoning via language modeling.
*arXiv preprint arXiv:2401.14273*.
- Yang et al. (2025)
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. 2025.
Towards thinking-optimal scaling of test-time compute for llm reasoning.
*arXiv preprint arXiv:2502.18080*.
- Yue et al. (2023)
Xiang Yue, Boshi Ma, Yifan Wang, Junhong Chen, Xuanhe Gu, and Diyi Liang. 2023.
Mammoth: Building math generalist models through hybrid instruction tuning.
In *Advances in Neural Information Processing Systems*.
- Zhang et al. (2024)
Jifan Zhang, Lalit Jain, Yang Guo, Jiayi Chen, Kuan Lok Zhou, Siddharth Suresh, Andrew Wagenmaker, Scott Sievert, Timothy Rogers, Kevin Jamieson, and 1 others. 2024.
Humor in ai: Massive scale crowd-sourced preferences and benchmarks for cartoon captioning.
*arXiv preprint arXiv:2406.10522*.
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.
Judging llm-as-a-judge with mt-bench and chatbot arena.
In *Advances in Neural Information Processing Systems*, volume 36.
- Zhou et al. (2023)
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2023.
Least-to-most prompting enables complex reasoning in large language models.
*International Conference on Learning Representations*.
- Zhou et al. (2025)
Kuan Lok Zhou, Jiayi Chen, Siddharth Suresh, Reuben Narad, Timothy T Rogers, Lalit K Jain, Robert D Nowak, Bob Mankoff, and Jifan Zhang. 2025.
Bridging the creativity understanding gap: Small-scale human alignment enables expert-level humor ranking in llms.
*arXiv preprint arXiv:2502.20356*.

## Appendix A Main Benchmark Prompt

You are a humor expert extraordinaire, judging the New Yorker Cartoon Caption Contest. Your current task is to help us understand the humor in various submitted captions. Given a cartoon description and a caption submission, explain (in less than 200 words) *what* the joke is, focusing on the material substance of the joke.
STRICTLY use the format: <explanation>explanation goes here</explanation>

Cartoon description: description
Caption: caption

## Appendix B Autograder Prompt

You will receive:
1. A short cartoon description
2. A winning funny caption
3. A student’s answer
4. A brief “anticipated answer point” that captures the crucial comedic device or element

Your job is to determine whether the student’s answer **explicitly covers** that “anticipated answer point.”

- If the student’s answer captures or discusses the key comedic element (even if the wording is different), **PASS**.
- If the student’s answer **omits** or **contradicts** that key comedic element, **FAIL**.
- Do not penalize extra details or expansions. Synonyms or paraphrasing are acceptable if they convey the same comedic logic.
- Be mindful: if the anticipated answer point emphasizes something specific (e.g. a pun, wordplay, or ironic twist), check that the student’s answer includes it.

At the end of your evaluation, provide exactly two XML tags:
1. <reasoning>Short explanation of your thought process</reasoning>
2. <judgement>PASS or FAIL</judgement>

Do not include additional commentary or deviation from this format.

Cartoon description: {description}
Caption: {caption}
Student’s answer: {explanation}
Anticipated answer point:{anticipated point}

## Appendix C Hard Example Details

Table: Table 4: Representative examples from the hard subset where a majority of evaluated LLMs failed to identify all required humor elements.

## Appendix D Model performance on HumorBench Hard

**Table 5: Accuracy on the HumorBench-Hard subset (100 items).**
| Model | Accuracy (%) |
| --- | --- |
| o3 | 59.85 |
| Claude 3.7 Sonnet | 54.27 |
| gemini-2.5-pro-preview-03-25 | 52.44 |
| deepseek/deepseek-r1-zero | 51.22 |
| deepseek-ai/DeepSeek-R1 | 51.22 |
| o1 | 50.00 |
| o4-mini | 46.34 |
| Grok 3 | 45.12 |
| gpt-4o | 42.68 |
| deepseek-ai/DeepSeek-V3 | 39.63 |
| meta-llama/Llama-4-Maverick-17B-128E | 35.98 |
| gemini-1.5-pro | 35.37 |
| o3-mini | 32.93 |
| meta-llama/Llama-4-Scout-17B-16E | 29.88 |
| Qwen/Qwen2.5-72B-Instruct-Turbo | 26.83 |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - What is the criterion you use to distinguish between subjective and objective elements?
- Why did you choose “whether it contains a joke element” as the evaluation criterion for the autogravder, rather than evaluating the complete output?
- Do you have plans to extend the dataset to include a version with images?
soundness: 3
strengths: - During the annotation of the dataset, elements that could subjectively influence humor understanding were deliberately removed.
- The authors avoid using multiple-choice and ranking formats, as these can limit the model’s reasonable divergence in humor understanding. Additionally, fixed options may inadvertently hint at the actual punchline, making it unclear whether the model is reasoning or simply guessing. This is a reasonable and effective improvement.
- The authors designed a multi-round data refinement method that uses two large language models to alternately evaluate the data.
summary: This paper constructs a new benchmark based on the New Yorker Cartoon Contest and CartoonStock datasets, through re-annotation and the removal of images. Unlike previous datasets, HumorBench relies solely on objective elements that aid in humor understanding and excludes images, making it a cleaner benchmark for evaluating humor comprehension.
weaknesses: - Some steps in the dataset construction process are not described in detail. (See Questions.)
- The absence of images in the dataset may weaken its impact, as the main limitation of current multimodal large models lies in their inability to effectively understand humor from images.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See above.
soundness: 2
strengths: 1. HumorBench introduces a new non-STEM reasoning task that isolates objective humour understanding, avoiding confounding subjective funniness and providing a valuable probe of high-level reasoning.

2. Through extensive experiments on frontier LLMs, the study reveals clear transfer from STEM reasoning to humour comprehension and mixed effects of test-time scaling, demonstrating both the benchmark’s sensitivity and the cross-domain generality of reasoning abilities.
summary: This paper presents HumorBench, a benchmark designed to evaluate the ability of large language models (LLMs) to reason about and explain sophisticated humour in cartoon captions. It includes approximately 300 unique cartoon-caption pairs from the New Yorker Caption Contest and Cartoonstock.com, with expert-annotated evaluation rubrics identifying essential joke elements. The authors also conducted a rich experiment by benchmarking current SOTA models and provided several insights for future work.
weaknesses: 1. Besides identifying individual elements, the evaluation should also consider the interactions or causal relations among them. Humour understanding depends on how well the model connects these elements, not just mentions them. Element-based evaluation can serve as a supplement. In fact, using an LLM as a judge to compare the response with the golden label can already effectively reflect joke understanding.

2. This conclusion is reasonable, but the paper does not explain why reasoning models improve humour understanding. The causes should be analysed further: Is the gain due to longer thinking tokens, potential training data leakage, or emergent self-search/retrieval behaviour of the model?

3. Although the text-only setup makes the task more controllable and annotations clearer, it may oversimplify the problem. Under this setting, GPT-4o already achieves around 92% accuracy, suggesting that the task ceiling may be too low to differentiate between stronger models.

4. Models with access to external tools (e.g., web search) may perform better in humour reasoning. Since the paper mentions that many hard cases involve hidden concept jumps or obscure references, evaluating models equipped with such tools would provide more meaningful insights.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See Weaknesses
soundness: 2
strengths: - The introduction of a rubric-based evaluation framework for humor understanding is novel and well-motivated.
- The proposed HumorBench benchmark has the potential to offer valuable insights and guide future research in multimodal humor understanding.
- The experimental design and analyses are comprehensive, effectively highlighting the limitations and challenges faced by current MLLMs in this domain.
summary: This paper introduces HumorBench, a new benchmark designed for evaluating humor understanding in models. HumorBench comprises 300 unique cartoon–caption pairs collected from the New Yorker Caption Contest and Cartoonstock.com, each annotated with human-defined rubrics. Unlike prior humor understanding benchmarks, HumorBench decomposes humor into several objective elements and assesses a model’s generated explanations against these elements using an automatic grading system. This approach aims to measure objective comprehension of humor rather than subjective interpretations. The authors evaluate several recent multimodal large language models (MLLMs) on HumorBench and present analyses that reveal their relative performance and limitations.
weaknesses: - The benchmark size is relatively small, containing only about 300 cartoons drawn from a limited range of sources (mainly The New Yorker Caption Contest and Cartoonstock.com). This restricted scope may limit the dataset’s generalizability and future applicability;
- While the authors argue that using humor elements as rubrics enables objective humor understanding, this claim raises concerns. The definition and categorization of these humor elements may be subjective and culturally biased, as different annotators or readers could conceptualize these humor elements differently. The paper would benefit from a clearer justification of how these elements were defined and whether they adequately capture the diversity of humor mechanisms. Moreover, important annotation details are missing—including the number and backgrounds of annotators (particularly their cultural contexts), and whether inter-annotator agreement was measured to ensure annotation reliability;
- The paper reports a correlation between STEM reasoning ability and humor understanding, but this interpretation may be confounded. The observed correlation could simply reflect that stronger MLLMs perform better across both STEM and Humor reasoning domains, rather than implying a causal link. Without a controlled experiment, for instance, fine-tuning the same base model on STEM data and comparing its humor understanding performance, the claim that enhancing STEM reasoning improves humor comprehension remains unsubstantiated;
- Regarding Ethics Statement: Including the background of annotators and how you ensure the annotations are not biased.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1.	The benchmark currently relies on textual descriptions of cartoons rather than the original images. How might removing the visual modality bias the evaluation? Do you plan to extend HumorBench into a multimodal setting?
2.	How do you think HumorBench generalizes across different cultures and humor traditions? Do you envision multilingual or cross-cultural extensions?
3.	Do you have hypotheses about why this “inverse scaling” occurs, and what it implies about LLM reasoning dynamics?
4.	Would you incorporate both objective and subjective dimensions in future benchmarks?
soundness: 3
strengths: 1.	The paper identifies a real gap in LLM evaluation: existing benchmarks overemphasize STEM reasoning, while humor demands cultural, linguistic, and inferential reasoning.

2. The dataset is constructed with rigorous curation, including expert validation and removal of inconsistent annotations, contributing to dataset quality and reliability.

3.	It provides systematic evaluation, tests across many models, compares base vs. reasoning optimized variants, and analyzes correlations with other benchmarks.
summary: This paper introduces HumorBench, a benchmark designed to evaluate LLM ability to reason about and explain sophisticated humor in cartoon-caption settings. It highlights the limitations of existing humor datasets, which often conflate subjective funniness with objective joke comprehension, and instead proposes a framework focused on objective, verifiable joke elements. To address this gap, the authors curate about 300 cartoon-caption pairs from the New Yorker Caption Contest and Cartoonstock, annotate each with core joke elements, and build an autograder to assess whether LLM explanations capture those elements. Through experiments with frontier models, the study shows (1) strong transfer of reasoning skills from STEM benchmarks to humor comprehension, (2) surprising competence of models trained only on STEM reasoning tasks, and (3) mixed effects of test-time scaling.
weaknesses: 1.	The size of the dataset may be limited. Although curated carefully, about 300 cartoon-caption pairs (499 elements) is relatively small compared to other reasoning benchmarks, potentially limiting generalization.
2.	Need further justification for the dataset diversity, especially w.r.t. cultural background and humor type. Considering the original sources are largely Western (New Yorker, Cartoonstock), and the dataset may not test models across diverse humor traditions.
3.	The validation shows a leniency bias (higher false positive rate), meaning scores are optimistic upper bounds. The reliance on LLM-as-judge could compromise fairness and transparency.
4.	While the benchmark shows transfer from STEM reasoning, the interpretation sometimes overstates the connection without fully probing why abstract reasoning transfers across domains.
5.	The explanations of why models fail certain jokes remain somewhat descriptive rather than deeply diagnostic.