Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps

## Abstract

Abstract Humor is a culturally nuanced aspect of human language that presents challenges for understanding and generation, requiring participants to possess good creativity and strong associative thinking. Similar to reasoning tasks like solving math problems, humor generation requires continuous reflection and revision to foster creative thinking, rather than relying on a sudden flash of inspiration like Creative Leap-of-Thought (CLoT) paradigm.
Although GPT-o1 can realize the ability of remote association generation, this paradigm fails to generate humor content.
Therefore, in this paper, we propose a systematic way of thinking about generating humor and based on it, we built Creative Leap of Structured Thought (CLoST) frame. First, a reward model is necessary achieve the purpose of being able to correct errors, since there is currently no expert model of humor and a usable rule to determine whether a piece of content is humorous. Judgement-oriented instructions are designed to improve the capability of a model, and we also propose an open-domain instruction evolutionary method to fully unleash the potential. Then, through reinforcement learning, the model learns to hone its rationales of the thought chain and refine the strategies it uses. Thus, it learns to recognize and correct its mistakes, and finally generate the most humorous and creative answer.
These findings deepen our understanding of the creative capabilities of LLMs and provide ways to enhance LLMs’ creative abilities for cross-domain innovative applications.

## 1 Introduction

Currently, humor is becoming increasingly prevalent because it not only alleviates stress and enhances interpersonal relationships but also brings a sense of ease and optimism in complex environments. The content of humor usually involves providing a creative answer or explanation to a specific question or phenomenon. However, even for humans, generating novel ideas is very challenging, primarily due to difficulties in creative association, making leaps in thinking, understanding punchlines, and determining whether they are humorous.

Figure: Figure 1: English Comparison Showcase (more showcases are in Appendix A.3). Compared to GPT-4o and CLoT, CLoST provides shorter and more conversational answers to questions. For instance in Case 2, while CLoST and CLoT may convey the same meaning, their different expressions produce different effects. Brief responses leave room for readers to ponder, enhancing interest and interactivity.
Refer to caption: x1.png

The Creative Leap of Thought (CLoT) paradigm, as proposed by Zhong et al. (), extends the Leap-of-Thought (LoT) paradigm (Bai et al., ; Kim et al., ) to enhance the model’s judgement ability and the connections between concepts by incorporating a seemingly unrelated concept into the input. This approach aims to boosting the creative thinking and the ability to make leaps in thought in Large Language Models (LLMs).
However, similar to how a human may think for a long time before responding to a difficult question and methods of reasoning tasks like solving math problems (Lightman et al., ), a flash of inspiration for humor generation like Creative Leap-of-Thought (CLoT) paradigm is not enough, it also requires continuous reflection and revision to foster creative thinking (Lightman et al., ).

Therefore, we’ve come up with a systematic way of thinking about generating humor. Inspired by knowledge graphs (Sun et al., ; Yang et al., ), where entities represent external correlations and their diverse relationships reinforce logical structures, we introduce the Creative Leap of Structured Thought (CLoST). CLoST enhances LLMs with advanced creative thinking, associative reasoning, and logical deduction through two stages: Automatic Associative Instruction Evolution (AAIE) and Guided Exploratory Self-Improvement Tuning (GESIT).
In AAIE, we develop human-designed judgement-related instructions and their derivatives to train the model’s judgement capabilities of humor. Additionally, an automatic instruction evolution method on humorous dialog are proposed to unleash the potential of LLMs in humor (Luo et al., ; Zhu et al., ; Xu et al., ) and boosting their creative thinking abilities based on a deeper understanding.

The second stage, GESIT, focuses on enhancing the model’s ability to generate humorous responses and understand humorous styles through bootstrap Direct Preference Optimization (DPO) (Rafailov et al., ) training. In this phase, the model learns from both an expert who provides rationales for jokes in dialogue to improve LLM performance (Zelikman et al., ), and from the model developed through AAIE, which has acquired judgement capabilities. Models trained with ”scratchpads” for intermediate reasoning steps achieve superior performance and better generalization compared to those trained to predict answers directly. Therefore, in GESIT, we strengthen the model’s ability to connect humorous responses with questions by iteratively guiding it to produce high-quality rationales through both expert guidance and its own generated outputs.

We evaluated various language models’ ability on humor discrimination using both Chinese and English humor datasets. CLoST outperformed other models on almost all the test sets. Additional experiments demonstrated that CLoST enhances the model’s divergent thinking ability and effectiveness. Our contributions are summarized as follows:

- 1.
We introduce a training framework called Creative Leap of Structured Thought, based on evolvable instruction tuning to unleash the full potential of large language models (LLMs). Additionally, we incorporate guided self-training DPO to enhance the logical relationship between questions and answers.
- 2.
Extensive evaluation on both Chinese and English benchmarks demonstrates the effectiveness and generality of CLoST, also enhancing the ability of humor discrimination at the state-of-the-art level .

## 2 Method

### 2.1 Problem Formulation

In general, the knowledge graph $\mathcal{G}$ is defined as a set of triples $\mathcal{G}=\{(e,r,e^{\prime})\mid e,e^{\prime}\in\mathcal{E},\ r\in\mathcal{R}\}$, where $\mathcal{E}$ is the set of entities and $\mathcal{R}$ is the set of relations. Each triple represents a relation $r$ from the head entity $e$ to the tail entity $e^{\prime}$ (Sun et al., ; Yang et al., ). In the special application of humor generation, we consider a knowledge graph composed of question-related entities $\mathcal{E}_{q}$ and answer-related entities $\mathcal{E}_{a}$, where the intersection $\mathcal{E}_{Z}=\mathcal{E}_{q}\cap\mathcal{E}_{a}$ is regarded as confounding variables (we refer to it here as confounding entities). Assuming the existence of causal relationships $\mathcal{E}_{Z}\rightarrow\mathcal{E}_{q}$ and $\mathcal{E}_{Z}\rightarrow\mathcal{E}_{a}$, we use the intervention formula $P(\mathcal{E}_{a}\mid do(\mathcal{E}_{q}))=\sum_{e_{z}\in\mathcal{E}_{Z}}P(
\mathcal{E}_{a}\mid\mathcal{E}_{q},\mathcal{E}_{Z}=e_{z})\cdot P(\mathcal{E}_{
Z}=e_{z})$ to show how to use confounding entities $\mathcal{E}_{Z}$ to get an answer.
Our goal is to utilize this causal relationship to generate creative and logical answers, while noting that the confounding entities $\mathcal{E}_{Z}$ are interconnected with $\mathcal{E}_{q}$ and $\mathcal{E}_{a}$ through the relations $\mathcal{R}_{qa}$ such as example in Figure 12 in Appendix A.2.

The overall training framework is illustrated in Figure 4. In the first stage, we train a model with judgement and divergent thinking abilities using human-designed and automatically evolving instructions. In the second stage, we employ this model as a humor judgement expert as well as an expert teaching rationales to generate online preference data for DPO training.

### 2.2 Associative Automatic Instruction Expansion

In this section, we mainly introduce the design of the discrimination template and how to design open-domain instructions to unleash the potential of LLMs. (Luo et al., ; Zhu et al., ; Xu et al., ).

Figure: Figure 2: The details of judgement-oriented instructions templates.
Refer to caption: x2.png

Discrimination Template Design.
Regarding humor discrimination, we designed multiple-choice questions based on the dataset introduced in Section 3.1 to enhance the humor discrimination ability of LLMs, specifically the selection skill. Additionally, since the dataset includes human preference annotations, such as the number of ’likes’ voted among multiple candidate responses, we designed ranking questions to improve another discrimination skill, namely ranking ability. Based on these two foundational skills, we designed various enhancement tasks to improve the model’s judgement of humor.

For multiple-choice questions in Figure 2(a), we introduce dataset-specific data processing methods (see Section 3.1) and adapt them to the ”task-specific prompt”.
For ranking ability in Figure 2(b), we construct them exclusively using data with user preferences. These tasks require the LLM to order multiple responses to a given question based on human preferences. By training on both multiple-choice and ranking task, we enable the LLM to identify humorous responses and align with human preferences, thereby enhancing its ability to discriminate, select, and rank humorous content.

In addition, we created two more tasks to enrich judgement instructions, further enhancing the model’s judgement capabilities.

(1) Answer Rewrite (see Figure 2(d)). Transforming a Non-Humorous Statement into a Humorous Expression.

(2) Teacher-Student Model Prompt Loop (see Figure 2(c)). We utilized two GPT-4o models, designating one as the teacher and the other as the student. The teacher provides prompts based on the given question and the ideal response. The student generates an answer using both the teacher’s prompts and the original question. If the student’s response does not align with the original witty response in terms of logic and humor, the teacher supplies additional prompts; otherwise, the prompting process ends. After a maximum number of attempts, the student’s response is accepted as the ground truth, and the teacher replies, ”Congratulations, you got it right.” At any intermediate stage, the teacher provides negative feedback if the student’s answer is incorrect.

Figure: Figure 3: Overall architecture of Associative Automatic Instruction Evolution.
Refer to caption: x3.png

Associative Automatic Instruction Evolution.
Human-designed instructions for model training like previous part are called closed-domain instructions, which have two main drawbacks. First, all samples in an dataset typically share only a few common instructions, which severely limits their diversity. Second, these instructions usually address only a single task. In contrast, real-life human instructions often involve multiple and diverse task demands (Xu et al., ).
Therefore, we use large language models (LLMs) instead of humans to generate extensive instruction data with varying complexities. This approach aims to maximize the range of knowledge covered in conversations.

Given a dataset $D=\{(q_{k}^{0},a_{k}^{0})\}_{k=0}^{N}$, where $q_{k}^{0}$ is a question, $a_{k}^{0}$ is a funny answer, and $I=\{i_{0},y_{0},e_{0}\}$, where $i_{0}$ is the instruction, $y_{0}$ is the reply based on $i_{0}$ and $e_{0}$ is the initial rule to make instructions complicated.
As shown in Figure 3, this framework involves three agents, each playing the roles of rewriter, imaginator, and analyst. Here’s how the process works:

1.Rewriter Agent: Given a conversation $(q_{k},a_{k})$ and an instruction $i_{0}$, such as ”What is the punchline of this response?”, the rewriter agent transforms it into a more complex version $i_{1}$ based on an initial rule $e_{0}$ (see Appendix A.1). And then the agent provides a reply $y_{1}$ based on the complex version of the instruction.

2.Imaginator Agent: The imaginator agent then takes the complex instruction $i_{1}$ and the reply $y_{1}$ to imagine a new conversation $(q_{1},a_{1})$.

3.Analyst Agent:
Finally, both the original conversation $(q_{k}^{0},a_{k}^{0})$ and the imagined conversation $(q_{k}^{1},a_{k}^{1})$ are sent to the analyst agent. The analyst evaluates ”whether the two conversations express the same viewpoint or punchline?”. If they do, the interaction between the agents ends.

If not, the analyst applies a second criterion: ”Is the instruction $i_{1}$ more complex than the initial instruction $i_{0}$?” If the answer is ”yes,” the original conversation is replaced with the imagined conversation and sent back to the rewriter, allowing the three agents to continue their interaction based on the imagined conversation. If the answer is ”no,” the analyst agent formulates a more complex rule $e_{1}$ and sends it to the rewriter. This process continues until the imagined and conversations in last round express different viewpoint or punchline, or until a maximum number of communication rounds $m_{k}$ is reached. Finally, the system outputs $D^{\prime}=\{(q_{k}^{0},a_{k}^{0},i_{0},y_{0},\dots,q_{k}^{m_{k}},a_{k}^{m_{k}
},i_{m_{k}},y_{m_{k}})\}_{0}^{m_{k}\times N}$, where $m_{k}$ is the maximum number of communication rounds between the three agents.

This iterative process ensures that instructions become increasingly complex, as shown in Appendix A.1, and that the model is trained on a diverse set of scenarios. This enhances its ability to handle a wide range of conversational topics and complexities. Additionally, the process covers conversational content across different knowledge domains, thereby expanding the entities within the large language model’s knowledge graph. Finally, we use these data to train a LoRA model named $J$.

### 2.3 Guided Explorative Self-Improvement Tuning

In the previous section, we developed model $J$, which possesses judgement and divergent capabilities. To enhance the graph’s accuracy and interpretability, we incorporate additional causal principles through online Direct Preference Optimization (DPO) (Rafailov et al., ). By increasing the number of entities and explicitly modeling the causal relationships ($\mathcal{R}_{cause}\subseteq\mathcal{R}_{qa}$), the model gains a deeper understanding of how changes in one entity influence others.

Figure: Algorithm 1 algorithm of GESIT

The algorithm details are provided in Algorithm 1. We start with an initial offline dataset without rationales, $D_{0}=\{(q_{i},a_{i}^{+},a_{i}^{-})\}_{i=0}^{M}$, and a judgement model $J$. During training, the policy model $\pi$ selects questions $q$ from the original dataset and, every $\frac{N}{T_{P}}$ steps, generates two responses independently. The judgement model then evaluates which of the two answers is more humorous. Additionally, an expert (GPT-4o as $E$) in causal reasoning provides a rationale for each answer in relation to the question (see Appendix A.2 for details). This process expands the preference dataset to $D=D_{0}\cup D_{\text{cand}}$, where $D_{\text{cand}}=\{q,\widetilde{a}^{+},\widetilde{a}^{-},\widetilde{r}^{+},
\widetilde{r}^{-}\}$.

Figure: Figure 4: Network architecture of training frame.
Refer to caption: x4.png

## 3 Experiments

In this section, we first construct a humor-related dataset in both Chinese and English, organize it into an appropriate format, and evaluate the performance of our method by comparing it with multiple large language models (LLMs) through various validation experiments.

### 3.1 Dataset and Task

Datasets.
Currently, there is limited data on humor-focused question-answer pairs. To enrich the English dataset, we collected various humor-related datasets, including short jokes and headline wordplays. We compiled English humor data from three sources—Oogiri-GO, SemEval 2020, and SemEval 2021—and organized them into the required format as described below. For Chinese data, due to the scarcity of relevant datasets beyond Oogiri-GO, we supplemented the training and testing sets with internally sourced data.

(1) Oogiri-GO (Zhong et al., ). In the ”Oogiri game”, participants need to give unexpected and humorous responses to given images, text, or both. This game requires a sudden burst of insight and strong associative thinking for the given context, and provides an ideal platform for evaluating the “Leap-of-Thought” (LoT) ability of large language models. Similar to the processing method of CLoT, we randomly select $95\%$ of the samples to construct the training dataset, and the remaining $5\%$ form the test dataset for validation and analysis. And also inspired by CLoT, we use GPT-4o to imitate the ground truth response to form a pair of comparison data.

(2) SemEval-2021 Task 7 Meaney et al. (). This dataset contains labels and ratings collected from a balanced age group ranging from 18 to 70 years old. The data relevant to our purpose includes:
Humor Discrimination: Whether the text would be considered humorous (for an average user). This is a binary label.
Humor Rating: If the text is classified as humorous, predict its degree of humor (for an average user). The values vary between 0 and 5.
As above, take the samples of humor discrimination as the comparison data for easy cases, and take the data of humor rating as hard cases. The training dataset and test set are built in the same way.

(3) SemEval-2020 Task 7 (Hossain et al., ). The dataset of this task contains news headlines in which short edits have been made to make them funny. The funniness of these edited headlines is rated through crowdsourcing. This task includes two subtasks. The first subtask is to estimate the funniness degree of headlines on a humor scale ranging from 0 to 3. The second subtask is to predict, for two edited versions of the same original headline, which one is the funnier version. The data construction method is the same as that of ’SemEval-2021 Task 7’.

(4) In-house Data. We use internal data that includes preference scores and organize these scores accordingly. For easy cases, we select comments with the highest and lowest scores as comparison data. For hard cases, we choose comments with smaller score differences. The test data is structured in the same manner.

Tasks.
Inspired by the task design in CLoT(Zhong et al., ), we develop multiple choice questions of varying degrees of difficulty are used to test the judgement ability of the model. Abbreviated as $mTn$, these questions require large language models to select $n$ humorous responses from $m$ options given an input. Here, we construct four types of $mTn$ questions, including $2T1$, $3T1$, $4T1$, and hard case $2T1$. $2T1$ represents two options, namely the response with the highest score and the response with the lowest score (generally equal to 0), which is also humor discrimination. The hard case $2T1$ is a choice between responses with relatively close scores. $3T1$ and $4T1$ add responses with intermediate scores respectively. For these questions, their difficulty gradually increases and they are diverse to ensure comprehensive evaluation. The top-1 accuracy is the evaluation metric. We did not specifically set up a ranking test task because our hard case multiple-choice questions can to some extent represent the ability of ranking.

#### 3.1.1 Implementation Details

Our model is fine-tuned based on QWEN1.5-32B-Chat Bai et al. () with fine-tuning method LoRA Hu et al. () on 8 A100 GPUs. For the first stage, we train the model on the $95\%$ of dataset mentioned above for 6 epochs with AdamW optimizer and the learning rate of $3e-4$. In the second stage, $5\%$ of the dataset are used to train GESIT for 3 epochs with AdamW optimizer and the learning rate of $2e-4$. The models are tested in the tasks introduced in previous part.

### 3.2 Results Analysis

Evaluation by Choice Questions in English Tasks.
The top-1 accuracy of completing each selection task and show the performance of several models in Table 1. Overall, compared with open-source language models including LLAMA3 and QWEN, the state-of-the-art closed-source large language models show impressive zero-shot performance on humor discrimination and rank tasks. By training on the English task instruction data designed by us, our model CLoST has a significant improvement compared with other models (such as LLAMA3-70B and GPT-4o) (with an average accuracy in diverse English benchmark increase of 4.55% and 5.91%).

Evaluation by Choice Questions in Chinese Tasks.

We also evaluate the accuracy rate (acc%) of completing each selection task in Chinese and show the performance of several models in Table 2. Overall, compared with open-source language models including LLaMA3 and QWEN, the state-of-the-art closed-source large language models show impressive zero-shot performance on humor discrimination and rank tasks. By training on the Chinese task instruction data designed by us, the CLoST has a significant improvement compared with other models (such as GPT-4o) (with an average accuracy in diverse Chinese benchmark increase of 16.22%).

**Table 1: The accuracy (%) of choice questions on various Algorithms in English benchmarks.**
| Model | SemEval 2021 | SemEval 2020 | Oogiri-GO |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 2T1 | 2T1(hard) | 3T1 | 4T1 | 2T1 | 2T1 |  |  |
| GPT | 4o | 85.09 | 60.77 | 43.71 | 34.63 | 55.08 | 85.09 |
| LLAMA3 | 8B | 43.85 | 54.23 | 39.81 | 29.57 | 59.93 | 72.05 |
| 70B | 93.60 | 58.08 | 39.81 | 31.82 | 60.73 | 88.51 |  |
| QWEN1.5 | 7B | 62.04 | 52.02 | 31.54 | 24.89 | 50.46 | 36.65 |
| 14B | 82.05 | 51.04 | 30.92 | 24.24 | 50.38 | 53.73 |  |
| 32B | 68.01 | 52.57 | 35.38 | 28.79 | 56.39 | 68.01 |  |
| QWEN2 | 7B | 56.55 | 50.63 | 32.31 | 23.38 | 50.08 | 62.11 |
| 57B | 83.30 | 52.02 | 37.08 | 28.79 | 48.29 | 48.14 |  |
| QWEN2.5 | 32B | 94.00 | 55.22 | 34.77 | 27.92 | 58.71 | 81.68 |
| Baichuan2 | 13B | 51.70 | 52.71 | 35.69 | 24.24 | 51.45 | 50.00 |
| CLoT |  | 52.49 | 51.74 | 34.46 | 23.59 | 53.50 | 52.49 |
| OURS | 32B | 96.58 | 57.45 | 48.06 | 35.90 | 64.57 | 97.20 |

**Table 2: The accuracy (%) of choice questions on various Algorithms in Chinese benchmarks.**
| Model | Chinese Benchmark |  |  |
| --- | --- | --- | --- |
| 2T1 | 2T1(hard) |  |  |
| GPT | 4o | 64.98 | 63.49 |
| LLAMA3 | 8B | 50.72 | 57.44 |
| 70B | 59.48 | 61.22 |  |
| QWEN1.5 | 7B | 54.82 | 51.71 |
| 14B | 53.45 | 57.41 |  |
| 32B | 52.71 | 56.27 |  |
| QWEN2 | 7B | 51.99 | 58.17 |
| 57B | 65.91 | 57.03 |  |
| QWEN2.5 | 32B | 61.53 | 60.46 |
| Baichuan2 | 13B | 50.56 | 53.61 |
| Oogiri-GO-ch |  | 52.12 | 34.46 |
| OURS | 32B | 90.95 | 69.97 |

Evaluation on Creative Task.
To evaluate the associative generalization capability of CLoST, we tested it on a creative task known as the Divergent Association Task (DAT) (Olson et al., ). DAT is a classic creativity test that requires participants to select words with large semantic distances from a set of 10 unrelated nouns. In the Chinese creativity test, we utilized Chinese Word Vectors (Li et al., ) to calculate the DAT score. First, we provided specific words and asked the model to generate associations and imaginations, resulting in 10 associated words. We then used these 11 words to calculate the DAT score. Additionally, we tested how the model’s DAT score varied as the number of test words increased, and, we tested the divergent ability of association on professional vocabulary in different fields (animal, financial and place name). It can be seen that as the number of words increases, the DAT score also tends to stabilize. It can be seen from the average of each domain that CLoST has the highest score after stabilization. As shown in Figure 5, CLoST has a slight improvement in performance on the mean value of DAT compared to Qwen1.5-32B and GPT-4o.

Furthermore, we utilize T-SNE to project the embedding vectors of these words into a two-dimensional space. Figures 5(a) and (b) position the target word at the center and draw a circle with a radius equal to the Euclidean distance between the target word’s embedding and that of the farthest associated word. Figures 5(c) and (d) illustrate the embedding vectors of five target words and their associated words, with different colors representing different target vectors. A larger circle indicates a broader semantic range for the target word, suggesting a greater number of associated words. In both tests, CLoST demonstrates an advantage.

**Table 3: Ablation on English benchmarks.**
| Model | SemEval 2021 | Oogiri-GO-en | Chinese Benchmark |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 2T1 | 2T1(hard) | 3T1 | 4T1 | 2T1 | 2T1 | 2T1(hard) |  |
| QWEN1.5-32B | 68.01 | 52.57 | 35.38 | 28.79 | 54.82 | 52.71 | 56.27 |
| +Figure 2(c) | 55.12 | 53.27 | 33.54 | 27.49 | 81.68 | 63.56 | 63.88 |
| +Figure 2(c,d) | 89.70 | 50.28 | 30.46 | 17.32 | 95.96 | 83.62 | 64.64 |
| AAIE | 88.40 | 50.07 | 30.92 | 19.48 | 96.58 | 90.34 | 64.64 |

Figure: Figure 5: DAT score of CLoST, QWEN1.5-32B, GPT-4o and CLoT in Chinese in different domain. (a). Words related to Animal. (b). Words related to Finance and Economics. (c). Words related to place name. (d). Mean score comparison.
Refer to caption: x5.png

Figure: Figure 6: (a) GPT-4 and GPT-4o logically evaluates the output of the model after AAIE and CLoST respectively. And associative range comparison: (b). 500 target words. (c). TSNE Results of Word Vectors Obtained by QWEN1.5-32B Associating Five Target Words. (d). TSNE Results of Word Vectors Obtained by CLoST Associating Five Target Words.
Refer to caption: x9.png

Ablation Study

We examine the ablation effects of different components in CLoST and report the performance results in Table 3. Additionally, we validate the model’s generalization ability on a ranking task after training on the humor discrimination task. In other words, we train the model solely on the Oogiri-GO training set, which contains only easy case comparison data. As the number of tasks increases while maintaining a $1:1$ data ratio per task, the model’s performance on the Oogiri-GO test set and the easy case choice task in SemEval 2021 progressively improves, as shown in the first and fourth columns. However, performance on the SemEval 2021 hard cases declines, indicating reduced generalization ability. This occurs because the Oogiri-GO dataset consists solely of easy cases, causing the model to perform better on data with clear boundaries. Furthermore, since GESIT primarily enhances the learning of preferences and causal logic relationships, it generally does not improve performance on judgement tasks.

To evaluate the performance of GESIT, which primarily enhances the model’s causal and logical reasoning capabilities (i.e., the ability to relate to internal contexts), we enlisted three experts to assess the logical coherence of responses on 200 examples from GPT-4o, GPT-4 and QWEN2-57B, respectively. The experimental results shown in Figure 6(a) demonstrate that incorporating GESIT significantly strengthens the logical reasoning in the model’s answers.

### 3.3 Related Works

Large Language Models (LLMs) and Their Creativity. Recently, language models (Bai et al., ; Wang et al., ; Liu et al., ; Chen et al., ) have garnered widespread attention due to their impressive reasoning capabilities (Wang et al., ; Saparov & He, ; Zeng et al., ; Driess et al., ; Dong et al., ; Ye et al., ; Liang et al., ). Additionally, an increasing number of studies are focusing on exploring the creativity of LLMs (Ling et al., ; Summers-Stay et al., ; Sun et al., ; Bhavya et al., ), with applications in fields such as scientific discovery (Park et al., ; Kang et al., ; Hope et al., ; Liang et al., ; Huang et al., ) and creative writing (Swanson et al., ; Chakrabarty et al., ; Wu et al., ; Mirowski et al., ; Dang et al., ).

Computational humor is a branch of computational linguistics and artificial intelligence that utilizes computers to study humor (Binsted et al., ). It encompasses various tasks, including humor discrimination (Shahaf et al., ; Tanaka et al., ; Xu et al., ; Chen & Zhang, ; Kumar et al., ; Wu et al., ; Ofer & Shahaf, ; Xie et al., ; Meaney et al., ; Hossain et al., ), humor interpretation (Hwang & Shwartz, ; Evans et al., ; Vásquez & Aslan, ), and humor generation (Amin & Burghardt, ; Zhang et al., ; Hossain et al., ; Valitutti et al., ; Chaudhary et al., ). With the advancements in generative LLMs, humor generation has become a hot focus. However, humor generation still faces challenges such as insufficient punchlines (Popova & Dadic, ).

Chain-of-thought prompting methods provide models with a ”chain of thought” (Kojima et al., ; Wei et al., ; Zhang et al., ; Yao et al., ; Long, ), which are reasoning examples (Wei et al., ), or a simple prompt like ”let’s think step by step” (Kojima et al., ), to encourage LLMs to engage in reasoning rather than just providing direct answers (Huang & Chang, ).

Instruction tuning has emerged as a key strategy for unlocking the potential of large language models (LLMs). By curating high-quality datasets, we can more effectively align these models with desired behaviors (Zhou et al., ). However, the challenge of expanding high-quality instruction data remains a central research focus. Some researchers prioritize human annotation to create instruction data, as seen in projects like OpenAssistant (Köpf et al., ). (Zeng et al., ; Xu et al., ) explore more efficient methods such as iterative approach for optimizing instruction-following data, generating more complex and diverse datasets, to surpass the quality limits of existing datasets in math and code problems. Since the content of the conversation varies widely, it is difficult to apply uniform rules to complicate the explanation. Therefore, the above approach may not work.

Preference Optimization. Aligning instruction-tuned language models with preference data has been shown to improve performance in both proprietary (Ouyang et al., ; Touvron et al., ) and open-source models. To facilitate further research, several preference datasets have been released recently (Bai et al., ; Cui et al., ). Algorithmically, simpler variants of standard Reinforcement Learning from Human Feedback (RLHF) have been proposed, such as reward-free methods that directly optimize using initial preference data (Rafailov et al., ; Azar et al., ; Ethayarajh et al., ) and iterative updates of preference data with gold annotators (Guo et al., ; Touvron et al., ). Alternative annotation strategies include using learned reward models (Zhao et al., ; Liu et al., ; Calandriello et al., ) or leveraging the policy model itself for preference labeling (Lee et al., ; Yuan et al., ). (Ahmadian et al., ) studied more stable reinforcement learning algorithms. However, these methods either ignore the distribution shift of policy models and its impact on reward model performance or rely entirely on expensive gold annotations during training. Our work bridges this gap by introducing a cost-effective method to prevent reward model degradation during policy training.

### 3.4 Conclusion

In this paper, we propose the Creative Leap of Structured Thought (CLoST) method to enhance the generation capabilities of large language models (LLMs). CLoST begins with transforming humor datasets into instruction-tuning data to train LLMs, thereby improving their LoT and judgement abilities. Subsequently, CLoST employs Guided Explorative Self-Improvement, which enables LLMs to generate more creative structured thought data through an understanding of rationales and to select high-quality data for self-refinement training. Experimental results demonstrate the effectiveness and generalization ability of CLoST across various tasks, including witty response generation and humor discrimination.

## Appendix A Appendix

### A.1 Detailed in AAIE

To illustrate the challenges encountered during the data evolution process, we designed an initial instruction (Figure 9 Turn 0) for empirical analysis. This instruction was subsequently evolved using the initial evolution method (Figure 7), resulting in the evolved instructions shown in Figure 9 Turn n. Our analysis identified several shortcomings of the initial evolution method. Specifically, it does not fully account for the inherent complexity of the evolved instructions, leading to incomplete answers that lack necessary background information and fail to consider factors such as social culture and psychological theory. Consequently, the problem comprehension in the answers remains superficial, hindering genuine divergent thinking.

To address these issues, we developed an improved evolution method through Analytic Agent analysis, as illustrated in Figure 8. In this figure, the blue elements represent the evolution method after the first correction, and the red elements represent the method after the second correction. This enhanced approach ensures more comprehensive and contextually aware responses, thereby fostering deeper understanding and more effective divergent thinking.

Figure: Figure 7: Initial Evolution Method.
Refer to caption: x13.png

Figure: Figure 8: Complex Evolution Method.
Refer to caption: x14.png

Figure: Figure 9: Changes in instructions.
Refer to caption: x15.png

### A.2 Template in GESIT

Inspired by (Vashishtha et al., ), to guide experts to output causality, we designed a template as shown in Figure 10. The output of the expert is stricted as Figure 11.

Figure: Figure 10: Prompts for Expert $E$.
Refer to caption: x16.png

Figure: Figure 11: Output of Expert $E$.
Refer to caption: x17.png

Figure: Figure 12: Rationale Showcase.
Refer to caption: x18.png

### A.3 Generation Showcase

We found some posts online and used the Algorithm 2 to output the most funny response. The showcases are shown in Figure 14

Figure: Algorithm 2 Inference Step of CLoST

Figure: Figure 13: Generation Prompts of CLoST.
Refer to caption: x19.png

Figure: Figure 14: Chinese Showcases.
Refer to caption: x20.png

Figure: Figure 15: English Showcases.
Refer to caption: x21.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the inclusion of AAIE necessary? The performance improvement does not seem substantial, as shown in Table 3. It would be beneficial to highlight the best-performing models in the table and provide a detailed analysis of possible reasons.
2. The paper mentions that the dataset includes varying difficulty levels. Is this variation primarily reflected in the choice options, such as 2-out-of-1 or 4-out-of-1 selections? It would be helpful to include a more in-depth analysis of how these different difficulty levels affect model performance.
3. There are minor issues, such as the use of incorrect quotation marks in LaTeX. Additionally, providing more details on experimental parameters, like the temperature setting used, would enhance the clarity and reproducibility of the experiments.
soundness: 3
strengths: - The paper presents the CLoST framework, a novel methodology for generating humor in LLMs. This structured approach integrates knowledge graphs and causal inference, which helps reinforce logical connections between seemingly unrelated concepts, thus facilitating more coherent humor generation. Notably, the research delves deeply into data augmentation techniques, enhancing the model’s ability to generate diverse and contextually relevant humorous responses.
- The authors conduct extensive testing of CLoST across multiple humor benchmarks. These experiments demonstrate that CLoST consistently outperforms existing humor generation models in terms of accuracy and robustness. The model’s performance improvements are particularly pronounced in both English and Chinese language settings.
summary: This paper introduces the Creative Leap of Structured Thought (CLoST) framework, which enhances large language models' ability to generate and recognize humor through structured thinking and self-improvement. CLoST introduces a two-stage approach: first, using Associative Automatic Instruction Evolution (AAIE) to diversify and refine humor judgment through complex instructions, and second, employing Guided Explorative Self-Improvement Tuning (GESIT) to strengthen logical reasoning and humor understanding via reinforcement learning. CLoST improves humor judgment and creativity across multiple language benchmarks.
weaknesses: - The inherent subjectivity of humor presents a major challenge. The current approach seems to emphasize pattern recognition, where the model identifies humor based on learned patterns rather than genuinely understanding or reasoning through the humor's intricacies. The empirical results, especially on hard humor tasks, underscore this shortcoming.

- The Associative Automatic Instruction Evolution (AAIE) method employs a multi-agent system—comprising a rewriter, imaginator, and analyst—to iteratively evolve instructions. While this approach is novel and showcases creative engineering, it introduces considerable computational overhead. More critically, the ablation study results suggest that the added complexity yields only marginal improvements in performance. Specifically, CLoST shows gains in four out of seven tasks, and even these are not substantial enough to justify the extensive computational resources required.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: /
soundness: 2
strengths: This paper introduces  the Creative Leap of Structured Thought (CLoST) framework, aiming to improve the humor understanding capabilities of LLMs. It consists of two stages: Associative Automatic Instruction Evolution (AAIE) and Guided Exploratory Self-Improvement Tuning (GESIT).
summary: This paper introduces  the Creative Leap of Structured Thought (CLoST) framework, aiming to improve the humor understanding capabilities of LLMs. It consists of two stages: Associative Automatic Instruction Evolution (AAIE) and Guided Exploratory Self-Improvement Tuning (GESIT). In AAIE, human-designed instructions and automated instruction evolution help the model develop humor understanding capabilities. In GESIT, the model’s humor generation is improved using bootstrap Direct Preference Optimization (DPO) training. By learning from expert-provided joke rationales and AAIE-trained judgment skills, the model iteratively refines its humorous responses through guided reasoning.
weaknesses: 1. This paper is very difficult to read and understand. Comes with a seemingly fancy title and introductory paragraphs not closely related with the goal. 

It claims "humor generative abilities" in the intro and the teaser figure, half of the method is for improving humor judgement ability and the second half is developed for improving generation abilities. But the test tasks are multiple choice questions of SemEval (humor classification/discrimination) and has nothing to do with generation. 

The motivation and method contains lots of unclear expressions. CLoT, the method that it is based on, is not explained clearly. As a reader I am so confused with this paper that I have to rely on AI to help me understand this paper better. Even the figure is confusing. For example in Figure 1, how come that a response by CLoST "Job? No No?" is more humorous and interesting than more informative ones such as "Of course not! If you are satisfied with your job, you're already one step ahead of the person who invented the snooze button" by GPT-4o?

2. The authors seems to lack sufficient background knowledge of NLP&humor basic concepts and existing works. Multiple-choice questions is not a faithful title/motivation to "humor generation abilities".

3. Experiments: The compared baselines are zero-shot LLMs such as GPT, QWEN, etc. The only prior work being compared is CLoT which is not discussed in detail and lacks any ablation of the introduced components.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.The Associative Automatic Instruction Evolution part is a bit hard to understand and can be presented with a short and clear example to go through. It’s confused to interpret the symbols.
2.For the AAIE process, what’s the output data format used to train a lora model?
3.For the guided explorative self-improvement tuning, why do we need r+ and r-? do we use them in the fine-tuning?
4.In Table 1, it’s suprised to see gpt 4o performs worse than Llama3 on SemEval 2020 and Oogiri-GO.
5.How stable is the three-agent system? Are there cases where it fails to converge?
soundness: 4
strengths: 1.The paper proposes a novel integration of structured reasoning with creative creation.
2.The paper proposes an innovative multi-agent instruction evolving system and incorporates the causal inference to humor understanding.
3.The proposed two stage approach shows clear performance improvement through comprehensive empirical evaluation across multiple datasets and languages.
summary: This paper introduces CLoST (Creative Leap of Structured Thought), a novel framework for enhancing humor generation and understanding in large language models (LLMs). The work builds upon the Creative Leap-of-Thought paradigm with two key innovations:
1.Automatic Associative Instruction Evolution (AAIE): A multi-agent system for evolving complex instruction sets
2.Guided Exploratory Self-Improvement Tuning (GESIT): A preference optimization approach incorporating causal reasoning
The paper demonstrates improved performance on both Chinese and English humor benchmarks, including SemEval tasks and Oogiri-GO dataset. The framework shows particular strength in handling complex humor discrimination tasks and generating creative responses.
weaknesses: 1.The experimental validation lack of human evaluation for generated humor quality.
2.The three-agent system is not well motivated and can be better presented with motivation, clear example and details.
3.Better exploration and analysis of failure cases

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - Can you provide more information about the "in-house data" used for training and testing? What are its sources, and is it possible to make this data publicly available to ensure reproducibility?
- How are the multiple-choice questions in the evaluation constructed? Are they standardized across all models, and how do you control for randomness in the selection of options?
soundness: 2
strengths: The paper addresses humor generation, a task that is famously challenging, introducing a framework that uses causal relationships to model associations between concepts. Furthermore, a two-stage training process is a good approach, separating the judgement part from the generation part in LLMs.
summary: The paper introduces a framework called Creative Leap of Structured Thought (CLoST) to reinforce humor generation in LLMs. Building upon the limitations of the CLoT paradigm, the authors propose a systematic method inspired by KGs and causal relationships. The framework consists of two stages: Associative Automatic Instruction Evolution (AAIE) and Guided Explorative Self-Improvement Tuning (GESIT). In the first, the model is trained with human-designed instructions to improve its humor-judgement capabilities. In GESIT, the model refines its ability to generate humorous responses through RL, learning from both an expert model and its own judgements. Experiments conducted on English and Chinese humor datasets seemingly demonstrate that CLoST outperforms existing models in humor discrimination and enhances the model's divergent thinking abilities.
weaknesses: Firstly, the paper lacks clarity in several sections, making it difficult to fully comprehend the proposed methods. For instance, the mathematical formulation in S2.1 seems disconnected from the rest of the paper, and is not well-explained. Secondly, there is insufficient detail about the datasets used, especially the "in-house data". The lack of information about data sources and availability (and code!) raises concerns about the reproducibility of the experiments. Then, the evaluation metrics and experimental setup are not discussed enough. It is unclear in my mind how the multiple-choice questions are constructed and whether the comparisons with baseline models are fair (e.g., are other models trained on the same datasets? If not, than it's a problem).
I will state here, since it's a problem of many papers in this field: the reliance on GPT-4o as an expert model is problematic, as it is a proprietary system. This raises issues regarding the accessibility and reproducibility of your method. Proprietary system's backends change with time and without alerting the users: they are not fit for experiments that need to keep reproducibility to the forefront.
Furthermore, let humans evaluate the generated humor. A good sample of annotators evaluating whether the produced content it is "funny" or not would be a better fit and a real contribution to the field.
Finally, the paper does not address ethical considerations related to HG, such as avoiding offensive or culturally insensitive content.

Meta-review: The paper proposes a new framework called Creative Leap of Structured Thought (CLoST) to reinforce humor understanding by LLMs. The authors propose a systematic method inspired by KGs and causal relationships. The framework consists of two stages: Associative Automatic Instruction Evolution (AAIE) with human-designed instructions, and Guided Explorative Self-Improvement Tuning (GESIT) with RL, learning from both an expert model and its own judgements. Experiments conducted on English and Chinese humor datasets seemingly demonstrate that CLoST significantly outperforms existing models in humor discrimination and enhances the model's divergent thinking abilities.

Strengths:
Novel approach to humor generation in LLMs (Unwt, JsCo, SKMr)
Uses structured thinking and knowledge graphs (SKMr, Unwt, JsCo)
Shows improved performance on several humor benchmarks (Unwt, JsCo)

Weaknesses:
Reviewer (gQyc) found the method complex, with unclear explanations.
Reviewer (SKMr) said the paper lacked human evaluation of generated humor (but this was addressed)

Based on the scores (3, 5, 6, 6), the paper is just below the acceptance bar but one reviewer (gQyc) did not engage the authors' rebuttal, and another reviewer (SKMr) made unreasonable requests to release training data and code for review (rather than with the paper). I am proposing that the lower scores could be bumped by a notch, and the paper would meet the acceptance bar.

Decision: Accept (Poster)