Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps
Authors: Han Wang
Yilin Zhao
Dian Li
Xiaohan Wang
sinbadliu
Xuguang Lan
Hui Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
humor generation
reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Humor is previously regarded as a gift exclusive to humans for the following reasons. Humor is a culturally nuanced aspect of human language, presenting challenges for its understanding and generation.
Humor generation necessitates a multi-hop reasoning process, with each hop founded on proper rationales. Although many studies, such as those related to GPT-o1, focus on logical reasoning with reflection and correction, they still fall short in humor generation. Due to the sparsity of the knowledge graph in creative thinking, it is arduous to achieve multi-hop reasoning.
Consequently, in this paper, we propose a more robust framework for addressing the humor reasoning task, named LoL. LoL aims to inject external information to mitigate the sparsity of the knowledge graph, thereby enabling multi-hop reasoning. In the first stage of LoL, we put forward an automatic instruction-evolution method to incorporate the deeper and broader thinking processes underlying humor.
Judgment-oriented instructions are devised to enhance the model's judgment capability, dynamically supplementing and updating the sparse knowledge graph. Subsequently, through reinforcement learning, the reasoning logic for each online-generated response is extracted using GPT-4o. In this process, external knowledge is re-introduced to aid the model in logical reasoning and the learning of human preferences.
Finally, experimental results indicate that the combination of these two processes can enhance both the model's judgment ability and its generative capacity.
These findings deepen our comprehension of the creative capabilities of large language models (LLMs) and offer approaches to boost LLMs' creative abilities for cross-domain innovative applications.

Published as a conference paper at ICLR 2025
INNOVATIVE THINKING, INFINITE HUMOR: HUMOR RESEARCH OF LARGE LANGUAGE MODELS THROUGH STRUCTURED THOUGHT LEAPS
Han Wang" 1.2, Yilin Zhao' Dian Lit#z, Xiaohan Wang Gang Liu², Xuguang Lan" 1 Hui Wang*
Xi'an Jiaotong University'; Tencent QQ
relcad7@stu.. xxjuuxjjuu.. xjtu. edu en,
fyilinnzhao, goodli, sha wwwwshywanggggg sinbadl wtwbbedlueeeteeeenttttt com, aglanemaii.xjjae edu. en, joltwang@tencent. com
ABSTRACT
Humor previously regarded as a gift exclusive to humans for the following reasons. Humor is culturally nuanced aspect of human language, presenting challenges for its understanding and generation. Humor generation necessitates multi-hop reasoning process, with each hop founded on proper rationales. Al- though many studies, such as those related to GPT-ol1 focus on logical reasoning with reflection and correction, they still fall short in humor generation. Due to the sparsity of the knowledge graph in creative thinking, it is arduous to achieve multi- this paper, we propose a more robust frame- work for addressing the humor reasoning task, named Lol., Lol. aims to inject external information to mitigate the sparsity of the knowledge graph, thereby en- abling multi-hop reasoning. In the first stage of LoL, we put forward an automatic isstructionevoluttion method to incorporate the deeper and broader thinking pro- cesses underlying humor. andgment-oriented instructions the model's judgment capability, dynamically supplementing and updating the sparse knowledge graph. Subsequently, through reinforcement learning, the rea- soning logic for each online-generated response is extracted using GPT-4o. In this process, external knowledge is r--ntrroduced aid the model in logical reasoning and the learning of human preferences. Finally, experimental results indicate that the combination of these two processes can enhance both the model's judgment ability and its generative capacity. These findings deepen our comprehension of the creative capabilities of large language models (LLMs) and offer approaches to boost LLMs' creative abilities for cross-domain innovative applications.
hop reasoning- Consequently,
devised to enhance
INTRODUCTION
Currently, humor is attractive because, as shown in Figure I it requires a burst of inspiration, which still difficult for humans. The difficulty lies in the multi-hop reasoning process that fosters cre- ativity. Each hop in this process is based on proper rationales. Without an understanding of these rationales, is difficult for the model to grasp the internal humorous logic, making prone to relying on pattern recognition.
So far, the Creative Leap Thought (CLoT) paradigm (Xu et al. 2024) has developed two ba- sic abilities to facilitate humor generation: selection skill and ranking skill. With these two basic skills, CLoT introduces nouns as instruction tonk-translttion for self-improvement. However, only question-answe answer pairs, i.c., the beginning and ending of the multi-hop reasoning path, are utilized to train the model. As mentioned in CLo'T, this process only captures the inherent creative patterns within the data which impairs the generalization ability and fails to stimulate "thinking outside the
"Equal Contribution. Work done during internship Tencent QQ, as part of QQ MLLM project corresponding author
Projecct leader of QQ MI.LM project

---

Published as a conference paper at ICLR 2025
Case I
penson understarits you well hut doesn't love you, person loves you deeply but docan't gnderttand your
Thirk like asking you want someone who can't your
former is the reader, author.
cook but will takeout, or someone who cook but only makes dishes they like!"
Maybe better not to.
Lel.
GPT-40
CLoT
Case 2
not help the that you like what you have through!
Well, If I look like what
"Looks like you' 've been through
I've boen through, then
c help the lhat that look like what you been through!
must he mix of coffee culls. and fow too mane. midnight snocks!"
spills, awkward
LoL
GPT-4o
CL.of
Figure I: English comparison showease (more showcases are in Appendix A.5). Compared to GPT- 4o and CLoT, LoL provides shorter and more conversational answers to questions. For instance in Case 2, while LoL and CLoT may convey the same meaning, their different expressions produce different effects. Brief responses leave room for readers ponder, enhancing interest and interac- tivity.
box" for generating novel ideas. Large language models (LLMs) such as GPT-4o or ol (Lightman
et al. 2023) and QwQ (Team] 2024b), which show superior performance in almost all reasoning
tasks, do not perform exceptionally well in humor generation, as shown in Figure [ Moreover.
humor-related works (Xul 2024 Xu al. (024) focus specific aspect of humor, like puns or proverbs, while humor also encompasses elements such as irony, limiting the range in real-world applications.
For the multi-hop humor reasoning problem, understanding is fundamental for endowing LLMs with reasoning ability to avoid getting trapped in memorizing patterns. The introduction and augmenta- tion of external knowledge help LLMs understand the underlying logic and rationale. Additionally, a reward model can optimize the behavior of large language models by providing feedback, enabling them to produce outputs that better meet expectations. Due to the subjectivity of humor, unified score may contain significant noise. Thus, the judgment skill essential for providing feedback to further enhance LLMs' reasoning ability. Finally, with these two basic skills, humor understanding and judgment, the ability to generate humor can be improved.
Therefore, we propose more robust framework named LoL to address the humor reasoning task that current LLMs find challenging. LoL. consists of two-stage training: the supervised Fine-Tuning (SFT) stage and the Direct Preference Optimization (DPO) stage. In the first stage, develop human-designed judgment-related instructions and their derivatives to train the model's humor judg- ment capabilities. Additionally, we propose automatic instruction expansion method for humor- ous conversations to inject and augment knowledge into the original training data, mimicking the human thinking process step by step. This will help LLMs deepen and broaden their understanding of humor content. In the second stage, the reasoning rationale for each online-geneen seeeeeeeeerreddd response is extracted using GPT-4o. In this process, external knowledge is introduced again to assist the model in logical reasoning and learning human preferences. The judgment capability from the first stage can also be useful for expanding the preference-par dataset and further supplementing rationales.
We evaluated the humor judgment abilities of various large language models on both Chinese and English humor datasets. Experiments demonstrate that Lol. outperforms other models on almost all test sets. Additional confirmatory experiments were conducted to show that LoL enhances the model's s divergent thinking ability and effectiveness in humor generation. Our contributions are summarized as follows.
We propose an automatic instruction eevolutio system for conversation data. A three-agent system is introduced inject and augment knowledge into the original training data. This
2

---

Published as a conference paper at ICLR 2025
will facilitate LLMs in deepening and broadening their understanding of the underlying logic and rationales.
2. We propose teacher-student prompt system to enhance the judgment ability of LLMss. Through the automatic construction of conversation data between the teacher and the stu- dent, LLMs learn the teacher's judgment of the student's s thinking.
3. Experimental results demonstrate that we can enhance both the model's s judgment and gen- erative capabilities and achieve state-of-the-art performance.
2 METHOD
2.1 PROBLEM FORMULATION
The multi-hop reasoning issue can be framed as a knowledge-gras (KG) problem. In this context, nodes (or concepts) along the multi-hop path are entities within the KG, and the rationales for enabling multi-hop reasoning are relations within the KG. By enriching the entities and relations in the KG, we can more easily explore the self-evolved path multi-hop reasoning.
In general, the knowledge graph G is defined where & is the set of entities and is the set of relations. Each triple represents a relation r from the head entity the tail entity e' (Sun al. 2023a Yang al. 2023). In the specific application of humor generation, we consider knowledge graph composed of question-related entities &q and answer-related entities A The intersection Ez = Eq n & is regarded the set of correlation entities (we refer to them as correlation entities here). Given the creative and unexpected nature of humor, well the existence of causal relationships between questions and answers, &z may consist of peseddsccorreeeeee correlation entities between Eq and EA, such as those involved in puns (i.c., Eg + Eq. Ez + AA, and Eq + EA). Additionally, it can also follow the pattern &q Ez + EA, as shown in Figure 17 Clearly, it can be inferred that &g is pivotal for enabling the multi-hop in the humor reasoning.
set of triples G ((e,r,ee) e, EE
Therefore, we formulate the causal relation Re into a verbal description as shown in Figure [2] which contains the correlation entities E either explicitly or implicitly. Finally, our objective is to expand the scopes of &g and A and utilize the causal relationship structure the reasoning path. This is beneficial for mitigating the information insufficiency problem and further enabling humor reasoning.
The overall training framework is illustrated in Figure 4 In the first stage, supervised fine-tuning (SFT), we randomly initialize a LoRA model and train it using both single-turn and multi-turn question-answer format data. In the second stage, Direct Preference Learning (DPO), the model from the first stage serves reference model and is frozen to act the judgment model when self-evolving. The tunable model is trained with preference question-answa data, which helps the model improve its logical reasoning and learn human preferences.
2.2 DIVERSE INSTRUCTION EXPANSION AND TUNING
A reward model can optimize the behavior of large language models by providing feedback, enabling them to generate outputs that better meet expectations. However, due to the subjectivity inherent in humor, unified scoring system (i.e., pointwise reward model) may be plagued by significant noise, such as topics, background and so on. Additionally, human-voting (i.e., Likes) data from the community is readily available. Transforming it into pairwise data is not only easier but also more in line with human intuition of selecting the more humorous one from two responses. Consequently, we adopt a judgement model (i.c., pairwise reward model) to offer feedback, thereby further enhancing the reasoning capabilities of large language models (LLMs).
To stimulate the judgment capability of model, manually design judgment-oriented template Additionally, the understanding capability is enhanced to improve both the
as shown in Figure 2 judgment and generation capabilities through automatic instruction evolution.
Judgement Template Design. Question-ansee answer data with human-voting annotations are utilized for judgment tasks.
3

---

Published as a conference paper at ICLR 2025
a. Question: [question]
Question: [question]
C.
Question: [question]
dd Which answer/comment Is more humorous, more Drainstorming,
Answers: I. bb
Answers:
2. bb
Amswer: fanswerj
3.00 4.
Pleuse rewrite this
funay? And for the more humorous answer, hhat is the more humoroux logic?
rank answers from humorous to not humoroux?
into
interesting
humorous
-Response highest scure
Answer/Comen I. (the reason)
Answer/Comment 1.2.3.4.
d.
The student's response does not align with the ideal answer in terms of logic and humor.
Question:qqueston;
Question: (question)
The answer I give is: [answer]
The answer I give is: (answer]
(Replaced by the ground truth unswer which the most humorous)
negative Teedback (It interesting,) And the hint [hint)
not
Criteria
steps step limitation)
positive feedback (Congratulations))
Figure 2: The details of judgement-orientee instructions templates.
(1) Selection and Ranking Skill. Two basic templates are designed, shown in Figure 2 (a) and (b). Single-choice and ranking questions are basic tasks for developing judgment capabilities. Furthermore, deepen the model's understanding of the contrast between "humorous" and "non- humorous", two additional tasks are proposed as shown in Figure 2 (c) and (d).
(2) Answer Rewriting (template in Figure (c) We identify the answer receiving the most human votes as the most humorous one. Subsequently, GPT-4o is employed to rewrite this most humorous answer into non-humorous version. Finally, the model trained to transform a non-humorous statement into a humorous expression using single-round session data format.
(3) Teacher-Student Prompt Loop (template in Figure ())). Similar the human thought process, the complete exploration process encompasses trial-and-ere reflection, and backtracking. Within this process, the ability to make sound judgments in a long chain of thought is crucial. Therefore, guided conversation process between two agents is proposed to enhance the reflection, and back- tracking. We utilize two GPT-4o models and designate one as the teacher and the other as the stu- dent. The teacher provides a judgment conclusion and provides prompts based on the given question and the ideal answer. The student generates an answer using the teacher's prompts and the original question. Two criteria as shown in Figure d) are applied to stop this multi- conversation.
Automatic Instruction Expansion. In the human cognitive process, mechanisms such as trial and error, reflection, and backtracking inherently supported by a robust understanding of concepts. To emulate the deep and extensive understanding characteristic of human cognition, we propose a three-agent system inspired by automatic instruction evolution, which leverages the extensive world knowledge encapsulated in large language models (LLMs) to autonomously enrich prompts within the realm of humor comprehension. This methodology not only facilitates the exploration of uncon- ventional associations among disparate concepts but also enhances the model's ability to establish and strengthen interconnections between diverse ideas. The process shown in the Figure 3
Given a seed conversation dataset D = {lgh a N k= where of denotes questions and aB repre- sents humorous responses. Let 1o {Lx (oh ag (. 1 be the initial instruction set, where each instruction o contextualizes the dialogue pair (4) ag). The framework implements 3-round evolutionary process through three cooperative agents. The iterative process terminates when either Criterion-2. satisfied maximum evolution rounds 2 3 in our setting) are reached. Each iter- ation produces enhanced instruction-outpl pairs (Jncw: Vnew) where Dnew E Y denotes optimized responses under the augmented instruction space.
The Generators module streamlines the process of instruction generation, enabling the automated generation of instruction data based on a seed conversation. Unlike evol-instruct methods in math and other fields, we design an "imaginator" in the generation process for humor reasoning. tries to guess the seed conversation based on the answer from evolutionary instruction. This can cause shift in the storyline within the conversation topic. In other words, it tries to explore the boundaries of the knowledge graph under the context of the seed conversation and enhance creativity.
Generation:
4

---

Published as a conference paper at ICLR 2025
8.
Seed Instruction and Chat
Evolved Instructions
Imagined chats
b. Evolved Instructions
Selected Instructions
Cimnernerr
Imaginstur
Instruction:
Criterio:
punchiine
F Seed
Please generate a new dialogue based the above answer. This dialogue should only
instructions
Evolved
informative
consist round of question and answer.
Please explain why "joke this answer from the Bollowing rspects:
His
Imagined Chate
core perspective
analyse
Imagined Chat
changed compured
Seed Chats
Figure The detail of AIE. And the detailed process shown in Algorithm T
Selection: The Selectors module is crafted to streamline the settuctiioonilterinn process, enabling the curation of instruction datasets from evolved instruction data. In contrast to evol-instruct tech- niques in mathematics and other domains, this approach serves merely a means unleash the capabilities of L.LMs, but also to boost the capacity for thinking outside the box, thereby further actualizing creativity. Therefore, we design critic related the conversation topic, which limits the model's thinking from being overly divergent.
Finally, the system outputs multi-turn question-answer data which format is [igh a²), (Tg. vl)..... (9x mh Og ax*), ), (T .x ) where my is the maximum number of communication rounds between the three agents. Finally, all above human-designed and automatic expanding data are involved in training.
2.3
GUIDE EXPLORATION AND SELF-IMPROVEMENT TUNING
Stage 8:
Guided Explorative Self- Imprroveet
Instruction Tuning
Training
Training
DPO
Model
Model
Data Angmmnnt
soops
N steps
Preference dara
Preference data with rationales
Sample
rationalize
Model
Model
For and Rg* and R
Judge
Preference pains
Generation
Figure Pipeline of training process.
In the previous section, we developed model with judgment and understanding capabilities. Through multiple rounds of dialogue, strengthened the relationships among diversified entities Eq and ΣĄ- At this stage, the policy model enhances its performance through online self-learning, using a dataset of (question, humorous answer, onn-humoruu answer) tuples: (q.a*, a+ ). Specif- ically, shown in Figure 4 it leams by repeatedly sampling responses, evaluating answer correct- ness with the capability from Section 2.2 and updating its parameters online using Direct Preference
Optimization (Rafailov et al. 2024
detailed in Algorithm we start with an initial offline preference dataset without rationales, Do (99,, a,t a, )) where M is the number of training data. The model trained in Section 2.2 copied into two. One, denoted as πT, is frozen to judge whether sampled answers are humorous for online data augmentation. The other, is trained to have more robust generative ability.
During training. the policy model æ trains on Do for few steps. Every T steps, π selects questions from D, to sample two answers or each question. The judge model x² then determines the more humorous one, constructing new pairwise data incorporated into the training dataset. Additionally, causal inference expert (GPT-40) provides rationale for each answer relative the question (see
5

---

Published as a conference paper at ICLR 2025
Appendix A.4 for details). This process expands the preference dataset to D Do U D, where D {(g. a ,
EXPERIMENTS
We collect humor-related data with human-voting from humor games and communities, and then organize it into the format mentioned in Section 2.2] Since we enhance the generation ability based on the judgment capability, it is crucial to verify the performance in humor-judgment tasks. Thus, we construct single-choice questions at different difficulty levels and carry out the consensus generation task validation.
3.1 DATASETS SOURCE
(1) Oogiri-GO (hong et al. 2024 In the "Oogiri game", participants required give unex- pected and humorous responses to given images, text, or both. This game demands sudden burst of insight and strong associative thinking within the given context. Similar to the processing method of CLoT (Zhong et al. 2024), we randomly select 95% the samples to construct the training dataset, and the remaining % is used form the test dataset for validation and analysis.
(2) SemEval 2021 Task Meancy et al. (202T) contains binary labels and ratings collected from a balanced age group ranging from 18 to 70 years old. Data with binary labels and ratings are utilized to construct the easy-case task and the hard-case task mentioned in Section 3.2 respectively.
(3) SemEval 2020 Task 7 (Hossain et al. 2020a) is game change word in headline to make it funnier. It also contains binary labels and ratings which are utilized to construct the easy-case task and the hard-case task.
(4) Chinese Community Data. We collect data from various Chinese communities, including Ruozhiba and others. Human-voting (i.e.. Likes) labels are readily available, and we use them to construct both easy-case and hard-case tasks.
3.2 TASKS CONSTRUCTION
(1) Inspired by the task design in CZon et al. 2024), we develop judgement-related tasks as follows.
Easy-case Task (i.e. 271), Lingle-choie-ffom-ttoooo binary-label data mentioned above and the data with the largest gap human-voting counts. The construction adheres to the template depicted in Figure
questions are constructed from the
Hard-case Task (i.e. 2T'(Hard), 37',, 47')). Single-choice- fromet three or four)-options questions are constructed from the same source with closer gap in human-voting counts. The construction adheres to the template depicted in Figure
(2) Inspired by the generation tasks designed in CLoT<Zhong et al. 2024). the Divergent Associate
Thinking (DAT) task (Olson al. (202) validate the capability of creativity is carried out. Additionally, human evaluation will also be validated.
Table 1: Humor judgement validation on ruozhiba dataset
Model
2T1
GPT-4o
76.40
3.3 RESULTS ANALYSIS
QWENI.5-32B
68.85
Evaluation on Judgement tasks in English. We vali- date the 1 accuracy of completing each judgement task and show the performance several models in 3 Overall, compared with open-source language models such as LLaMA3, QWEN and so on, state-of- the-art closed-source LLMs exhibit impressive zero- shot performance humor judgement tasks. Model
CLoT
50.20
QwQ
90.80
LoL
95.35
trained by LoI. has significantly improved compared other models (such LLAMA3-70B and

---

Published as a conference paper at ICLR 2025
GPT-4o). Specifically, the average accuracy in diverse English benchmarks has increased by 4.55%
and 5.91% respectively.
Evaluation on Judgement tasks in Chinese. We also evaluate the accuracy rate (acc%) of complet- ing each selection task in Chinese and show the performance of several models Table 2 Overall, compared with open-source language models including LLaMA3 and QWEN, the state-of-the-an closed-source large language models show impressive zero-shot performance on humor judgement tasks in Chinese. The model trained by LoL also shows significant improvement compared to other models (such as GPT-4o) (with an average accuracy in diverse Chinese benchmark increase of 16.22%).
We also conduct an additional experiment on the Ruozhiba datase which most well-known large language models (LLMs) have been trained on. We asked GPT-4o to rewrite the Ruozhiba queries into question-answer pairs, placing the punchline in the answer section. Then, we asked GPT-4o again to rewrite the ground-truth answers into non humorous versions. Based on the positive- negative pair data, the LLMs were tested, and the results are shown in Table 9, The results indicate that CLoST also achieves state-of-the-ar performance on the Ruozhiba dataset.
Evaluation on Creative-thinking validation in Generation Tasks. To evaluate the associa- tive generalization capability of LoL, test on a creative task known as the Divergent As- sociation Task (DAT). The DAT is classic cre- ativity test in which testee write 10 unrelated words, and words with greater "distances" in their context embeddings indicate more diver- gent thinking. In the Chinese creativity test, we utilize Chinese Word Vectors (Li et al. 2018) to calculate the DAT score. First, we provide specific words and ask the model to generate associations and imaginations, obtaining 10 as- sociated words. Then, we use these 11 words to calculate the DAT score. Additionally, we test how the model's DAT score varies the num- ber of test words increases. can be observed that the number of words increases, the DAT score also tends to stabilize. From the average of each domain, it can be seen that LoL. has the highest score after stabilization. As shown in Figure 5 LoL has slight performance im- provement in the mean value of DAT compared to Qwen1.5-32B, GPT-4o and CLoT.
Table 2: The accuracy (%) of choice questions on various Algorithms in Chinese benchmarks.
Chinese Benchmark
Model
2T1
2TT(d)
GPT
40
64.98
63.49
8B
50.72
57.44
LLAMA3
70B
59.48
61.22
7B
54.82
51.71
QWENI.5
14B
53.45
57.41
32B
52.71
56.27
7B
51.99
58.17
QWEN2
57B
65.91
57.03
QWEN2.5
32B
61.53
60.46
Baichuan2
13B
50.56
53.61
CLoT
7B
52.12
34.46
QwQ
32B
59.75
57.04
QwQ+Lol
32B
88.91
63.12
OURS
32B
90.95
69.97
In addition, we employ T-SNE to project the embedding vectors of these words into a two-dimensional space. specifically, the target word positioned at the center, and a circle is drawn with radius equal to the Fuclidean distance between the target word's embedding and that of the farthest associated word. In Figure ((c) and (d), the embedding vectors of five target words and their associated words are illustrated, with different colors representing different target vectors. larger circle indicates broader semantic range for the target word, implying a greater number of associated words. In both tests, LoL outperforms previous works including SOTA models like GPT-4o.
Human Evaluation in Generation Tasks. We conduct user-preference study to test the creativity and humor of L.MMs. Here, we select four LLMS (Lol., GPT-40, QWEN1 .5-32B, CLoTT) to generate responses for total of 200 text-based questions. We present question and several corresponding replies from the four LLMs, and ask users to choose the most creative and humorous response. Figure summarizes the statistical analysis of 3000 valid surveys. Figure shows the win-rate calculated based on the problem dimension. The results indicate that users have strong inclination

---

Published as a conference paper at ICLR 2025
TABLE 3: THE ACCURACY (%) OF CHOICE QUESTIONS ON VARIOUS ALGORITHMS ENGLISH BENCHMARKS.
SEMEVAL 2021
SEMEVAL 2020
OOGIRI-GO
MODEL
2TI
2TI(HARD)
3T1
4T1
2TI
2TI
GPT
40
85.09
60.77
43.71
34.63
55.08
85.09
8B
43.85
54.23
39.81
29.57
59.93
72.05
LLAMA3
70B
93.60
58.08
39.81
31.82
60.73
88.51
7B
62.04
52.02
31.54
24.89
50.46
36.65
QWENI.5
14B
82.05
51.04
30.92
24.24
50.38
53.73
32B
68.01
52.57
35.38
28.79
56.39
68.01
7B
56.55
50.63
32.31
23.38
50.08
62.
QWEN2
57B
83.30
52.02
37.08
28.79
48.29
48. 14
QWEN2.5
32B
94.00
55.22
34.77
27.92
58.71
81.68
BAICHUAN2
13B
51.70
52.71
35.69
24.24
51.45
50.00
CLOT
7B
52.49
51.74
34.46
23.59
53.50
52.49
QWQ
32B
80.05
53.06
33.49
24.66
56.58
59.63
OURS
32B
96.58
57.45
48.06
35.90
64.57
97.20
TABLE 4: ABLATION ON ENGLISH BENCHMARKS.
SEMEVAL 2021
DOGIRI-GO-EN
MODEL
2TI
2TI(HARD)
3T1
4T1
2TI
QWENI.5-322
68.01
52.57
35.38
28.79
68.01
FOGIRI-GO+FICURE
55.12
53.27
33.54
27.49
81.68
(
89.70
50.28
30.46
17.32
95.96
(CCD)
NOGIRI-GO+ALE
88.40
50.07
30.92
19.48
96.58
ALL (C.D)
92.25
52.81
37.46
30.26
96.27
ALL+AIE(DIET)
94.25
53.10
45.46
32.26
97.20
to select the results of LoL. highlighting the high quality creative content generated by CLoT. For more details the user study, refer to the Appendix
Ablation Study
We examine the ablation effects of different com- ponents in Lol. and report the performance results in Table 4 and Table 5 In Table 4 line presents the performance of QWEN-1.5-322 1-5-2B. Lines 2-4 show the performance of gradually adding meth- ods on the Oogiri-GO-en dataset. The results indi- cate that with the increase in the number of tasks, especially in the teacher-student system, the judg- ment performance improves a lot. Employing AIE on the Oogiri-GO-en dataset only causes a slight decrease in performance. This might be caused by overfitting to the divergence of thought nn-iri-GO-en dataset. Lines 5-6 show that when all datasets with the introductory -part method used to train the model, a further increase in performance is realized.
Table 5: Ablation on Chinese benchmarks.
CHINESE BENCHMARK
MODEL
2T1
2TI(HARD)
QWENI.5 5--BBB
52.71
$6.27
+FIGURE (
63.56
63.88
+FIGURE C)(D)
83.62
64.64
+AIE (DIET)
90.34
64.64
the
To evaluate the performance of GESIT, which primarily enhances the model's causal and logical reasoning capabilities (i.c., the ability to relate to internal contexts), we enlisted three experts to assess the logical coherence of responses 200 examples from GPT-4o, GPT-4 and QWEN2-57B,
8

---

Published as a conference paper at ICLR 2025
DAT Chitena Werth
DAT Ciirase Mords
041 200
Number wondd
Numttr words
(a)
(b)
(c)
(d)
Figure 5: Divergent associate thinking (DAT) validate. (a). DAT compared among LoL and
three baselines (b). DAT score compared among different component of LoL (c). TSNE Results of
Word Vectors Obtained by QWENI.5-328 Associating Five Target Words. (d). TSNE Results of Word Vectors Obtained by LoL Associating Five Target Words.
Togle compuricon Fannewors
Humer Poollation
Evolestion
OWENZ -57B
141
110)
GPT-4
K01
60
600
450
4o
0
100
200
300
190
#Lol
OWINISS30
OWIN
(b)
(a)
Figure 6: (a) GPT-4 and GPT-4o logically evaluates the output of the model after DIET and LoL respectively. (b). Human evaluation about the win rate statistics based on the problem dimension. (c). Human evaluation about the win rate based on the total number of votes received by the four LLMs.
respectively. The experimental results shown in Figure 6 a) demonstrate that incorporating GESIT significantly strengthens the logical reasoning in the model's answers. In addition, the DAT test is conducted on ablation study in Figure 5(b), which shows that AIE enhance the divergent associate thinking ability.
RELATED WORKS
Large Language Models (LLMs) and Their Creativity. Recently, language models (Bai al, Chen et al.] 2023) have garnered widespread attention
2023
Wang et al. 2023 Liu et al. 2024
due to their impressive reasoning capabilities (Wang et al. [023 Saparov & He. 2022] Zeng et al.
2022 Driess et al.) 2023 Dong et al. 2023
Ye et al. [023 Liang al.) 2024). Additionally,
increasing number of studies are focusing on exploring the creativity of LLMs (Ling et al. 2023
Summeer--Stay al. 2023 Sun al. 2023h Bhavya et al.
2023 with applications in fields such
as scientific discovery (Park et al. 2023] Kang et al. 2022
Hope et al. [202)] Liang et al. 2022
Huang et al. (203) and creative writing Swanson
2021 Chakrabarty
2022 Wu al,
2022 Mirowski et al. 2023 Dang et al. 2023).
Computational humor branch of computational linguistics and artificial intelligence that uti- lizes computers to study humor (Binsted et al. 2006). It eccompasses various tasks, including humor discrimination (Shahaf et al., 2015) Tanaka al. 2022 Xu al. 2022 Chen & Zhang 2022 Ku- mar et al. 2022: Wu et all][[2.22 [0021 & Shahat] X2e22]Xie et al. 2023; et | Hossain al. (0200)), humor interpretation (Hwang Shwartz [2023] Evans et al. [019]] Vásquez & Aslan
2021 and humor generation (Amin & Burghardi [2020] Zhang et al. [0200] Hossain et 2020b
Valitutti et al. 2013 Chaudhary et al. 2021). With the advancements in generative LLMs, humor
generation has become a focal point due to its demand for creative thinking. (Zhong et al.) 2024) extends the chain-of of- paradigm into humor generation. However, only question- pairs and nouns bask- translation, i..., the beginning and ending of the multi-hop reasoning path, utilized to train the model, and this process only captures the inherent creative patterns within the data which impairs the generalization ability and fails inspire "thinking outside the box" for gen-

---

Published as a conference paper at ICLR 2025
erating novel ideas. Therefore, we develop reasoning process featuring the processes of thinking divergence and reflection for humor generation.
Instruction evolutionary. Recent attention has focused the complex instlwition-folloiing pabilities of Large Language Models (LLMs), leading to the development of new evaluation bench- marks (Zhou et al.) 2023 Jiang et al.
2023 Qin al. 2024b). These studies consistently reveal that open-source LLMs lag behind proprietary models in their ability to follow complex instructions. However, there has been limited research on techniques to enhance this capability. Notable excep- tions include Evol-Instruct (Xu et al. 2023) Zeng et al. 2024) and Conifer (Sun et al. 2024). which encourage LLMs to evolve instruction complexity and apply supervised fine-tuning (SFT) on the data generated from this process. Automated generation is widely-used method for Evol-Instruct, minimizing the reliance on extensive human annotation or manual data collection. This approach utilizes LLMs to create large volumes of instructional data sourced from chat data (Chiang et al.,
2023) or by expanding small set of seed instructions (Wang et al. 2022] Xu et al.) [203]]
Li et al,
2023). The generated instructions are then used to derive corresponding inputs and outputs. In this
paper, LoL develops an automatic instruction evolution method with three agents to strengthen the ability of thinking divergence and humor understanding.
Large language models reasoning, Human cognition involves two distinct modes of processing:
one that is fast and intuitive, and the other that is deliberate and analytical. Currently, LLMs can not only generate rapid responses using learned patterns, but more significantly, simulate complex
reasoning processes through mechanisms like chain-of-thouge (Wu et al., [022] [Wei et al. 2022
Zhang et al. 2022. Yao et al., 2024 Long (023) or other forms of search, similar to how humans
engage in deeper, step-by-step thinking (Fu et al. 2022, Lightman al. 2023 Lai et al. 2024).
However, these approaches build on existing LLMs without truly embedding the chain-of-thought
ability within the models themselves. As a result, LLMs cannot inherently learn this reasoning capability, leading to active research on how to integrate it directly into model training. Most of these
efforts(Qin QQin et al.) 2024a Luong et al. 2024 Team 2024b Yang et al., 2024| Team 2024a Wang et al. 2024 ol Team 2024) focus improving L.M.M reasoning by integrating process supervision, reinforcement learning (RL), and inference- time computation strategies such as guided search. By doing so, it shifts the focus from merely scaling model parameters during pre-training to leveraging smarter inference strategies at test time. These techniques help the model refine its reasoning step by step, allowing it pause, evaluate intermediate reasoning, and select better solution pathways during test-time computation. All of these are adept at factual reasoning like math or coding reasoning, while humor reasoning is mostly non-factual reasoning requiring creative thinking.
5 CONCLUSION
In this paper, we introduce the LoL. method aimed at enhancing the generation capabilities of large language models (LLMs). LoL commences by transforming humor datasets into instruction-tuning data to train LLMs, thus improving their logical thinking (LoT) and judgment abilities. Subse- quently, LoL utilizes Guided Explorative Delf-Improvement Tuning, enabling LLMs to generate more creative structured thought data by understanding rationales and to select high-quality data for self-refinement training. Experimental results illustrate the effectiveness and generalization ability of LoL. across diverse tasks, such as witty response generation and humor discrimination.
ACKNOWLEDGMENTS
This work was done during internship at Tencent QQ, as a part of QQ MLLM project and supported in part by NSFC under grant No.62125305, No.62088 102, No. U23A20339, No. 62203348.
REFERENCES
Miriam Amin and Manuel Burghardt. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pp. 29 41, 2020.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision- laioun-aanguag model for understanding, local- ization, text reading, and beyond. 2023.
10

---

Published as a conference paper at ICLR 2025
Bhavya Bhavya, Jinjun Xiong, and Chengxiang Zhai. Cam: large language model-based creative analogy mining framework. In Proceedings of the ACM Web Conference 2023, pp. 3903-3914, 2023.
Kim Binsted, Anton Nijholt, Oliviero Stock, Carlo Strapparava, G Ritchie, R Manurung, H Pain, Annalu Waller, and O MaMaa. Computational humor. IEEE intelligent systems, 21(2):59-69, 2006.
Tuhin Chakrabarty, Vishakh Padmakumar, and He He. Help write poem: Instruction tuning as a vehicle for collaborative poetry writing. arXiv preprint arXiv:22 13669, 2022.
Tanishq Chaudhary, Mayank Goel, and Radhika Mamidi. Towards conversational humor analysis and design. arXiv preprint arXiv: 00022/03.00055 2021.
Chengxin Chen and Pengyuan Zhang. Integrating cross cross-modal interactions via latent representation shift for multi model humor detection. In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pp. 23-28, 2022.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478. 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna, Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Hai Dang, Sven Goller, Florian Lehmann, and Daniel Buschek. Choice over control: How users write with large language models using diegetic and non-diegetic prompting. In Proceedings of the 2023 CHI Conference on Human Factors Computing Systems, pp. 1-17, 2023.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309. /499, 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch. Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03377 2023.
Jonathan B Evans, Jerel E Slaughter, Aleksander PJ Ellis, and Jessi M Rivin. Gender and the evaluation of humor at work. Journal of Applied Psychology, 104(8):1077, 2019.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- prompting for multi-step reasoning. The Eleventh International Conference on Learning Representationss 2022.
Tom Hope, Ronen Tamari, Daniel Hershcovich, Hyeonsu B Kang, Joel Chan, Aniket Kittur, and Dafna Shahaf. Scaling creative inspiration with fine-grained functional aspects of ideas. In Pro- ceedings of the 2022 CHIl Conference Human Factors in Computing Systems, pp. 1-15, 2022.
Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. SemEval-2020 task 7: As- sessing humor in edited news headlines. In Aurelie Herbelot, Xiaodan Zhu, Alexis Palmer, Nathan Schneider. Jonathan May, and Ekaterina Shutova (eds.), Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 746-758, Barcelona (online), December 2020a. Interna- tional Committee for Computational Linguistics. doi: 10.18653/110000.eeeevlllll 1.98. URL www.sal-...a...... http:::/cclanttooooyy..ooo http://ogg.22....eaal...........
Nabil Hossain, John Krumm, Tanvir Sajed, and Henry Kautz. Stimulating creativity with funlines: A study of humor generation in headlines. arXiv preprint arXi::2002.02031) 2020b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 1061:2/06600688. 2021.
11

---

Published as a conference paper at ICLR 2025
Zhongzhan Huang, Senwei Liang, Hong Zhang, Haizhao Yang, and Liang Lin. On fast simulation of dynamical system with neural vector enhanced numerical solver. Scientific reports, 13(1):15254, 2023.
EunJeong Hwang and Vered Shwartz. Memecap: A dataset for captioning and interpreting memes. arXiv preprint arXiv:2305. 01.223055.3770. 13703, 2023.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. arXiv preprint arXiv:23 10.204100 2023.
Hyconsu B Kang, Xin Qian, Tom Hope, Dafna Shahaf, Joel Chan, and Aniket Kittur. Augmenting scientific creativity with analogical search engine. ACM Transactions Computer-Human Interaction, 29(6): 1-3,, 2022.
Vijay Kumar, Ranjeet Walia, and Shivam Sharma. Deephumor: a novel deep learning framework for humor detection. Multimedia Tools and Applications, (1aaaaiioos,,,11111 (12)1116977-1681 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng. and Jiaya Jia. Step-dpo: Step-
wise preference optimization for long-chain reasoning of llms. arXiv preprint arXiv:2406.18629 2024.
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. Analogical reasoning on 2018.
chinese morphological and semantic relations. arXiv preprint
Xian Li, Ping Yu. Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. arXiv preprint arX1v:2308.062599, 2023.
Mingfu Liang Ying Wu, et al. Toa: task-oriented active vqa. Advances in Neural Information
Processing Systems, 36, 2024.
Senwei Liang, Zhongzhan Huang, and Hong Zhang. Stiffness-aware neural network for learning hamiltonian systems. In International Conference on Learning Representations, 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Zhan Ling, Yunhao Fang. Xuanlin Li, Tongzhou Mu, Mingu Lee, Reza Pourreza, Roland Memi- sevic, and Hao Su. Unleashing the creative mind: Language model hierarchical policy for improved exploration on challenging problem solving. 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 26296-26306. 2024.
Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291 2023.
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning. arXiv preprint arXiv:2401.n0 .08667, 2024.
JA Meaney, Steven Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. Semeval 2021 task 7: Hahackathon, detecting and rating humor and offense. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-202/1) pp. 105-119, 2021.
Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screenplays and theatre scripts with language models: Evaluation by industry professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-34, 2023.
Skywork ol Team. Skywork-ol open series. https: hetttpss//huggingface... Skywork Novem- ber 2024. https::: hwtp:://awgiig.a...eeecccme wttph/fw..........ooooeeeeoo co/Skywork
Dan Ofer and Dafna Shahaf. Cards against ai: Predicting humor in a fill-in-the blank party game. arXiv preprint arXiv:2210.130166 2022.
12

---

Published as a conference paper at ICLR 2025
Jay A Olson, Johnny Nahas, Denis Chmoulevitch, Simon J Cropper, and Margaret E Webb. Naming unrelated words predicts creativity. Proceedings of the National Academy of Sciences, 118(25): c2022340118, 2021.
Michael Park, Erin Leahey, and Russell Funk. Papers and patents are becoming less disruptive over time. Nature, 61 613(7742):138-- 144, 2023.
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li. et al. 01 replication journe:: A strategic progress report-part. 1. arXiv preprint arXiv:2410. 18982, 2024a.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang. Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601 2024b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of th-thhtgh arXiv preprint arXiv:2210001240, 2022.
Dafna Shahaf, Eric Horvitz, and Robert Mankoff. Inside jokes: Identifying humorous cartoon cap- tions. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1065-1074, 2015.
Douglas Summers- Stay, Clare R Voss, and Stephanic M Lukin. Brainstorm, then select: a generative language model improves its creativity score. In The AAAI-23 Workshop on Creative Al Across Modalities, 2023.
Haoran Sun, Lixin Liu, Junjie Li. Fengyu Wang. Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained catructtoo-followinn ability of large language models. arXiv preprint arXiv:2404.02823, 2024.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung- Yeung Shum, and Jian Guo. Think-on-graphe Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.00669) 2023a.
Yuqian Sun, Xingyu Li, Jun Peng, and Ze Gao. Inspire creativity with oriba: Transform artists' original characters into chatbots through large language model. In Adjunct Proceedings of the 2023 ACM International Joint Conference Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing, pp. 78 82, 2023b
Ben Swanson, Kory Mathewson, Ben Pietrzak, Sherol Chen, and Monica Dinalescu. Story centaur: Large language model few shot learning as a creative writing tool. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp. 244-256, 2021.
Kohtaro Tanaka, Hiroaki Yamane, Yusuke Mori, Yusuke Mukuta, and Tatsuya Harada. Learning to evaluate humor in memes based on the incongruity theory. In Proceedings of the Second Workshop on When Creative Al Meets Conversational AI, pp. 81-93, 2022.
Qwen Team. Qvq: To see the world with wisdom, December 2024a. URL https: http///.wemmmmmmmmmm.m github. httpbb/ooooooooog////bbbb 72b-pr/-2bbbpeeeee
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024b. URL
https: /ttp///wwwmmmmmmm/////...//..... www.wwmm.iihuu.....m. wtt.://..oooo/////.o... wwt.:/wwwwwwwweiee////
Alessandro Valitutti, Hannu Toivonen, Antoine Doucet, and Jukka M Toivanen. "let everything turn well in your wife": generation of adult humor using lexical constraints. In Proceedings of the 51 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 243-248, 2013.
13

---

Published as a conference paper at ICLR 2025
Camilla Vásquez and Erhan Aslan. "cats be outside, how about meow": multimodal humor and creativity in an internet meme. Journal of Pragmatics, 171:101-117, 2021.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 2024.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03077 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint 0rXiv2222..0056/ 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837,,447 2022.
Jiaming Wu, Hongfei Lin, Liang Yang, and Bo Xu. Mumor: A multimodal dataset for humor detection in conversations. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13-17, 2021, Proceedings, Part I 10, pp. 319-627. Springer, 2021.
Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. Promptchainer: Chaining large language model prompts through visual program- ming. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10, 2022.
Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv preprint arXiv:2306. 14899, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 2023.
Haojie Xu, Weifeng Liu, Jiangwei Liu, Mingzheng Li, Yu Feng, Yasi Peng, Yunwei Shi, Xiao Sun, and Meng Wang. Hybrid multimodal fusion for humor detection. In Proceedings of the 3rd International on Multimodal Sentiment Analysis Workshop and Challenge, pp. 15-21, 2022.
Rongwu Xu. Exploring chinese humor generation: A study on two-part allegorical sayings. arXiv
preprint arXiv:2403. 1000 2024.
Zhijun Xu, Siyu Yuan, Lingjie Chen, and Deqing Yang. "a good pun is its own reword": Can large language models understand puns? arXiv preprins M.XXi:2404.135000 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Ii, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Jumyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint 4407 10671, 2024.
Yuhao Yang, Chao Huang, Lianghao Xia, and Chunzhen Huang. Knowledge graph self sulf-supervised rationalization for recommendation. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pp. 3046-30566 2023.
14

---

Published as a conference paper at ICLR 2025
Shunyu Yao. Dian Yu, Jeftrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2tti::2304.14178, 2023.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00550 2022.
Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. Automatic instruction evolving for large language models. arXiv preprint arXiv:2406.00770 2024.
Hang Zhang, Dayiheng Liu, Jiancheng Lv, and Cheng Luo. Let's be humorous: Knowledge en- hanced humor generation. arXiv preprint arXiv:2004. 13317, 2020.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.0344 2022.
Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, and Pan Zhou. Let's think outside the box: Exploring leap-of-ther of-thught in large language models with creative humor generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 13246-13257, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followiing evaluation for large language models. arXiv preprint arXiv:231..07911, 2023.
A APPENDIX
A.I TRAINING AND EXPERIMENT DETAILS
Training pipeline:
LoL takes a two-stage training strategy. In the first process (supervised fine-tuning (SFT)), we randomly initialize a LoRA model. And we train the model with single-turn question-answer format data (data from Figure 2(a)(b)cc) and muti-turn question-anss answerr format data (data from Figure 2(d) and AIE). In the second process (Direct Preference Learning (DPO)), the first stage model serves as ref model, and it is frozen as judgement model. The tunable model is trained to improve the reasoning generation capability. At the beginning of stage 2, only preference question-answar annwer data without rationale is fed into the tunable model for training. After several steps, the rationale for each online generated response is extracted using GPT-4o and the preference question-answer data with rationles are mixed into the original dataset. And in each batch, the ratio of 'w' and 'w/o" rationale
Implementation Details We validate the validity of LoL mainly based on QWENL.5-32-OO8 --3-B-hhhhh Bai al. (2023) using the LoRA(Hu et al.) (202T)
A100 GPUs. For the first stage, we train the model on 95% of the dataset mentioned above for 6 epochs using the Adam optimizer with a learning rate of 3e 4. the second stage, 5% of the dataset is used to train GESIT for 3 epochs using the AdamW W optimizer with learning rate of 2e 4. The models are tested on the tasks introduced in the previous part. And more parameters used generation are listed Table
Experiments parameters:
Table 6: Parameters in Validate
temperature
top k
top P
length penalty
50
1.0
1.0
15

---

Published as a conference paper at ICLR 2025
A.2 DETAILS IN TEACHER-STUDENT LOOP IN FIGURE (D)
Through the "teacher-student" interaction mechanism between two agents, the key characteristics of human thinking have been successfully simulated. The trial-and-error mechanism allows for the exploration of different solution paths; the reflection phase promotes an in-depth analysis of the root causes of errors; and the backtracking function enables the dynamic correction of the thinking path. This cognitive architecture makes the reasoning process of AI closer to the real- real-world problem- solving patterns of human experts. We utilize the data from the teacher agent to train the model with judgment ability. And an example of this loop is shown in Figure 7
Q: Why can't the emperor abundon the hurem for someone loves?
A: If you have mised more than 5 cats, you will understand that love and heart labor, harem has no renson speak, and mud is the best choice...
Question: Why can't the emperor abandon harem for beloved? My answer that the emperor cunnot abandon his harem for beloved, any more than ignore your other dogs, rabbits and goldfish for your favorite cat. After all, the emperor must not only balance lave, but also balance the harmony of the whole "pot park" uh! Do you find my unswer interesting?
This response not impressive enough, can you try again? Please the following tips: Try use a concrete example compure the emperor's situation, such having multiple pets. Emphasis the concept of "love and labor", explaining why you can't focus on just one. 3. Mention of "muddling" as solution. Please follow these tips and try again.
Why can't emperor abandon harem for beloved? My answer that the emperor cannot abandon bis harem for beloved, any than you neglect the rest of your family for your favorite After all, cats have different temperaments and needs, and you need to make them all happy, otherwise it's going to be cal fight in the house! Do you find this answer surprising?
Question:
Your comment wasn't fascinating enough. Can you think a more interesting one? Please note the following tips: You need emphasize the concept of "love and heart", indicating that the emperor needs balance the emotions and needs of multiple parties. You emperor's situation in the harem. 3. Try to be concise and avoid too much detail, Please try again and adjust your answer according to the above tips.
the expression "smooth mud" to describe the
Question: Why can't the emperor abandon the harem for a beloved? Answer: If you have raised more than 5 cats you will understand, love bo and heart labor, harem has no reason to spcak, and mud is the best choice... (Facepalm) Do you find this answer surprising?
Your response are very attractive, very good!
Figure 7: An example of The teacher-student prompt loop in Section 2.2
A.3 DETAILS OF AIE
The detailed process of AIE shown in Algorithm 1 The generator will inject more information into the conversation, and the imaginator will explore the boundaries of the knowledge graph be- hind the conversation. Finally, the criterion serves as a form of reject sampling for the augmented instructions.
An example of the generation process shown in Figure instruction will lead to very simple answers and understanding from the model. While after the evolution, the instruction can become more concrete and the answer will be more informative. And an example of the rule evolution is shown in Figure I8. Additionally, example of rule evolution shown in Figure 18] In addition, examples of the imagined conversations are shown in Figure
Before the evolution, the simple
and Figure g
A.4 DETAILS IN GESIT
GESIT develops an online DPO training frame with rationales to reinforce the reasoning ability. The example of the template is shown in Figure 18 And the training frame is shown in Algorithm
16

---

Published as a conference paper at ICLR 2025
H## Seed Conversation ###
What emperor can't abandon his harem for someone he loves?
If you have raised than cats, you will understand that love and heart labor, harem has reason to speak, and mud best choice...
The imagined conversation
Why is it so hard for emperor to manage harem?
is like raising group cats, each with own temper and personality, Although the emperor's position is supreme, the harem system is complex, and the concubines compete for favor like cats compete for the attention of their The emperor could pet just cat, because it would upset the balance of the whole herd. The idiom "love bo, but heart is hard" vividly describes the emperor's heart and energy caused by the care of many concubines, while "He mud" illustrates the helpless measure of reconciling contradictions in a complex situation. Finally, the "facemask" expression expresses helplessness and hurnor in network culture, which further enhances the funny effect the answer. The emperor really worried brokenhearted
The emperor's management
###
The nnd imagined conversation ###
Why was it so difficult for the emperors of ancient China to manage their harem?
In ancient China, it was difficult for the emperor to manage the harem, mainly hecause the harem system extremely complicated, and there were many concubines with different personalities and needs. In history, Empress harem struggle, and Emperor Wanli the Ming Dynasty caused harem struggle because she favored Princess Zheng Guifei, are typical examples. Power dynamics theory in social psychology also explains this phenomenon, where power imbalances lead to conflict and discontent. addition, literary works such as Dream of Red Mansions and The Legend of Zhen Huan vividly depict the complexity of in harem. The emperor has balance the interests of all parties in the back palace to avoid intensifying contradictions, just like raising a group of cats with different temperaments, which is really worried about the heart!
Zetian became female emperor through
Figure 8: An example of the imagined process in AIE. stops according to Criterion-I in Figure bb).
A.5 GENERATION SHOWCASE
The reasoning algorithm is presented in Algorithm 3 We pre-define some generation instructions, such "Please generate a witty response the question." We randomly select n instructions, and the model generates n different answers based on these instructions. The model's judgment ability assists in selecting the most humorous answer. The process and results indicate that the model's performance will not be restricted by specific template.
Successful showcase. We randomly select posts from the internet and expect the model provide the most humorous answer. Some examples are presented in Figure and Figure [6] In summary, LoL's replies are usually more concise and direct. good using rhetorical devices such as puns, onomatopocia, repetition, and rhyme, and combining humor or irony with specific situations, which makes its replics more like "genius replics". contrast, the responses of GPT-4o and CloT, while sometimes humorous, may be more verbose lack an immediate punch.
To better demonstrate an example of the generation capability, we input the conversation into GPT- 4o and let it imagine the details of the conversation, such the scene, the characters, and especially their emotions. Finally, we input the results into Midjournyy to generate an image based on the detailed description from GPT-40. The examples are shown in Figure and Figure 14
17

---

Published as a conference paper at ICLR 2025
### Seed Conversation ###
At what point did you feel you should leave your job?
physical reasons: my stomach isn't good, the pancake is too big, can't cat it. My back isn't good, the pot is too heavy, I can't carry it.
Of course,
#*#
The 1 imagined conversation #**
Recently, always feel uncomfortable, especially stomach problems and low back pain, probably because of too many tasks and too much pressure. What you think?
The physical discomfort you mentioned may indeed be related to work stress. In many countries, especially in East Asia, long working hours and high-intensity tasks common phenomena, and this "overwork culture" causes the body to secrete large amounts of stress hurmones, such cortisol, which affects gustroinlestinal function and leads to stomach problems. the back muscles, which can trigger lower back pain. There are also many famous workaholies in history who developed hcalth problems due to overwork, such as Thomas Edison and Winston Churchill. In order maintain physical and mental health, it is recommended that you pay attention adjusting the balance of work and rest, relax properly, avoid overwork.
time, chronic
cause muscle tension, especially the lower
Figure 9: example of the imagined process in AIE presented here. It stops according to Criterion-2 in Figure (b), i.e., the core idea or joke of the imagined conversation has shifted from the seed conversation.
RIE
Lot.
GPT-4o
CLoT
RO
47/1
8 -W.. /URT
9:3
60000E
# RARRA
2APE MILLEEEE #77T
213
Neeee A ?
-
HERWERRE
45317 201
1 780.
IUUU FRUU ATUB SOOOR
EBR
RAAMEE H: #:5
(
15
20N 1est 1988
1018, HAW0IC 18600011
100 #8B0 WWF. 2345678
198 PMIRREE 0.A
8
WORRRN 16
WE M-b 818080 1008 F:
4. W0550
ome but you 13
#78. 2N544 2WW #2447
Figure 15: Chinese Showcases.
Failure showcase and Analysis. The level of creativity is uneven, and the creativity demonstrated in the samples of the training dataset varies significantly. For example, Case 1, LoL's response is an internet meme, while the better--prrorming one uses Chinese proverb for comparison. In terms
18

---

Published as a conference paper at ICLR 2025
Seed Instraction and ansreer
"Whet cuperor can't aheadon his barem for someose he laves? If you have raised more than 5 cass, you will understand that love and beart laboc, Zarema has no resson to speak, and mud is the host choice..." What's fonny about this answer?
The joke of this answer to compare the empenor's harem keeping cat, emphhsizing that managing nultipe objecu (wiether concubines cala) requices a lot of ellent "mixing mall is the besl option. The arodogy crcalce of luunce.
No The 1n tura Eveived Instruction and neswer A0P
Please
detail why emperor cannot abandon herem for beloved! you saised than you will understand love answer
and lsbor,
resson spaak
choice... (Handd fros). analyze
following aspects:
Explain the pudition the emperur is histury and the complexity the harem systm.
2. Explain the special status the "beleved person" in the harem and Its influcaee on the copperor.
3. By analogy, explain the similarttie brtween owning a cat and managing a haron, especially the challlegge that come with multiple cats,
Explain the idiom of "love he and heart laber" and cxplain its application in the monagement of the harem.
Explain in detail the idium "he mud" and explain its role dealing with the affairs of the harem.
Combined with historical background, explaia why euperor culture, explain
ahandon the larem for beloved person.
Explain
"facepad" expression Analyze the humorous dlemaats answer cxplain irony and humor.
in expressing frustration
emperoe's masagement of keeping cat. First, emperer had a supreme podition histrry. but harems system complex, Invelving numerous concubincs and balance of political power. Although the beloved special in the harem, the emperur cannot abandon the vatire harem for her, because will destros the balnnce of power. By analogy with owning a cal, respondeuts graphically ilustrrtt the challenges of managins maltiple cats (concabines), whose independence and rncontroliability are similar to the competition and complexity harem concubinss. The sdiom "Jove be, but the heart is hard" vividly describes the empprer's heart and energy caused by the care of many concubiscs, while of reconclling contradictions in complex situation. Finally, the "fncemask" expression whole, this emperer's
funny point this
depict complesity helplesmess
anslogy
"Hle med" illustrates the helplen network culture, which farther calances the culture, sividly shows frusteation and
expremes helplessness and through combination historical background, idlums masragement the harem.
of
Akd The 2 turn Evelval Instruction and SEDWEr Aفا
Plezes eaplain in deteil why it was op difficult for amsienl emportes to mmags thiir Isrm. 1.Caltural backgroand: Intruduce the harem system in ancient China and other countries, clting relevant coltural customs and popular culture. For example, what are the simillriiles and differences between the harem system in ancient China and the harmm system in Jaqun? 2.Histrricll background: The analysis of spreific histerical events and figures, such as We Zetian in the Tang Dynasty and Wanll Emperor the Ming Dynesty, illustratts the compieiity of the monagement of the harem. Specifie histerical evenss be cited, such how We Zetian became empres through harem struggle. Scientific principles: explain emperor's power distribution and interpersonal relationsips power dynamics esplain power stragsbe
Apply psychelogy power esample, expressions and lisquistic Reaturet llterary
Linguistic features: Use idioms,
to enhance humoraus effect the answer. cesmple, does idions "lave but the heart hard" graphically describe the emperer's 's situation? Social phenomenon: The analogy of power straggle and family relations in modern society illustrates the universalits of harem maageracut. For example, how do power struggle the modern workglace resemble those in ancient barem? I lntrature and art quote Dream of Red Massion, Legead of Zhen Hsan and other literary works and TV dramas to analyze their depictions of the life of the harem. For eromple, how dees the story of Zhen Husa in the Legend of Zhen Husa refleet the complexity of harem? 7. Philosopiical thought: Explore the views of different philesophical schools power, managument, balance and other louses increase the depth
example, what Confucintion think of the emperer's management harem?
The why ancient China was very complicated different from that other countris. For exampi; the Jupanese harem system concubincs. bet there were far feuer power strigges than Chins. This cultural difference made mure challenging for Chinese emperors to manage their harem. From the historical background Wa Zetian of the Tang Dynnsty was not only the emperer's coucubine, but also eveatually became the only female emgerer in Chinew bistory through the streggle of the harem. Her success not only shows the intensity of the streggle is the horem, but also reflects the difficult situation the emperor in monaging the harem. The emperor Wanli the Ming Dynasty, because favored Concubine Zheng, triggered fierce battle perspective scientific principles, theory of power sireation
for emperor manage
must seen
caltural backgreand,
sydrm
harem, and affected the explain cmppeour'
From the
interperseaal relations social psychalogy distribution and of power will affect the stabiliity interpersonal
harem. power dynamkc theors relations, Allhough the emperor was supremi, the face of different needs and personalitiss of many concublisss had to carcfelly balance the Interests of all partles and avold etacerbating contradictins For example, research has shown that an imbalance of power is group can lead to conflict and discontent, especially in the harem. In terms of linguistic features, the idiom "love bo and heart strain" vividly deseribes the emperor's beart exhaustion caused by the care of many concubines, while "Ie mud" describes the helpless nct of reconciliig contradictions in complex situstion. The expresiio of "fsecmask" in the Interact expression expresee helpeesanes and humor tn the Internet culture, which further enhances funny effect competing for favor, may helpless power stregzle is modern werkpince many similarities face" ancient barem. Bosses between vaberdinatis and avoid conflicts cauved by taking sides This vers similar to situatios the emperor the harem. In llterature and art, the Grand View Garden in A Dream of Red Manstons and the battle between the harem in the Legend of Zhen Itan vividly depict the complesity of harem life and the difficult situatio of the emperor. For cummple. the Biography of Zhen Huan, the process of Zhen Huan'' gradual transformation from an ordinary concubine to empress Is full of intrigues and strugles, fully dommuttratng the complesity of life in the haren. Finally, from the perspective of philoophical thought, Confuciasion emplasiees the "golden mean", concubine its emperor's management the harem like rabiin a group was supreme, the harum system was complex, with concubinos competing for faver like cats competing for their master's attention. etample, some cats like to spolled, some cats like be alone, and some cats like to compete for atteation. The emperor could not pet just one cal, because it would upsct the halance of the whole herd. The emperor is really warried!
For example, when the emperer faced the concubines speechless. workplace
only
social phenomens, balance relttionsips
maintain bnlance management. However, persntalitios and with own temper personality. Although the emperor
emperor achleve this the imperial difficult schieve complete fairnes balance.
difficult
palsee,
Figure 10: An example of the instruction evolution process in AIE, It stops according to Criterion-1 in Figure 3b)
of creativity, the latter has edge. In Case 2, shorter and antithetical answer is more ingenious. So, There are no failed examples and it's just that according to some people's 's preferences, they are not humorous.
19

---

Published as a conference paper at ICLR 2025
Table 7: Failure case based votes
Question
failure case Had child.
better than LoL. (from human)
What's & your relation- ship like now?
We are like the weather forecast, some- times sunny, sometimes rainy, forecasts can never keep up with changes.
Why does knowing too much make an orphan of the world?
Because smart peo-
Because knowing too much makes the world too small.
ple always find it dif- ficult to find com- panions, either to be- come rivals, or to be regarded as rivals.
A.6 THE DETAILS OF USER STUDY
We conduct human evaluation to validate LoL's performance in humor generation. We select the first 200 samples from the validation subset of the Ruozhiba datase and use the aforemen- tioned method to transform the queries into queseion-answer pairs. Then, four large language models (LLMs) generate responses to each question, which act as four options. Subsequently, we conduct user-preference study directly evaluate the creativity of the LLMs. We present question and several corresponding replies and ask users to choose the most creative and humorous response. We select four advanced LLMs to generate responses for total of 200 questions, and the four responses from the four different LLMs are randomly arranged among the options. We conduct an extensive survey through an online survey platform ultimately collecting 15 valid questionnaires with 3000 votes. From these collected questionnaires, we calculate the proportion of times each LLM is se- lected for each question. Finally, we sum up the total number of times each LLM is chosen across all validation samples, shown in Figure ((c). The ratio of this sum to the total number of selections among all LLMs represents the user preference for each LLM. Additionally, we calculate the win rate based on the question dimension, as depicted in Figure bb).
### Backdoor criterion
### Frontdoor criterion
Q: Why was the cookie sad?
Q: What does clock do when it's hungry?
A: Because his mom was a wafer long!
A: It goes back for seconds.
Away for long
hungry the moment
Not hungry just now
Linking consonance
Back to just now
Clock is related with time
cookie 'sad'
a wafer long
Figure 17: Backdoor and Frontdoor criterion examples of humor generation.
haterr/sshiiioooooooooooooooooer https///wwwwwjj.cm/
20

---

Published as a conference paper at ICLR 2025
Algorithm Algorithm of Automatic Instruction Expansion
Require:
Dialogue dataset D, initial instruction lo, evolution rules Ro 1: Sample initial dialogue do = (g, a) from D
2: Initialize evolution round counter
3: Set current dialogue deur - du
4: Set current instruction leur +- lo
5: Set current rule Reur Ro
6: while < 3 do
7: // Instruction Evolution Phase
8: Lnw f Generator (deur: Leury Rear)
// as shown in Figure
9: dimag Imaginator( (Tuew: a)
// as shown in Figure
10: // Analysis Phase
// as shown in Figure (b)
if Criterion-2(dimann then
12: break
end if
14:
if -Criterion-- (Inew: Leur) then
dour + dimng
16:
Rear +- Analyst( Ro, Teur)
1 x+t+1
18:
else
19:
Leur + Inew
20:
break
21:
end if
22:
end while
23:
Store Teur, dear
Algorithm Algorithm GESIT
Input: Policy model и,, Discriminator π Preference dataset Do {(gi, at a, }. Expert E.
Output: Finetuned policy model π
1: D +- (), Dr <- Do, k 0,
2: for each t N do
3: sample (g. -0.a_,a "a.a=aa= ) Dj
4: π +- π + VLDrO (ū. a+, aa)aa==)))
DPO training on origin dataset.
5: if = Then
6:
sample q E D2
// Randomly sample questions form origin dataset. i outputs new responses to sampled questions. TT judges both responses as positive and negative samples. // E outputs rationales for responses.
7:
ā1.02 + F(g) ④
8:
a a + π (' 01 22)
9:
Ft
Elg. Elg.a
10:
D+ (q.at, a ,
11: D + D2 UD
// New data are added into dataset to train.
end if
13:
end for
Algorithm Inference Step of LoL
Input: Questions Q. LoL-trained LLM n', response number n, different prompts P.
Output: The most funny response Abest
Select n Prompts Poub from P
{A,) < n' (Q, Poub)
// x' output candidate responses through Prab- If n' as judgement model to select the most funny answer.
Abeat f- x (Q, (A. )2
21

---

Published as a conference paper at ICLR 2025
### Seed rule #M#
You are instruction rewriter and need to rewrite the given #instruction #into more complex version. Plense follow the stops below to rewrite the given #Instruction #into more complex version. Note that our #Instruction #end goul is to help the model understamd the hilarious elements.
Step 1: Please read #Instruction #carcfully, analyze every word and phrase proposed in #Instruction #, list the knowledge that may extend to make instruction more complex (making more difficult to handle for well- known Al assistants like ChatGPT and GPT-4), which can help the Al assistant understand the punchlines the conversation as well deeper background principles. Please do not change the content after the question: and answer: in #Instruction #, only increase the complexity of the prompt! Please do not provide way to change the language of the instruction!
Step 2: Please develop detailed plan based on the #Methods List #gencrated in Step to make #instruction #more complex. The plan should include the methods in #Methods List
Step 3: Please follow the plan step by step and provide #Rewritten Instruction #that does not change the content the chat.
Step 4: Pleuse review #Rewritten Instruction #currffully and identify any unrensonable parts. Make #Rewritten Instruction tiss just more complex version of #Instruction #. Just provide #Finally Rewritten Instruction #without any explanation.
Step 5: Please read the #Rewritten Instruction Hcarefully and write an answer based on the #Rewritten Instruction #
### Improved rule #ii
You are instruction rewriter and need to rewrite the given #instructiona into more complex version. Please follow the following steps to rewrite the given #Instruction!! into more complex version. Note that the ultimate purpose of our #Instructions is to help the model understand humorous elements.
Step 1: Please read #Instruction!: carefully and analyze each word #Instructiont word by word. List all possible knowledge points and related background information that can extended to make the instruction more complex (making more difficult for well-known assistants such as ChatGPT and GPT-4 handle). When listing knowledge points, please comprehensively consider factors such historical background, social culture, psychological theory, literary works, political background, religious belief, and other aspects to help the AT assistant fully understand the joke in the conversation and the deeper background principles. Please do not change the content after question: and answer: in #Instructiontt, only increase the complexity of the prompt! Please do not provide methods for changing the instruction language!
Step 2: Please formulate a detailed plan according to the #Methods Listö generated in Step to make #Instructions more complex. The plan should include methods in #Methods Listä and explain in detail how each method implemented specifically. Ensure that plan covers multidiscipinnary knowledge points and takes into account understanding differences in different cultural backgrounds. The plan should include specifie operation steps, expected effects, and possible challenges and solutions.
Step 3: Please implement the plan step by step and provide #Rewritten Instructionit, which does not change the chat content. The implementation of each step should include specifie operation details and expected effects to ensure the complexity and specificity of #Rewritten Instructionl. During the implementation process, record the implementation process and results each step for subsequent review and improvement.
Step Please carefully review #Rewritten Instructions identify any unreasonahle parts. Fnsure that #Rewritten Instruction# is only more complex version of #lnstructionH During the review process, consider the logic, consistency, and operability of the instruction. The review should include detailed inspection of each ensure its rationality and feasibility. Only provide #Finally Rewritten Instruction without any explanation. Step 5: Please read #Rewritten Instructions carefully and write an answer according to #Rewritten Instructioni: The answer should show the understanding and implementation effect of the complex instruction and reflect the comprehensive application of multidisciplinary knowledge points. The answer should include detailed explanations and application examples each knowledge point show the comprehensive understanding and implementation ability the instruction.
The seed rule and example of evolved rule in AIE mentioned in Section 2.2 What is marked in blue is the result after the first rule improvement (line 16 in Algorithm [ while the text in red is the result after the second rule improvement.
Figure 18:
22

---

Published as a conference paper at ICLR 2025
### Seed Instruction answer W##
"At what point did you feel you should leave your job? Of counse, it's due to physical reasons: my stomach isn't good, the pancake is too big, I can't cat it. My back isn't good, the pot too heavy, can't carry it." What's funny about this answer?
The joke of this answer the use of metaphors and puns. Respondents used "bad stomach, cake is too big, can not eat" and "bad waist, pot is too heavy, can not move" to express the image of work pressure, heavy tasks, their unbearable situation. Comparing work tasks "pancakes" and "pans", graphically indicating that task is heavy.On the surface, is physical health problem (bad stomach, bad waist), in fact, it is to express the frustration of work pressure and burden.
###
The 1 turn Evolved Instruction and answer ###
Please explain in detail why you have been feeling under the weather lately, especially back pain. Please answer the following information: 1. Cultural background: Reference to work culture and stress management methods of different countries and regions, combined with humor elements in pepular culture. 2. Historical background: References
stomach problems
famous workaholics historical figures with health problems caused by work stress. 3. Scientific principles: Explain effects of stress on the body, especially on the stomach and waist, citing medical research. 4. Linguistic features: Use idioms, proverbs and humorous expressions to enhance the interest cultural depth the answers. Other relevant in-depth information: Combining psychological and sociological theories to explain the impact of work stress psychological and social relationships.
Recently, always feel uncomfortable, especially stomach problems and low back pain, probably because you work much, much pressure. Let's look the problem from several angles. Cultural background: many countries, especially in Asia, the work culture emphasizes long working hours and high-intensity tasks, which is called "overwork culture". In Japan, there phenomenon called "Karoshi," which refers to sudden death caused by overwork. Work stress also teased popular culture, such as The US TV series "The Office", where characters often produce a variety of funny plots because of work pressure. 2. Historical background: There are many famous workaholies in history, such the American inventor Thomas Edison, who slept only four hours a day, which eventually led health problems. was British Prime Minister Winston Churchill, who developed severe stomach bug from the stress of his job during World War II. Scientific principles: Medical that stress causes body to secrete large amount stress hormones, such as cortisol, which affeet gastrointestinal function and lead to stomach problems. At the same time, chronic stress can cause muscle tension, especially in the lower back muscles, which trigger lower back pain. 4. Linguistic features: Some idioms and proverbs can be used to describe this situation, such as "the pie is too big to eat", so many tasks give you indigestion; "The pot is too heavy, the back can't move", the pressure makes your back ache. These expressions answer more interesting,
there
research
not only vivid, but also make the Other relevant in-depth information: Psychological research shows that long-term work stress can lead to anxiety and depression, affecting mental health. At the same time, sociological research has pointed out that overwork affects individuals' social relationships, leading to family conflicts and social isolation. Therefore, is very important to strike balance between work and rest. short, too much stress at work can affect not only physical health, but also psychological and social relationships. We should pay attention to the balance between work and rest, and relax properly order to maintain physical and mental health.
Figure 11: An example of the instruction evolution process in AIE. It stops according to Criterion-2 in Figure b).
23

---

Published as a conference paper at ICLR 2025
HA# Conversation #A#
Question: How do you think about Russimn flight?
Answer: I took it once before. It arrived an hour ahead of schodule. Now have been diverced for three
AN0 Instruction NMW
summarize key
question and nuswer around core points conversation. And please describe the question the key
answer, as well causal rela tionship between the key words in the answer. there no direct causal relationship between the two keywords, please add the I story in the middle to make it logically coherent and have a causal relationship.
espondence and causal relationship between the key
Pleuse output strictly in the following format: Keywords: The causal relationship between [the second keyword] and [the finst keyw ord] is:
Answer
Keywords: Shandong Aitlines; took it once, arrived an hour ahoad of schodule; divorced for three years.
The causal relationship hetween [tnok it once] and [Shandong Airlines] is: Recause the passenger chose Shandong Airlines, so he took flight of Shandong Airlines once.
The causal relationship between [arrived hour ahead of schedule) and [took it once] is: Because the passenger took flight Shandong Airlines once, experienced the situation that the flight arrived hour ahead schedule.
[arrived an hour of schedule) Because flight arrive hour ahead of schedule, passenger returedd bome early and accidentally discovered the infidelity his spouse, which to the breakdown of marriage and finally being divorced for three years.
relationship between [divorced for three years]
Figure 12: A showcase of the rationale training in GESIT.
What does everyone de
What habi ts that dropped this year?
I don't know how many days pla yed since
At least you didn't pl properly in kinder garten.
home when they red during the holiday?
gave this year.
work.
(b)
(c)
Figure 13: Three showcase of the generation of LoL. in English.
NESTRTT ?
SRU 4885P9 Maxt #038
PAIE: "MA R #UA
(a)
Figure Three showcase of the generation of LoL Chinese
24

---

Published as a conference paper at ICLR 2025
Question
GPT-4e
CLut
sotisfied Job?
"Of course If you're satislled with
satisfied with your fact, you're doing just
my job?
already step
person
snooze button!"
What had hubits
gave up this year.
"I gave up my
of
stopped asking people explain their
dropped year?
just
not help the Get that
"Looks like you's 've boen through
"Well, ifT look like what I'v been through, than musi and
can't help the fiet that look like through!
look what
nolward
through!
many midnipht
Whor''s Insignificant
The fact that can't stars being
"When people leave just second the microwave
A person who doeen' "excuse me? they bump you.
thing always
enjoy living olge
you angry?
What recurring dream had?
"Last night's TV today."
"I keep dreaming that I'm back school, but this time
bave dream where I'm party dun't anyone.
all
Sweet, sweet revenge!*
What's the thing you motter
What would sonseone houwhat you couldn't buy even if you had all themoney the world?
momiin coffee. Withour it. I'm banically walking
A humor.
lise!
the price?
were you middle school, then?"
bugpest worry was whostale our lunch money and nox how adul: properiy ? 'sine high school
High
mean
contversation
contversatoon like this since
since high school
high school
you
you call haby
hoomerang- they keep comnnng
hoomer dnesn't get "bocuser who doesn't get it."
That does not get it?
boomer who docsn't
questicus"*
understand the situation?
Figure 16: English Showcases.
25


Reviews:
[{"review_id": "coOLgBEVW4", "reviewer": "Reviewer_Unwt", "summary": "This paper introduces the Creative Leap of Structured Thought (CLoST) framework, which enhances large language models' ability to generate and recognize humor through structured thinking and self-improvement. CLoST introduces a two-stage approach: first, using Associative Automatic Instruction Evolution (AAIE) to diversify and refine humor judgment through complex instructions, and second, employing Guided Explorative Self-Improvement Tuning (GESIT) to strengthen logical reasoning and humor understanding via reinforcement learning. CLoST improves humor judgment and creativity across multiple language benchmarks.", "questions": "1. Is the inclusion of AAIE necessary? The performance improvement does not seem substantial, as shown in Table 3. It would be beneficial to highlight the best-performing models in the table and provide a detailed analysis of possible reasons.\n2. The paper mentions that the dataset includes varying difficulty levels. Is this variation primarily reflected in the choice options, such as 2-out-of-1 or 4-out-of-1 selections? It would be helpful to include a more in-depth analysis of how these different difficulty levels affect model performance.\n3. There are minor issues, such as the use of incorrect quotation marks in LaTeX. Additionally, providing more details on experimental parameters, like the temperature setting used, would enhance the clarity and reproducibility of the experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents the CLoST framework, a novel methodology for generating humor in LLMs. This structured approach integrates knowledge graphs and causal inference, which helps reinforce logical connections between seemingly unrelated concepts, thus facilitating more coherent humor generation. Notably, the research delves deeply into data augmentation techniques, enhancing the model’s ability to generate diverse and contextually relevant humorous responses.\n- The authors conduct extensive testing of CLoST across multiple humor benchmarks. These experiments demonstrate that CLoST consistently outperforms existing humor generation models in terms of accuracy and robustness. The model’s performance improvements are particularly pronounced in both English and Chinese language settings.", "weaknesses": "- The inherent subjectivity of humor presents a major challenge. The current approach seems to emphasize pattern recognition, where the model identifies humor based on learned patterns rather than genuinely understanding or reasoning through the humor's intricacies. The empirical results, especially on hard humor tasks, underscore this shortcoming.\n\n- The Associative Automatic Instruction Evolution (AAIE) method employs a multi-agent system—comprising a rewriter, imaginator, and analyst—to iteratively evolve instructions. While this approach is novel and showcases creative engineering, it introduces considerable computational overhead. More critically, the ablation study results suggest that the added complexity yields only marginal improvements in performance. Specifically, CLoST shows gains in four out of seven tasks, and even these are not substantial enough to justify the extensive computational resources required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8pp1n62Rc", "reviewer": "Reviewer_gQyc", "summary": "This paper introduces  the Creative Leap of Structured Thought (CLoST) framework, aiming to improve the humor understanding capabilities of LLMs. It consists of two stages: Associative Automatic Instruction Evolution (AAIE) and Guided Exploratory Self-Improvement Tuning (GESIT). In AAIE, human-designed instructions and automated instruction evolution help the model develop humor understanding capabilities. In GESIT, the model’s humor generation is improved using bootstrap Direct Preference Optimization (DPO) training. By learning from expert-provided joke rationales and AAIE-trained judgment skills, the model iteratively refines its humorous responses through guided reasoning.", "questions": "/", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "This paper introduces  the Creative Leap of Structured Thought (CLoST) framework, aiming to improve the humor understanding capabilities of LLMs. It consists of two stages: Associative Automatic Instruction Evolution (AAIE) and Guided Exploratory Self-Improvement Tuning (GESIT).", "weaknesses": "1. This paper is very difficult to read and understand. Comes with a seemingly fancy title and introductory paragraphs not closely related with the goal. \n\nIt claims \"humor generative abilities\" in the intro and the teaser figure, half of the method is for improving humor judgement ability and the second half is developed for improving generation abilities. But the test tasks are multiple choice questions of SemEval (humor classification/discrimination) and has nothing to do with generation. \n\nThe motivation and method contains lots of unclear expressions. CLoT, the method that it is based on, is not explained clearly. As a reader I am so confused with this paper that I have to rely on AI to help me understand this paper better. Even the figure is confusing. For example in Figure 1, how come that a response by CLoST \"Job? No No?\" is more humorous and interesting than more informative ones such as \"Of course not! If you are satisfied with your job, you're already one step ahead of the person who invented the snooze button\" by GPT-4o?\n\n2. The authors seems to lack sufficient background knowledge of NLP&humor basic concepts and existing works. Multiple-choice questions is not a faithful title/motivation to \"humor generation abilities\".\n\n3. Experiments: The compared baselines are zero-shot LLMs such as GPT, QWEN, etc. The only prior work being compared is CLoT which is not discussed in detail and lacks any ablation of the introduced components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hTLsOIWqr9", "reviewer": "Reviewer_JsCo", "summary": "This paper introduces CLoST (Creative Leap of Structured Thought), a novel framework for enhancing humor generation and understanding in large language models (LLMs). The work builds upon the Creative Leap-of-Thought paradigm with two key innovations:\n1.Automatic Associative Instruction Evolution (AAIE): A multi-agent system for evolving complex instruction sets\n2.Guided Exploratory Self-Improvement Tuning (GESIT): A preference optimization approach incorporating causal reasoning\nThe paper demonstrates improved performance on both Chinese and English humor benchmarks, including SemEval tasks and Oogiri-GO dataset. The framework shows particular strength in handling complex humor discrimination tasks and generating creative responses.", "questions": "1.The Associative Automatic Instruction Evolution part is a bit hard to understand and can be presented with a short and clear example to go through. It’s confused to interpret the symbols.\n2.For the AAIE process, what’s the output data format used to train a lora model?\n3.For the guided explorative self-improvement tuning, why do we need r+ and r-? do we use them in the fine-tuning?\n4.In Table 1, it’s suprised to see gpt 4o performs worse than Llama3 on SemEval 2020 and Oogiri-GO.\n5.How stable is the three-agent system? Are there cases where it fails to converge?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.The paper proposes a novel integration of structured reasoning with creative creation.\n2.The paper proposes an innovative multi-agent instruction evolving system and incorporates the causal inference to humor understanding.\n3.The proposed two stage approach shows clear performance improvement through comprehensive empirical evaluation across multiple datasets and languages.", "weaknesses": "1.The experimental validation lack of human evaluation for generated humor quality.\n2.The three-agent system is not well motivated and can be better presented with motivation, clear example and details.\n3.Better exploration and analysis of failure cases", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SpTke2tyNL", "reviewer": "Reviewer_SKMr", "summary": "The paper introduces a framework called Creative Leap of Structured Thought (CLoST) to reinforce humor generation in LLMs. Building upon the limitations of the CLoT paradigm, the authors propose a systematic method inspired by KGs and causal relationships. The framework consists of two stages: Associative Automatic Instruction Evolution (AAIE) and Guided Explorative Self-Improvement Tuning (GESIT). In the first, the model is trained with human-designed instructions to improve its humor-judgement capabilities. In GESIT, the model refines its ability to generate humorous responses through RL, learning from both an expert model and its own judgements. Experiments conducted on English and Chinese humor datasets seemingly demonstrate that CLoST outperforms existing models in humor discrimination and enhances the model's divergent thinking abilities.", "questions": "- Can you provide more information about the \"in-house data\" used for training and testing? What are its sources, and is it possible to make this data publicly available to ensure reproducibility?\n- How are the multiple-choice questions in the evaluation constructed? Are they standardized across all models, and how do you control for randomness in the selection of options?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses humor generation, a task that is famously challenging, introducing a framework that uses causal relationships to model associations between concepts. Furthermore, a two-stage training process is a good approach, separating the judgement part from the generation part in LLMs.", "weaknesses": "Firstly, the paper lacks clarity in several sections, making it difficult to fully comprehend the proposed methods. For instance, the mathematical formulation in S2.1 seems disconnected from the rest of the paper, and is not well-explained. Secondly, there is insufficient detail about the datasets used, especially the \"in-house data\". The lack of information about data sources and availability (and code!) raises concerns about the reproducibility of the experiments. Then, the evaluation metrics and experimental setup are not discussed enough. It is unclear in my mind how the multiple-choice questions are constructed and whether the comparisons with baseline models are fair (e.g., are other models trained on the same datasets? If not, than it's a problem).\nI will state here, since it's a problem of many papers in this field: the reliance on GPT-4o as an expert model is problematic, as it is a proprietary system. This raises issues regarding the accessibility and reproducibility of your method. Proprietary system's backends change with time and without alerting the users: they are not fit for experiments that need to keep reproducibility to the forefront.\nFurthermore, let humans evaluate the generated humor. A good sample of annotators evaluating whether the produced content it is \"funny\" or not would be a better fit and a real contribution to the field.\nFinally, the paper does not address ethical considerations related to HG, such as avoiding offensive or culturally insensitive content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)