Are U a Joke Master? Pun Generation via Multi-Stage Curriculum Learning towards a Humor LLM (2024)

Abstract: Although large language models (LLMs) acquire extensive world knowledge and some reasoning abilities, their proficiency in generating humorous sentences remains a challenge. Previous research has demonstrated that the humor generation capabilities of ChatGPT are confined to producing merely 25 unique jokes. In this work, we concentrate on endowing LLMs with the ability of generating puns, a particular category of humor by preference learning method. We propose a multistage curriculum preference learning framework to optimize both pun structure preferences and humor preferences. Specifically, we improve the Direct Preference Optimization (DPO) algorithm to address the challenge of multi-objective alignment problem. Besides, to facilitate further advancement in this field, we collect a Chinese Pun (ChinesePun) dataset, containing 2.1k puns and corresponding annotations. Experimental results on both Chinese and English benchmark datasets demonstrate that our method significantly outperforms all the baseline models.

heading: 1 Introduction
text: Humor serves various purposes and offers numerous benefits for humans, including the relief of anxiety, avoidance of painful emotions, and facilitation of learning (Buxman, 2008). However, it remains a challenge to let machines generate humorous sentences as humans (Jentzsch and Kersting, 2023). Empowering LLMs with a sense of humor has become an increasingly important topic in the field of natural language generation. As a particular category of humor, the creative utilization of puns, wordplay, and ambiguity plays a significant role in humor generation (Chiaro, 2006). Puns, utilizing identical or phonetically similar words, are categorized into homographic and homophonic types
* Corresponding author.
(Miller et al., 2017). Homographic puns play on different meanings of the same word; for instance, ’all right’ can signify ’satisfactory’ or ’not left’. This word creates humor through contrasting interpretations within context. On the other hand, homophonic puns involve words with similar sounds but distinct meanings, like ’weak’ and ’week’.
As pun generation aims to generate a pun to create humorous effects given a pair of pun word and alternative word as shown in Table 1, it substantially contains two targets: pun structure and humor effect. Generating puns requires vast world knowledge and common sense (Sun et al., 2022b). Due to the constrained model size, previous approaches (He et al., 2019; Tian et al., 2022; Sun et al., 2022b) primarily achieve limited performance improvement. With the rise of LLMs, the ability to generate text has been greatly improved. However, Jentzsch and Kersting (2023) reveal that LLMs’ proficiency in generating humorous sentences and puns still remains a challenge. To further improve the capability of LLMs on an alternative aspect, preference learning/alignment has become one of the most commonly used approaches (Casper et al., 2023). Directly utilizing preference learning methods like Direct Preference Optimization (DPO) (Rafailov et al., 2023) in pun generation is not very efficient because we need to align both the pun structure preference and humor preference at the same time on a relatively small-scale dataset.
To address this problem, we propose a multistage curriculum learning framework to generate a sentence satisfying both the provided pun structure and high humorous level. Inspired by the paradigm of curriculum learning which learns samples from easy to hard, our model learns the two preference targets separately from easy to hard, i.e., first learns pun structure preference in stage 1, and then learns humor preference in stage 2. Specifically, to address the inherent catastrophic forgetting problem
878
Type Dataset Pun pun word, alternative word
homophonic ChinesePun 篮球队教练在开会时抱怨道：“我们球队的配置太差了，现在需要一个投篮的。”过了一会， 助理教练领着机器猫走了进来：“教练，你看他的头蓝吗？” The coach of the basketball team was in a meeting and complained, "Our team is so poorly set up, we need a shooter now." A little while later, the assistant coach walks in with Robot Cat: "Coach, do you see his head in blue?" 头蓝,投篮 the head is blue, shoot a basketball
SemEval I lift weights only on Saturday and Sunday because Monday to Friday are weak days. weak, week
homographic ChinesePun 路上看到消防车开过去,大人说:他们去救火. 孩子说:救火?应该是灭火,火是坏的,还要救它呀！ The adults said as a fire truck drove by on the road: ’They’re going to put out a fire.’ The child replied: ’Put out a fire? Shouldn’t it be extinguishing the fire? Fire is bad; why would we save it?’ 救,救 put out the fire, save sb’s life
SemEval Did you hear about the guy whose whole left side was cut off? He’s all right now. all right, all right
Table 1: Examples of homophonic and homographic puns in the ChinesePun and SemEval 2017 Task 7 dataset. Each annotation includes a pair consisting of the pun word and its alternative word. To facilitate better understanding for the readers, we have translated the Chinese examples in the table into English.
of these kinds of multi-stage preference learning, we introduce an improved humor preference aligning algorithm in stage 2. Different from standard DPO which learns from pairs of positive and negative samples, we make each training sample a triplet by adding an extra generation output from the model of stage 1 by the same input prompt. We propose a new loss function to adapt to this multiobjective alignment task and to alleviate the effects of catastrophic forgetting.
Additionally, previous researches have focused on the English pun dataset from SemEval 2017 task 7 (SemEval) (Miller et al., 2017) for evaluation. The dataset contains 1298 homographic puns and 1098 homophonic puns annotated with pun words and alternative words. To verify the broad applicability of our method in different languages, we manually annotated the first Chinese dataset in the field, which is called ChinesePun. Table 1 shows examples in our dataset and SemEval 2017 Task 7 dataset. This dataset comprises 1,049 homophonic puns and 1,057 homographic puns, where each pun is annotated with its corresponding pun words and alternative words. The experimental results on both datasets indicate that our proposed method effectively enhances the ability of LLMs to generate puns and significantly outperforms existing methods. Our contributions are summarized as follows:
• We introduce a multi-stage curriculum learning framework with a novel triplet preference learning method to enhance LLMs’ ability to generate humorous puns.
• We release the very first dataset for the Chinese pun generation task, aiming to stimulate advancements in the field of Chinese humor understanding and generation1.
• We conduct extensive experiments on both the English and Chinese datasets to demonstrate the superiority of our proposed method.

heading: 2 Related Work

heading: 2.1 Pun Generation
text: Previous researches have focused on the SemEval 2017 Task 7 dataset (Miller et al., 2017), which contains 1,298 homographic puns and 1,098 homophonic puns, annotated with pun words and alternative words. Prior works on pun generation primarily targeted phonological or syntactic patterns over semantic ones (Miller and Gurevych, 2015; Hong and Ong, 2009; Petrović and Matthews, 2013; Valitutti et al., 2013), sacrificing flexibility. He et al. (2019) used the local-global surprisal principle to create homophonic puns, whereas Yu et al. (2020) utilized constrained lexical rewriting for the same purpose. Hashimoto et al. (2018) employed a retrieve-edit approach for homographic puns, and Yu et al. (2018); Luo et al. (2019) proposed advanced neural models like constrained language models and GANs. Mittal et al. (2022) generated homographic puns from polysemes and sought to incorporate their multiple senses. Tian et al. (2022) proposed a unified framework for both homographic and homophonic puns, leveraging humor principles. The keyword-conditioned pun generation setup can also facilitate more engaging pun generation scenarios such as context-situated pun generation (Sun et al., 2022b). However, all the aforementioned methods are designed specifically for small-scale models. To the best of our knowledge, our work is the first to enhance the pun generation capability of LLMs.
1Code, data, and resources are publicly available for research purposes: https://github.com/cubenlp/PGCL.

heading: 2.2 Preference Learning
text: Preference learning/alignment, especially via Reinforcement Learning with Human Feedback (RLHF) (Ziegler et al., 2020), is a prominent trend catalyzed by advancements in LLMs like ChatGPT (Ouyang et al., 2022), LLaMA (Touvron et al., 2023), and Baichuan (Yang et al., 2023). However, RLHF faces challenges such as instability, inefficiency, and vulnerability to exploitation (Casper et al., 2023; Skalse et al., 2022). Addressing these issues, novel methods have emerged, notably Direct Preference Optimization (DPO) (Rafailov et al., 2023), which aligns models with reference ones using paired preferences, enhancing efficiency and stability. Statistical Rejection Sampling Optimization (RSO) (Liu et al., 2023), an advancement building on DPO and SLiC (Zhao et al., 2022), employs rejection sampling for more effective optimization. Additionally, Xu et al. (2023) introduces a contrastive post-training curriculum, progressively shifting from simpler to harder preference pairs to improve efficacy. In contrast to these works, we introduce a new multi-stage curriculum learning framework and a triplet preference learning loss aiming at enhancing the stability of multi-objective preference alignment in the task of pun generation.

heading: 3 Problem Definition
text: The input to our system consists of a pun word pair, which includes a pun word (pw, e.g., weak) and a corresponding alternative word (aw, e.g., week). pw are defined as words that evoke humor by possessing conflicting and ambiguous meanings within jokes. In the case of homophonic puns, the pun words have a homonym aw with the same or similar pronunciation but different meanings. For homographic puns where the pun word carries dual meanings that are logically coherent within the context, we adopt a same representation, setting pw = aw, following Tian et al. (2022). The objective is typically to generate a pun that contains pw with high humorous level, e.g., "I lift weights only on Saturday and Sunday because Monday to Friday are weak days.". Thus, the generated pun should comply with two conditions: pun structure (e.g., containing pun word) and humorous.

heading: 4 Methodology

heading: 4.1 Multi-Stage Curriculum Learning
text: Different from direct preference alignment on LLMs, we present a multi-stage curriculum learn-
ing framework to offer a smoother and more effective preference learning trajectory on pun generation. We utilize the Direct Preference Optimization (DPO) method in two stages to steer the model toward optimizing two key preferences: pun structure and humor. As illustrated in Figure 1, the framework, which we called Pun Generation with Curriculum Learning (PGCL), comprises two main components: (i) a structure preference optimization module (top) designed to enhance the LLM’s ability to satisfy pun structures in the first stage, and (ii) a humor preference optimization module (bottom) aimed at aligning with the more challenging humor preference in the second stage.
Structure Preference Alignment In this stage, we optimize the structure preference of LLMs with the DPO algorithm. Given a pun word and alternative word pair (pw, aw), we transform it to a prompt x with the designed template (e.g., for English homophonic puns, the prompt template is "Given the pun word ’pw’ and its homophonic alternative word ’aw’, please generate a humorous homophonic pun. Ensure that the sentence reflects the contexts of both words and make sure the pun word ’pw’ is present in the sentence.". Other prompt templates can be found in Appendix A).
DPO is an offline preference optimization technique that takes a pre-computed pair of positive samples y+ and negative samples y−, both corresponding to the same prompt x. We make the labeled pun in the training dataset the positive sample y+. Then, we produce negative samples using SFT LLMs (e.g., LLaMA2). Specifically, we randomly generate a pun with the prompt x, and introduce a structure discriminator to judge whether the pun satisfies the structure: 1) for English homographic and Chinese homographic puns, as pw = aw, we actually only need pw present in the sentence. 2) For English homophonic puns, based on the settings of SemEval dataset, the sentence must have pw without aw. 3) For Chinese homophonic puns, both pw and aw must be presented in the sentence. If the generated pun does NOT comply with the above structure, we accept it as the negative sample y−. Consequently, we construct the structure preference pairs (y+, y−s ) to do structure DPO.
Following the standard DPO algorithm which aims to increase the likelihood of the positive example while reducing that of the negative example. The loss function of the structure DPO is presented
as follows:
r+(θ) = β(log πθ(y +|x)− log πsft(y+|x)) (1)
r−s (θ) = β(log πθ(y − s |x)− log πsft(y−s |x)) (2)
Lstructure−dpo(θ) = − log σ(r+(θ)− r−s (θ)) (3)
where β is a temperature hyperparameter; πθ is the language model to be optimized and πsft is the SFT language model; σ is the sigmoid function; r+ and r−s are the two pseudo-rewards that resemble the reward function in RLHF. DPO requires the logits from the frozen reference model (i.e., the SFT model) for both the positive and negative sequences. It enhances the performance by increasing the discrepancy between the r+ and r−s .
Humor Preference Alignment In the second stage, we attempt to use the similar method to further optimize the humor preference. We use the same prompts x and positive samples y+ in the training dataset with the first stage. Then, we generate the candidate sentences from the LLMs which have been structurally optimized in the first stage. Similarly, we train a humor discriminator to judge the humor level. Specifically, for the English humor discriminator, we follow Mittal et al. (2022) to fine-tune RoBERTa-large (Liu et al., 2019) using the ColBERT dataset (Annamoradnejad and Zoghi, 2022), which comprises 200,000 examples of jokes and non-jokes for humor detection. For the Chinese humor discriminator, we use the humor
recognition dataset from Chen et al. (2023), which comprises 18709 examples of jokes and 7709 examples of non-jokes, and fine-tune RoBERTa-large (Cui et al., 2020, 2021) model on the dataset. The output probability from the classification model is then employed to select negative samples y−h . Consequently, we can get the humor preference pairs (y+, y−h ).
Subsequently, we employ the following loss function to optimize the humor preference on the structure-optimized LLM of stage 1:
r+ϕ (θ) = β(log πθ(y +|x)− log πϕ(y+|x)) (4) r−h (θ) = β(log πθ(y − h |x)− log πϕ(y−h |x)) (5)
Lhumor−dpo(θ) = − log σ(r+ϕ (θ)− r−h (θ)) (6) where rϕ+ and r−h are the corresponding pseudorewards; πϕ is the language model that has been structurally optimized in stage 1.

heading: 4.2 Improved Humor DPO
text: Due to the limited training data, multi-stage preference alignment faces the issue of catastrophic forgetting, meaning that the effect of structure preference alignment decreases in stage 2. To mitigate the impact of catastrophic forgetting, we propose an improved triplet humor alignment DPO in the second stage.
Specifically, in order to preserve the model’s preference objectives of the previous stage, we initially sample candidate sentences from the structurally optimized LLMs of stage 1 using the same
prompts x. The sentences that satisfy the pun structure by our structure discriminator described in Section 4.1 are retained, which we call "rumination" sample y∗. Then, we select negative samples y−h∗ among the candidate sentences that lack humor and do not satisfy pun structures with both the structure discriminator and humor discriminator. We use the same positive samples y+ with the standard humor DPO we described in the last section to construct the humor preference triplets (y+, y−h∗ , y
∗). Then, we optimize the humor preference of the
LLM with a new triplet loss:
r−h∗(θ) = β(log πθ(y − h∗ |x)− log πϕ(y−h∗ |x)) (7)
r∗(θ) = β(log πθ(y ∗|x)− log πϕ(y∗|x)) (8)
LI−humor−dpo (θ) =− log σ(r+ϕ (θ)− r∗(θ)) − log σ(r∗(θ)− r−h∗(θ))
(9)
where r∗ means the pseudo-reward of rumination samples; r+ϕ is same with Equation 4. The objective is to maximize the difference in reward values between positive and rumination sequences, as well as the difference between the reward values of rumination and negative sequences. We can obtain the LLM which is well aligned with both structure preference and humor preference through the above loss function.
5 ChinesePun Dataset

heading: 5.1 Data Preparation
text: To construct a Chinese Pun dataset, we first gathered original jokes from various sources, including Github2 and humor websites 3, and obtained a dataset of 28,269 jokes. Some of these jokes have been labeled as homographic or homophonic types, while others have yet to be classified. Subsequently, we developed an automatic approach for labeling the remaining unclassified data.
We introduced a classification model to assist us in selecting homographic and homophonic puns. To achieve this, we trained the RoBERTa-Large (Liu et al., 2019) model on the humor classification dataset from CCL2018 Task44, which includes homographic puns, homophonic puns and reverse jokes that have already been classified. We used the
2https://github.com/liuhuanyong/ ChineseHumorSentiment
3http://www.jokeji.cn; http://www.yiyixh.com/a/ wenzixiaohua; http://m.jokedw.com/joke
4http://www.cips-cl.org/static/CCL2018
model’s output to extract candidate sentences that may contain homographic or homophonic puns. Upon processing the unclassified data, we combined the classified puns with the candidate sentences to obtain 2,394 pseudo-homophonic puns and 4,147 pseudo-homographic puns, which are used for the pun words annotation.

heading: 5.2 Dataset Annotation
text: To begin with, we utilized few-shot prompting on ChatGPT (Ouyang et al., 2022) for pre-annotation to decrease the amount of manual annotation required. Specifically, we first randomly selected three sentences each from homophonic puns and homographic puns and manually annotated them. Then, we used the original text and corresponding annotation results as examples and concatenated them with an instruction as a prompt, which was inputted into ChatGPT. We processed each sentence using ChatGPT and obtained shallow annotations.
We then revised the original text and corresponding annotation results manually. Three postgraduate students worked together to complete the annotation of the pun dataset. To ensure agreement among the annotation results, any disagreements were discussed by the entire group, and the final result was determined by the option receiving the majority of votes. To check inter-annotator agreement (IAA), we collected multiple annotations for 150 instances and measured agreement using Fleiss’ kappa (Fleiss and Cohen, 1973) (κ = 0.68), suggesting high agreement.
During the annotation process, we carefully reviewed each sentence to determine whether it contains a pun. If a sentence is found not including a pun, we removed it from the list of sentences to be annotated. This step ensured that we focus our attention solely on the sentences that require annotation and helped to improve the efficiency and accuracy of the overall annotation process.

heading: 5.3 Data Statistics
text: Upon completion of all the annotation processes, we report details of the dataset in Table 2. The
dataset contains 1,049 homophonic puns, 1,057 homographic puns, and 187,315 words in total. Upon analyzing the count of pun words in our dataset, we discovered that the majority of puns include only one pun word pair. Consequently, our study narrows its focus exclusively to this scenario.

heading: 6 Experiments

heading: 6.1 Dataset
text: To prove the effectiveness of our method, we conduct experiments on ChinesePun and SemEval 2017 Task 7 dataset. Table 3 shows the data statistics of both datasets. For the ChinesePun dataset, we partition it into 1684 samples for training and 422 for testing. Regarding the SemEval 2017 Task 7 dataset, we allocate 1918 samples for the training set and 478 for the testing set. Both training and testing sets maintain an equal distribution of homophonic and homographic data.

heading: 6.2 Metrics
text: Automatic Evaluation To measure the model’s ability to incorporate pun word pairs in the final generation, we utilize the structure success rate (Structure Succ.) as our primary automatic evaluation metric, following Sun et al. (2022a,b). To evaluate diversity, we follow Luo et al. (2019); Yu et al. (2018); Mittal et al. (2022) to calculate distinct unigrams (Dist-1) and bigrams (Dist-2) in terms of sentence level and corpus level. Average sentence length (Avg-Length) is also reported.
Human Evaluation 50 pun word pairs are sampled randomly from the test dataset for human evaluation. To evaluate humor capabilities, we conduct a Human A/B-test to compare our models with ChatGPT. We pair the sentences generated by models and ask annotators to choose the better sentence based on the humor. If the difference is not significant, a tie is allowed. To evaluate the success rate of pun generation (Pun Succ.), following Mittal et al. (2022), we invite evaluators to classify each
sentence into one of two categories: "pun" or "nonpun". We use the pairwise kappa coefficient to measure the inter-annotator agreement (IAA). The average inter-annotator agreement of all evaluators for humor capability and pun success are 0.47 and 0.58, meaning that annotators moderately agree with each other. Details of the human evaluation are provided in Appendix D.

heading: 6.3 Baselines
text: AmbiPun AmbiPun (Sun et al., 2022b) is the state-of-the-art pun generation model fine-tuned on T5 (Raffel et al., 2023), which utilizes "generate sentence: {pw}, {aw}" for homophonic puns and "generate sentence: {pw}" for homographic puns as the prompt. To adapt to the Chinese pun generation, we fine-tune it on ChinesePun dataset.
LLMs LLaMA2-7B and Baichuan2-7B are used for generating English and Chinese puns respectively. We fine-tune these models through the standard DPO, denoted by LLaMA2dpo and Baichuan2dpo and instruction tuning denoted by LLaMA2sft and Baichuan2sft as the baseline models. For the standard DPO method, we select the negative samples with humor discriminator and structure discriminator simultaneously as in Section 4, and choose the labeled puns from pun datasets as the positive samples. Besides, we also choose ChatGPT as a baseline, which is used to generate puns through a few-shot prompt as shown in Appendix A.

heading: 6.4 Experiment Setup
text: The ChatGPT model we use is gpt-3.5-turbo.The LLaMA2-7B and Baichuan2-7B are acquired from huggingface Transformers5. We adopt AdamW optimizer (Loshchilov and Hutter, 2017), set learning rate as 5e-5 and batch size as 4. We employ the LoRA strategy (Hu et al., 2021) for fine-tuning the LLMs. For the hyperparameter in DPO training, we set β = 0.5. The temperature is set to 0.95, the top-p is set to 0.95, and the top-k is set to 5 in the decoding strategy. For the preference data sampling, we require the number of preference data pairs or triplets to reach 10,000. The details of preference data sampling can be found in Appendix B. The entire project is based on the LLaMA-Factory6, and all other settings are default parameters. All our experiments are performed on 2 RTX 3090.
5https://github.com/huggingface/transformers 6https://github.com/hiyouga/LLaMA-Factory

heading: 6.5 Results
text: Table 4 displays a comparison of results between our model and baselines, from which we can get the following conclusions. First, our method achieves the new SOTA performance with substantial improvements on both pun datasets, which proves the superiority and generalization of our approach (We show the examples of generated puns in Appendix E). Specifically, when considering the pun success rate, it obtains 16% and 22% improvements over the best results of baselines on ChinesePun and SemEval datasets. In terms of humor degree and structure success rate, it also performs consistently better than most other models. Nevertheless, when considering the diversity of the generated sentences, ChatGPT exhibits a notable increase in diversity compared to other models. This can be attributed to the fact that, unlike ChatGPT, the other models have been fine-tuned on the specific dataset. Second, it can be observed that Baichuan2dpo and LLaMA2dpo outperforms Baichuan2sft and LLaMA2sft respectively, which proves that preference alignment supports the LLMs to obtain the specific capability (e.g., structure or humor) more effectively than instruction tuning. Third, compared with the Baichuan2dpo and LLaMA2dpo, the most significant improvement of our approach is utilizing multi-stage curriculum learning to optimize the two preference targets separately from easy to hard, which indicates the effectiveness of our method.

heading: 6.6 Ablation Study
text: To evaluate the effects of different components, we compare PGCL with its variants: 1) w/o Improved Humor DPO. In this variant, we use the standard DPO algorithm instead of the improved humor DPO algorithm in the humor preference alignment process; 2) w/o Humor. In this variant, we remove the humor preference alignment stage, i.e., only learn the structure preference. We intend to explore whether using the new triplet alignment loss for humor preference optimization contributes to alleviating the effect of catastrophic forgetting.
From Table 5, we can observe that our PGCL method consistently exhibits better performance than their corresponding variants across both ChinesePun and SemEval datasets. Specifically, the structure success rate decreases by 11.02% and 8.36% respectively compared with LLMs tuned by the standard DPO algorithm in the second stage, which demonstrates that the standard DPO algorithm can not maintain good structure preferences while performing humor preference alignment. Moreover, the novel triplet loss brings significant performance improvements (i.e., PGCL vs. PGCL w/o Improved Humor DPO), with about 8.64% and 7.31% gains in structure success rate on both datasets. The improvement in satisfying pun structure further increases the pun success rate by about 6% and 10%. It proves that the new triplet alignment loss can guide LLMs to keep a balance between structure and humor preference.

heading: 6.7 Further Discussions
text: At the very beginning, we set up a "Previous" category in the following discussions to represent the model from the previous stage, which serves as the basis for the comparison experiments.
DPO vs. instruction tuning Figure 2 illustrates the performance difference between continuing to use instruction tuning and standard DPO method based on the previous stage model. The standard DPO utilizes the (positive, negative) data pairs, and the instruction tuning only employs positive samples from the data pairs. Both Figure 2(a) and Figure 2(b) prove that the DPO method significantly
improves the performance of LLMs compared to instruction tuning.
Is Curriculum Learning Effective? Figure 3 illustrates the performance of different strategies to enhance a model’s ability of multi-objective preference alignment on ChinesePun and SemEval datasets. DPO-A involves simultaneous optimization of the model’s structural and humorous capabilities. DPO-H-S signifies prioritizing the optimization of the model’s humorous capability, followed by enhancing its ability to satisfy pun structures. DPO-S-H (Ours) follows the opposite sequence. Both DPO-H-S and DPO-S-H use the new triplet alignment loss in the second stage. The details of DPO-H-S can be found in Appendix C. We can see that the success rate of the pun structure, denoted by green lines, peaks at DPO-H-S, while the humor degree, illustrated by blue and orange lines, attains its optimum at DPO-S-H. It indicates that LLMs tend to prioritize the final stage in multi-stage preference learning. However, DPO-H-S shows only slight improvements in pun structure success rates over DPO-S-H, with gains of 3.33% and 0.84% on the ChinesePun and SemEval datasets, respectively. In contrast, switching from DPO-H-S to DPO-S-H results in a substantial increase in humor win rates by 32% and 22% along with a significant decrease in humor loss rates, by 24% and 22%, respectively. It indicates that multi-stage curriculum learning enables LLMs to concentrate on hard tasks without compromising their performance. DPO-A does not enhance the model’s performance significantly, as the inherent difficulty of directly generating puns makes it challenging for the model to optimize in that direction immediately.

heading: 7 Conclusion
text: In this work, we propose a multi-stage curriculum learning approach with the Direct Preference Optimization technique to effectively endow LLMs with the capability of generating humorous puns.
Specifically, to alleviate the effects of catastrophic forgetting in the multi-objective preference alignment, we present a novel triplets preference learning schema, which is quite different from standard DPO. To verify the broad applicability of our method in different languages, we construct a new benchmark dataset, called ChinesePun, which is the very first dataset on the Chinese pun generation task. The evaluation shows that our method significantly outperforms all the existing methods in Chinese and English, even for ChatGPT.

heading: 8 Limitations
text: In our research, we concentrate on pun generation, a niche within creative language and humor production. However, our method still fails in recognizing humor’s subjective nature. Due to diverse backgrounds and experiences, what is funny to one individual might not be to another. In our future work, we will aim at offering broader insights into humor’s variability, influenced by contextual subtleties and personal interpretations.

heading: 9 Acknowledgement
text: The authors thank all the anonymous reviewers for their valuable comments and constructive feedback. This work was supported by Ant Group Research Fund & the Science and Technology Commission of Shanghai Municipality Grant (No. 22511105901).

heading: A Prompt Template
text: The prompt templats we used in SFT stage, structure preference optimization and humor preference optimization are as Figure 4 and Figure 5. For the few-shot ChatGPT, Figure 6 utilized identical prompts and incorporated three data examples as context.

heading: B Preference Data Construction
text: Structure Preference Data As outlined in Section 4.1, we initially sample outputs from the SFT
model using the same specifically designed prompt employed in the SFT step. Subsequently, these outputs are classified by a structural discriminator into groups of structurally compliant and non-compliant sentences. We selected 10,000 structure preference pairs randomly to do structural preference aligning.
Humor Preference Data We sampled outputs from the structurally optimized model and then categorized them into rumination and negative samples using the humor discriminator and structure discriminator. Concurrently, humor preference triplets are constructed based on the pun data corresponding to the used prompts, as well as the rumination and negative samples. We randomly selected 10,000 triplets for humor preference alignment.

heading: C Details in Experiment of DPO-H-S
text: As the settings of our main experiment, DPO-H-S uses the standard DPO loss at stage 1 to optimize the humor preference. We generate candidate sentences from the SFT model and use the humor discriminator to select the negative samples as we described in Section 4.1. The positive samples we used are puns from pun datasets. we use these positive samples and negative samples to do the standard humor DPO.
Then, DPO-H-S uses the new triplet alignment loss at stage 2 to optimize the structure preference. We also generate candidate sentences from the humorous optimized LLM which in the first stage. Then, we use the humor discriminator to identify the humorous sentence from candidate sentences. These sentences are used as rumination samples. The humorless sentences from candidate sentences are further fed to the structure discriminator to choose the samples that do not fit the pun structure. These samples are used as negative samples. The
positive samples are puns from the pun datasets. we use these positive samples, rumination samples, and negative samples to do the structure DPO with the new triplet loss.

heading: D Detailed Human Annotation and Evaluation Process
text: The evaluators are the same as in the dataset annotation because they have experience with puns. Our annotation process was carried out by a team composed of three postgraduates from languagerelated fields. Before the actual annotation process, the team underwent a training session to familiarize themselves with the tasks. The tasks included defining pun words, defining pun sentences, and locating the pun word from the pun sentence. Each sentence was annotated by all of them and any disagreements were discussed by the entire group. The final result was determined by the option of receiving the majority of votes. This process was aimed at maintaining the quality and consistency of the annotations. In human evaluation, we invited them to evaluate our experiments. In the human A/B-test for humor evaluation, we paired the sentences generated by models and asked annotators to choose the better sentence based on the humor. If the difference is not significant, a tie is allowed. In the evaluation of the success rate of pun generation, we asked them to classify each sentence into one of two categories: "pun" or "non-pun". The workers were paid 60 yuan per hour.

heading: E Examples of Generated Puns
text: A comparison between our model and the baselines is in Table 6. Our method is capable of generating humorous puns that adhere to the defined structure, akin to human creativity. In contrast, it proves challenging for other models to produce even basic humorous sentences, let alone high-quality humorous puns.

heading: Future Work
text: In our future work, we will aim at offering broader insights into humor’s variability, influenced by contextual subtleties and personal interpretations