XMeCap: Meme Caption Generation with Sub-Image Adaptability

Authors: yuyan  chen
yanghua  xiao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:14:56

Abstract: Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. While advances have been made in natural language processing, real-world humor often thrives in a multi-modal context, encapsulated distinctively by memes. This paper poses a particular emphasis on the impact of multi-images on meme captioning. After that, we introduce the \textsc{XMeCap} framework, a novel approach that adopts supervised fine-tuning and reinforcement learning based on an innovative reward model, which factors in both global and local similarities between visuals and text. Our results, benchmarked against contemporary models, manifest a marked improvement in caption generation for both single-image and multi-image memes, as well as different meme categories. \textsc{XMeCap} achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 3.71\% and 4.82\%, respectively. This research not only establishes a new frontier in meme-related studies but also underscores the potential of machines in understanding and generating humor in a multi-modal setting.

Main paper: 
 introduction: 
 As the rise of social media, memes have become a ubiquitous form of communication and entertainment. A key research task relevant to meme is meme caption generation, which involves creating captions that complements the humor or message of the image to enhance its appeal and shareability [14, 19, 20] . Meme caption generation is valuable for applications and scenarios such as social media content creation, marketing strategies, and digital communication enhancement where engaging visual content is crucial [16, 22, 30, 35, 36, 45, 60] . * The corresponding authors. The existing research efforts on meme caption generation mainly focus on using single-image memes [5, 41, 50, 53, 54] . As the example shown in Fig. 1(a) , given the image of a meme featuring a man with a pained expression, the multi-modal model is requested to generate a fitting and humorous caption that aligns with the image's content, such as "When my dad saw that I got a beating from my mom for doing something he had allowed me to do. ", indicating the ironic situation where the father permits something that the mother punishes, leading to the man's distressed expression. To achieve this, some work such as Chauhan et al. [8] and Li et al. [32] provide humor datasets from TV and memes respectively, while others such as Ritschel et al. [49] and Ritschel et al. [48] focus on robot humor.However, all the existing methods can not be adopted when there are multiple sub-images in a meme, each of which probably has different theme and emotion. As the example shown in Fig. 1(b ) which is a multi-image meme consisting of three separate sub-images stacked vertically: The first sub-image is a drawing of a character smiling with a good signal Wi-Fi icon. The second sub-image shows the same character looking very upset, with a Wi-Fi icon that has only one bar. In the third sub-image, the character has the same expression as the second one but combining with a red background with an exclamation mark inside the Wi-Fi icon, indicating no internet connection. The caption is "Connected, but no internet.". This multi-image meme tells a story of someone's deteriorating mood as their internet connectivity worsens, which is relatable to many people's experiences with internet issues. Understanding the relationships between these sub-images and their connection to the overall meme theme poses two challenges. i) Effective Integration of Composite Information: In multi-image memes, it's crucial to effectively integrate information from all sub-images to generate a caption that reflects the characteristics of each sub-image while fitting the overall meme context. ii) Handling the Complexity of Shared Captions: As the captions for multi-image memes are generally shared across all sub-images, the generated caption needs to maintain consistency between different sub-images while being flexible enough to accommodate the unique content of each sub-image.Given the absent of benchmark on caption generation for multiimage memes, in this work we construct a new meme datatset (in Chinese) including both single-image and multi-image memes. Specifically, we construct our dataset by sourcing memes from open platforms and meme websites, categorize them by structure (i.e. single-image, multi-image) and emotion (i.e. self-praise, praise of others, self-mockery, and mockery of others), and then balance the collection through downsampling to construct a representative mix of single-image and multi-image memes with varied emotions.Based on this dataset, we novelly propose XMeCap, a new approach which employs separate feature extraction for images and captions, introduces an adaptive transformation to capture the global and token-wise connections between the two, and uses the global and local similarities as supervision signals. We further enhance the LLMs by applying Supervised Fine-Tuning and Reinforcement Learning. Our reward model also incorporates these multigranularity similarities as a part of the reward signal. Through comprehensive experiments, we demonstrate superior performance in caption generation for both single and multi-image memes.To summarize, our contributions are in three-fold:• We recognize the impact of multi-images on meme captioning and offer a novel methodology named XMeCap for meme caption generation. XMeCap is characterised by supervised fine-tuning and reinforcement learning based on an innovative reward model, which factors in both global and local similarities between visuals and text. • We conduct extensive experiments to validate the superiority of our approach over current benchmarks in both single and multi-image meme caption generation, alongside promising results in conventional multi-modal humor detection tasks. • Through visualization analysis, our research further underscores the ability of our proposed XMeCap to discern intricate associations between memes and their corresponding captions, setting the stage for advanced meme-related research in the future. 
task and dataset: 
 The challenge we address is the automatic generation of captions for a diverse array of memes. The problem is two-fold: first, to correctly classify memes into two structural categories (single-image or multi-image) and second, to identify the sentiment of the meme, categorized as self-praise, praise others, self-mockery, or mockery of others. The ultimate goal is to develop an algorithm capable of generating a caption that is congruent with both the meme's structure and sentiment, enhancing the humor and communicative intent of the meme. Task definition. The meme caption generation task 𝐺 aims to produce a caption ĉ𝑖 for a given image 𝑚 𝑖 in a meme. The performance of this task can be measured by a scoring function 𝑆 (𝑚 𝑖 , 𝑐 𝑖 , ĉ𝑖 ) which evaluates the generated caption ĉ𝑖 against the ground truth 𝑐 𝑖 . The overall goal is to maximize the performance score across all memes in the given dataset as follows:EQUATIONDataset construction. We construct a large-scale meme dataset, comprising 18,110 memes from open-sourced platforms 1 2 and memes from various Chinese meme images websites. We classify memes based on their structure into single-image memes and multiimage memes. The latter emphasizes connections between individual sub-images alongside the relationship with the caption. We further categorized memes by their sentiment into four types: selfpraise, praise others, self-mock, and mock others. To maintain a balanced dataset, we undertook downsampling. The final set contains 12,320 memes with 54% single-image memes and 46% multi-image memes. In terms of sentiment types, we self-praise accounts for 21%, praise others 23%, self-mockery 29%, and mock others 27%. Caption lengths are detailed in Table 1 . 
methodolology: 
 In this section, we propose a novel Chinese meme caption generation approach named XMeCap, which incorporates visual and textual features through adaptive transformation layer and utilize image-text attention to generate captions. The framework is shown in Fig. 2 .Our proposal first separately enhances and extracts features from images and captions. Specifically, we first categorize meme images into single-image memes and multi-image memes manually. For multi-image memes, we use OpenCV-Python to precisely identify each sub-image and its boundaries and capture their coordinates by selecting regions of interest (ROIs). This process involves declaring a mouse click function to outline the boundaries of each sub-image. Once these boundaries are established, we apply image enhancement to each sub-image individually. With AutoAugment [23] , we transform each original sub-image of an image (denoted as 𝐼 ) into enhanced versions. This includes applying techniques such as cropping and rotating, which are customized according to the specific features of each sub-image. For feature extraction, we adopt a powerful large multi-modal model (referred to as LMM, we use LLaVA-1.5-7B here) as a visual encoder to extract deep features from each original and enhanced sub-image. This process allows us to capture unique detailed features in each sub-image, such as the morphology and color gradients of objects. Since the text areas lie outside the boundaries of all sub-images in multi-image memes, we consider the text as a shared caption across all sub-images. The text enhancement process, using back-translation and a Transformer-based LLM (we use Baichuan2-7B here), extracts deep features for these shared texts. These extracted features are then used for each sub-image, considering the shared text as a common caption complementing the entire set of sub-images.To generate effective meme captions, our proposal generates descriptions for each sub-image based on the LMM. Then, these descriptions are transformed into structured text with a "chainof-humor" template [15, 21] . For each sub-image, this template includes creating a narrative that encompasses the core concept (such as the main object in the sub-image), emotion (such as surprise), event (such as sharing a photo), consequence (such as discussing the photo), and humor elements (such as using anthropomorphism). Each step of this process, including feature extraction, adaptive transformation, and attention-guided text generation, takes the specific sub-image source (i.e., the index of the sub-image relative to the meme image) as input, ensuring a tailored approach for both single-image memes and multi-image memes.This process aims to effectively merge the image features and caption features of each image area into a unified space [18, 35] . Each feature uses an independent trainable linear layer for projection. This transformation is achieved through trainable weight matrices and corresponding bias terms. Specifically, for each image area (i.e. part of the image and generally not the sub-image) represented as 𝑀 𝐼 𝑖 and its corresponding caption features 𝑇 𝐼 , we apply the following linear transformation:EQUATIONwhere 𝑊 𝑀 𝐼 𝑖 and 𝑊 𝑇 𝐼 are trainable weight matrices, and 𝑏 𝑀 𝐼 𝑖 and 𝑏 𝑇 𝐼 represent the respective bias terms. Next, we use the self-attention mechanism of the LLM to calculate the token-level similarity between each area in the image and each word in the caption, denoted as 𝑆 𝐼 𝑖 . Note that the self-attention mechanism embedded in LLMs' architecture sometimes differs from the general self-attention mechanism, such as the ROPE encoding in Baichuan. The entire calculation process is as follows:EQUATIONEQUATIONwhere 𝑊 𝑄 and 𝑊 𝐾 are the weight matrices of queries and keys in the self-attention mechanism, 𝑑 𝑘 is the dimension of the key vectors, which adjusts the scale of the dot product, 𝐸 𝐼 𝑖 ,𝑗 is the unnormalized attention weight, and 𝑆 𝐼 𝑖 ,𝑗 is the normalized attention weight representing the similarity between the 𝑖-th image area and the 𝑗-th caption word. To compute the global attention, denoted as 𝑆 𝐼,𝑗 , we average the token-level attention across the image and caption to obtain a global view. The computation is as follows:EQUATIONFinally, we use the global attention and token-level attention to construct a loss function aimed at maximizing the similarity between images and captions while minimizing the similarity of unrelated combinations. This is achieved through contrastive loss, computed as follows:EQUATIONwhere 𝑆 𝐼 𝑖 ,𝑗 is the similarity score between the 𝑖-th image area and the 𝑗-th caption word, 𝑆 𝐼 𝑖 ,𝑗 is the similarity score between the 𝑖-th image area and all words in a caption (including matching and non-matching, denoted as 𝑘), 𝑁 ′ is the number of caption words, and 𝜏 is the temperature parameter, which adjusts the sensitivity of the loss function.Given the prior knowledge provided by the aforementioned global and token-level similarities representing the relevance between images and captions, our proposal adopt attention-guided text generation method including Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to improve the performance of generating meme captions [11, 13, 43, 44] . First is the SFT step, where the LLM aims to generate meme captions that are close to ground-truth captions. A key aspect of this process is to ensure that the LLM's predicted similarity aligns well with the similarity derived from prior knowledge (global and token-level). To this end, we introduce an additional loss component that utilizes the Kullback-Leibler (KL) divergence to measure the difference between these similarities, as shown below:EQUATIONEQUATIONEQUATIONwhere 𝑆 SFT denotes the similarity score predicted by the LLM for the image and its caption. Specifically, it represents the cosine similarity between the captions generated through Supervised Fine-Tuning (SFT) and the ground-truth captions. 𝑆 𝐼 refers to the similarity score between the image and its caption, calculated based on prior knowledge, including global and token-level similarities. 𝑆 SFT 𝐼 𝑖 ,𝑗 indicates the similarity score between the 𝑖-th image area and the 𝑗-th word in the caption as predicted by the LLM. 𝑆 𝐼 𝑖 ,𝑗 is the similarity score for captions obtained through SFT, which is determined by the cosine similarity between captions generated through SFT and the ground-truth captions. This score represents the similarity between the 𝑖-th image area and the 𝑗-th caption word, based on prior knowledge such as global and token-level similarities. Here, the total loss for SFT, including the original SFT loss and the introduced loss based on prior knowledge similarities at the global and token levels, becomes:EQUATIONwhere 𝜆 SFT-ori , 𝜆 g and 𝜆 t is a trainable weight. The next step involves building a reward model to align captions with human preferences. The reward model links the generated caption 𝑦 with the ground truth one 𝑡 to calculate a reward 𝑟 𝑖 = 𝑅 𝑖 (𝑦, 𝑡). In detail, we initially manually evaluate 1% of the captions generated during the SFT process with the criteria shown in Table 2 .Here, the annotators are three volunteers. They kindly provide help without any compensation. They rank according to the scoring criteria mentioned in the response to W21. These individuals have not seen the ground truth in advance, and the order is only accepted if the agreement exceeds 0.7. This process results in a ranked sequence, represented as {𝑐 1 , 𝑐 2 , . . . , 𝑐 𝑘 -1 , 𝑐 𝑘 }. Based on the calculated rewards, we construct ordered sequences for captions on each image, such as {𝑐 1 < 𝑐 2 < . . . < 𝑐 𝑘 -1 < 𝑐 𝑘 }, where a larger reward indicates a better match between 𝑦 and 𝑡. During caption generation, as tokens emerge, we compute their attention towards specific image areas. If a token's attention closely matches prior attention distribution, it receives a reward; if not, it's given a penalty. This process yields a new ranking. By balancing human evaluations with attention-based rankings, we determine the final sequence for captions generated in the SFT process. To train the reward model, we also employ "Baichuan2-7B" as the backbone, modifying its softmax layer to a linear one. This reward model takes a caption and outputs a score which represents the caption's quality. We collate captions from the final ranking sequence and apply the Pairwise Ranking Loss, depicted as follows: (11) where 𝑥 symbolizes the original caption. 𝑦 𝑤 and 𝑦 𝑙 are indicative of captions with higher and lower scores in the given ranking pair, respectively. 𝑟 𝜃 is the scalar output from the reward model, with 𝐷 being the set of ranking pairs, and 𝑘 representing the count of captions produced during the SFT process. This reward model's efficacy lies in its capacity to attribute higher scores (rewards) to superior captions and lower scores (penalties) to inferior captions, adeptly mirroring human preferences and LLM's preference.𝐿 𝑟 = - 1 𝑘 2 𝐸 (𝑥,𝑦𝑤 ,𝑦 𝑙 ) ∼𝐷 [𝑙𝑜𝑔 (𝜎 (𝑟 𝜃 (𝑥, 𝑦 𝑤 ) -𝑟 𝜃 (𝑥, 𝑦 𝑙 ) ) ) ],After that, we feed captions 𝑐 generated by the SFT model into the RL model 𝜋 𝑅𝐿 𝜙 to obtain a more human-preferred captions 𝑦. We first input (𝑥, 𝑦) into the reward model 𝑟 𝜃 and calculate a score (i.e., reward), which represents the real-time feedback from the reward model. Next, we aims to maintain similarity between the RL model and the SFT model with Kullback-Leibler (KL) divergence. Finally, we combine the two loss functions as follows:EQUATIONEQUATIONEQUATIONwhere 𝜋 𝑅𝐿 𝜙 (𝑦|𝑥) and 𝜋 𝑆𝐹𝑇 (𝑦|𝑥) represent captions generated by RL model and the SFT model, respectively, 𝑤 𝑅𝐿 1 and 𝑤 𝑅𝐿 2 are trainable weights. 
experiments: 
 In this section, we evaluate the performance of our proposed XMe-Cap framework on memes caption generation dataset, including single-image memes and multi-image memes. We conduct our experiments on four Nvidia A100 GPUs, each with 80GB of memory, using PyTorch in Python. For enhanced training efficiency, we utilize DeepSpeed. We set the maximum sequence length for both input and output sequences to 1024 tokens. The training process is set to 20 epochs. 𝜆 SFT-ori , 𝜆 𝑔 , and 𝜆 𝑡 , which control the weight of original SFT process, the weight of global similarity loss (L g ), and the weight of token-wise similarity loss (L t ), in the total loss, are initially set to 0.4, 0.2, and 0.4, respectively.𝑤 𝑅𝐿 1 and 𝑤 𝑅𝐿 2 , which adjust the weight of the reward model loss, as well as the weight of the loss for maintaining similarity between the RL model and the SFT model, are initially set to 0.4 and 0.6, respectively.Specifically, for 𝜆 SFT-ori , 𝜆 𝑔 , and 𝜆 𝑡 , we believe SFT is more important because fine-tuning has been proven to be more effective in improving the performance of downstream tasks and is supervised. Since we are capturing the relationship between each sub-image and the caption, we consider the token level to be more important. Therefore, the order of the values is 𝜆 SFT-ori ≥ 𝜆 𝑡 ≥ 𝜆 𝑔 . We try different combinations using prior knowledge, with 𝜆 SFT-ori , 𝜆 𝑡 , and 𝜆 𝑔 being 0.4, 0.4, and 0.2; 0.5, 0.3, and 0.2; and 0.6, 0.2, and 0.2, respectively. For 𝑤 𝑅𝐿 1 and 𝑤 𝑅𝐿 2 , we use prior knowledge to try different combinations, including 0.5 and 0.5, 0.4 and 0.6, and 0.6 and 0.4. Each experiment is trained with these sets of parameters, and the best-performing set was chosen. Each baseline is optimized through SFT and RL with the training set. The loss function of SFT for these baselines is only L SFT-ori while that for XMeCap is 𝜆 SFT-ori , 𝜆 g and 𝜆 t .Due to many LMMs being pre-trained on English corpora, to minimize the impact of language on model performance while preserving the characteristics of Chinese memes, we utilize Tencent Cloud's API3 to translate Chinese captions for training and evaluation. Ultimately, the LMMs output results in English, which we then translate back into Chinese for presentation.The selected datasets contain UR-FUNNY [27] which is designed for humor understanding, MUStARD [6] which focuses on multimodal sarcasm detection, MHD [47] which predicts laughter in sitcoms based on multimodal data. All results are reported on the 20% subset split from the original dataset. The remaining 80% subset is regarded as the training set. Specifically, we separately compare the results on single-image memes and multi-image memes. The prompt template for generating captions with baselines is "What is a humorous short sentence that complements the image as a meme?". We categorize our evaluation metrics into two groups: memespecific metrics and general metrics. Meme-specific metrics contain informativeness, relevance, creativity, and humorous, measuring the human-like quality of the generated captions. General metrics contain BLEU [46] , ROUGE [34] , CIDEr [55] , and METEOR [4] , measuring the relevance and diversity of the generated captions. The rating scale of meme-specific metrics are all from 1 to 5, where 1 means the worst and 5 means the best. The final scores will be scaled to 1-100. We enroll three volunteers, and each of them is required to give scores for the randomly selected 1000 memes with generated captions. We also calculate Inter-rater agreement of Krippendorff's Alpha (IRA) to ensure the confidence of human ratings. For the controversial ratings which have low agreements (<0.7), we discard this caption.Based on the presented experimental results in Table 3 and Table 4, our method XMeCap exhibits impressive performance in both single-image and multi-image meme caption generation. Compared to existing baselines, XMeCap consistently achieves higher scores across various metrics, demonstrating its efficacy. Notably, its performance is closely aligned with that of GPT4, suggesting that our method is competitive and can rival state-of-the-art models in this domain. Moreover, we test 500 samples with GPT-4o, using the same dimensions as human evaluation, and each sample is tested three times. We only use a score if at least two out of three results are the same. We find that the correlation between GPT-4o scores and human scores is very high, reaching 0.935, which indicates that human evaluation can partly be replaced with GPT-4o scoring, thereby reducing manual efforts. From the results presented in Table 7 , it's evident that our method XMeCap exhibits varied performance across different meme types, both in singleimage and multi-image formats. Interestingly, XMeCap performs notably better in memes with negative connotations, such as "Selfmock" and "Mock others", compared to those with positive themes like "Self-praise" and "Praise others". This suggests that XMeCap has seemingly captured the contrastive nature inherent in memes, finding a more conducive space for caption generation in images with a negative nuance. This insight underscores XMeCap's subimage adaptability of meme sentiment. The results from Table 8 highlight the performance of our method, XMeCap, after being fine-tuned for modal-humor detection tasks. It is evident that XMe-Cap achieves commendable results across multiple modal-humor detection benchmarks, exhibiting performance closely rivaling that of GPT4. This underscores the generalization capabilities of our approach. Take multi-image meme caption generation as an example, we evaluate the impact of each component in our proposed XMeCap for single-image and multi-image memes as shown in Table 5 and Table 6 , respectively. Specifically, w/o IA, Without Image Augmentation, refers to not applying image enhancement techniques to the original image during the feature extraction process. This means The remote control is watching me as I search for it like an idiot.When the song you chose is up next, but the current person with the microphone hasn't planned on letting go.Even the sponge is applauding for me.The fluffy hand of a cat.The long cat. I can't believe it.Dank Learning The underwater party. The cat's masseuse. I'm like a cat. Its gaze is sharper.Transformer The world's happiness is all because of me.The benefactor of the pet world.My laziness is like a cat's.It's as if I've seen the look of your secret.Every party, I'm the brightest star.Who says humans and animals are different? Just watch their fingertips communicate.My dream working state.Surely saw through you again.BLIP-2-7B SpongeBob SquarePants.The orange cat puts its paws on a person's hand.Black and white cat lying on the couch.Brown and white dog stand in the middle of the stones.MiniGPT Once the best of friends, always the best of friends.Every now and then, it's me.Smokers in their 20s. Smokers in their 30s.The wind comes, and I still stand.That dog looks so good in the water. When you're trying to quit smoking but still need that nicotine fix.Two hairy dogs. Dog and a group of ducks.I am staring at two bottles.A powerful cat and a cat on crutches.A dog and ducks walk on the beach.I am thirsity and I need two bottles of water.What the cat looks like after smoking.Qwen-VL-Chat-7B Dog: It's too hard for me, the wind is so strong that I'm out of shape.What a strange combination of dogs and ducks walking on the beach.I need two bottles of water: one to drink and the other to prove that I'm not thirsty.Cats can't escape time.This dog looks like a little cartoon character.It's a duck and it's running on the beach.He's a handsome guy and looks funny.This cat looks like it smokes, and it becomes like this when it smokes half of it.When you accidentally open the front camera.When you lied on your resume about being a great swimmer.When you buy a bottle of water, only to find that you are already in the water. Among the components, the attention mechanism displays the most pronounced effect, underscoring the significance of crossmodal interaction. The performance when removing "chain-ofhumor" further emphasizes the importance of its cross-modal bridges. Contrastive Learning effectively distinguishes the latent relationships between individual memes and their corresponding captions. However, the effects of image and text augmentation were relatively subdued, suggesting that mere alterations without changing the core content of images or captions may not substantially enhance performance.Meme caption generation hinges on pinpointing textual cues that align with the image's core humor. A successful approach should emphasize key phrases in harmony with the image's salient features [12, 17, 39, 40] . We use blue and orange to highlight the corresponding text. In the single-image meme above, "integrate" and "expose" stand out. We adopt FLIP [31] to draw a heatmap, and find that "integrate" corresponds to the yellow-highlighted region, representing a black dog among black ducks. Conversely, "expose" relates to the blue-highlighted area, marking a distinctive white feather on the dog's nose. This captures the humor of an effort to fit in yet unintentionally standing out. For the multi-image meme below, "different" and "greeting" are focal. The heatmap pinpoints two areas: a handshake and raised eyebrows. Although distinct, both gestures signify greetings, highlighting varied forms of acknowledgment. Our framework also highlights key parts in more multi-image memes such as the "head and cigarette" in both two subimages in the third meme (Fig. 4 (a)), and the "simple face" in the left subimage and the "detailed face" in the right subimage in the fourth meme (Fig. 4(b) ) with orange boxes for generating humorous captions.The case study in Table 9 and Table 10 indicates that our XMeCap closely matches GPT4's performance, notably in negative categories (self-mockery and mock others). Compared to baselines like s2s, Dank Learning, and transformer, our approach surpasses, showing a deeper grasp of humor nuances. However, the XMeCap needs enhancement in the positive categories (self-praise and praise others), suggesting further refinements. Error analysis points to challenges in accurate caption generation. Context discrepancies can dilute humor. Although XMeCap is precise in image description, it sometimes miss humor or creativity. There's also the issue of cultural sensitivity and potential offensiveness. Difficulties with multi-image memes emphasize the need for improved captioning techniques. 
related work: 
 Multi-modal humor research is expanding. Wu et al. [56] explore TV-sitcom dialogues, while Patro et al. [47] focus on 'Big Bang Theory' for humor detection. Chauhan et al. [8] and Li et al. [32] provide humor datasets from TV and memes respectively. Devillers et al. [25] consider laughter in robot interactions. Chen and Jiang [10] and Tsakona [52] address humor theories. Hasan et al. [26] and Yang et al. [57] present models for multimodal humor understanding and labeling. Alnajjar et al. [1] and Chauhan et al. [7] delve into TV shows and sentiment in humor. Regarding humor generation, Ritschel et al. [49] and Ritschel et al. [48] focus on robot humor. In meme generation, Sadasivam et al. [50] to Wang et al. [54] offer tools, datasets, and systems for memes. However, distinguishing between single and multi-image memes hasn't been a focus until our research, which offers a distinct approach for both single-image and multi-image memes.Textual humor generation seeks to produce comedic content. Templates often involve lexical changes via tools like WordNet, as demonstrated by Sjöbergh and Araki [51] in Japanese comedy and Hong and Ong [28] for puns. However, they can be formulaic. Conversely, neural models promise more originality. For example, works by Li et al. [33] and Yu et al. [58] use such models for pun creation. However, humor isn't solely textual; images play a role, which our research emphasizes. 
conclusions: 
 Humor poses a great challenge for human-machine interaction. This study has illuminated the intricate dynamics of humor in the multi-modal realm of memes, emphasizing on the impact of multiimages on meme captioning. Through our innovative XMeCap framework, we have established a deeper comprehension of meme image-text relationships, achieving state-of-the-art performance in meme caption generation as well as multi-modal humor detection. As we advance, it's imperative to explore the cross-cultural adaptability of our method, understanding the subtle variations in humor across different societies. Additionally, integrating more sophisticated semantic analysis tools could further refine the quality of generated captions. 
 
 Appendix: https://github.com/chineselzf/memeplatehttps://github.com/HumorComputing/CCL2021-Humor-Computationhttps://cloud.tencent.com/ 


Decision: The paper presents XMECAP, a system designed to generate humorous captions for sequences of images, achieving modest performance improvements over existing methods. The key strengths include the innovative combination of existing techniques for sub image identification, phrase generation, and stitching results using a "humor grammar," as well as a large, well-annotated meme database. The experimentation and evaluation are thorough, with believable examples provided. However, the contributions lack clear novelty, relying on existing methods rather than new insights, and the abstract needs to be more specific about the approach, datasets, and performance. The explanation of the system's unique contributions is vague, and the ablation studies suggest the importance of attention over the "chain of humor," which is not well-explained. To address these concerns, the authors should clarify the unique contributions of their system, provide a detailed analysis of the attention mechanism versus the "chain of humor," and enhance the abstract with quantifiable details. Additionally, improving the clarity and consistency of the figures and descriptions, expanding the human evaluation criteria, and exploring alternative LLM prompting templates would strengthen the paper for the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper reports on a system for generating humorous captions for a sequence of two or more images.  It claims to improve performance, as measured by several human and machine evaluations, by about 5%. The system also can be used for single images, with an improvement of about 4%.  The system relies on a custom-created database, as well as the ground truth provided for it by three human annotators.
strength_weakness: strength_weakness: What is new: a new combination of off-the-shelf methods: for finding subimages, for creating phrases for them, and for stitching the language results together according to a "humor grammar".  The contribution appears to be that of a particular synthesis, rather than any new insights or methods, as indicated by the marginal improvement in performance.  The database appears more interesting, given its size (18K memes) and four-part annotation (praise/mock x self/others).

The experimentation and evaluation are well done (Tables 4, 5), and believable examples are presented (Table 8, 9), perhaps even somewhat to excess.
limitations: limitations: The Abstract is generic and over-enthusiastic ("a new frontier"?!).  It needs some quantification about approach, datasets, competitors, and performance.  In fact, the reader is surprised to discover later in the paper how extensive the evaluations were, even if the results appear modest.

The Figure 1 multi-image has only *three* subimages, but the text talks about *four*.of them (on text line 54).

It never becomes clear why this particular system is a contribution: the three contributions listed at the end of the Introduction don't say anything unique to this paper: "novel methodology", "extensive experiments", "intricate associations"?  If there is anything that may be truly original, it might be the reward model, Equation 13, which gets its strength from the sequential ordering of captions based on human ground truth.

The ablation studies in Table 5 seem to suggest that attention is far more important than "chain of humor".  But this is not explained, and it seems to suggest that attention to detail is more important for humor than a global overview, somewhat contradicting one of the proposed virtues of the system.

The Conclusions don't really say anything and can be omitted.
paper_summary: paper_summary: This paper breaks down memes into multiple images for better image captioning and introduces XMECAP to combine reinforcement learning and supervised learning for encouraging similarity between text and images, similar to CLIP pre-training, but supervised. They also construct their own dataset containing 12,320 Chinese memes from social media.
strength_weakness: strength_weakness: 1. Breaking down memes into multiple images is necessary in this evolving digital landscape where memes are increasingly creative and may have multiple meanings.
2. The comparison with GPT-4 proves that XMECAP is at least up-to-par with SoTA models.
3. A wide variety of metrics are used to evaluate the generated captions. 
4. Table 8 and 9 prove how contemporary models may generate misleading or opposite captions. XMECAP has the upper hand here.
limitations: limitations: 1. Using LLMs (and VLMs) to decode memes is intuitive but very computationally expensive for real-world usage and content moderation. This framework could partially benefit from distillation.
2. Human evaluation may include more criteria: for example, sarcasm, irony, etc. which are particularly challenging for vision-language models (although it slightly coincides with creativity).
3. For LLM prompting, it would be good to experiment with other templates besides the Chain-of-Humour.
paper_summary: paper_summary: This paper presents a novel framework for meme caption generation that focuses on handling multi-image memes. The framework employs supervised fine-tuning and reinforcement learning with an innovative reward model considering both global and local similarities between visuals and text. The framework aims to address the unique challenges posed by multi-image memes, such as integrating composite information and maintaining consistency across shared captions. The authors constructed a new meme dataset, categorized by structure (single-image and multi-image) and emotion (self-praise, praise of others, self-mockery, and mockery of others), to train and evaluate the model. The experimental results demonstrate significant improvements in caption generation for both single and multi-image memes compared to existing benchmarks.
strength_weakness: strength_weakness: Theoretical Approach: The paper provides a robust theoretical foundation for the proposed framework. The use of adaptive transformation layers and attention-guided text generation is well-motivated, and the detailed explanation of the reward model enhances the understanding of the approach.

Comprehensive Dataset: The construction of a large-scale, well-balanced meme dataset, categorized by structure and emotion, is a notable contribution. This dataset enables thorough evaluation and facilitates future research in meme caption generation.

Evaluation and Results: Extensive experiments validate the superiority of the XMeCap framework over current benchmarks. The performance improvements in both single and multi-image meme caption generation are well-documented, with detailed comparisons to various baselines.

Clarity: This paper is well-written and easy-to-follow with detailed explanation and interpretable figures.
limitations: limitations: Language Limitation: The dataset and experiments are focused on Chinese memes, which might limit the generalizability of the results. While the authors use translation tools to mitigate this, the nuances of humor in different languages may not be fully captured.

Evaluation Metrics: The evaluation relies heavily on human judgment for metrics like informativeness, relevance, creativity, and humor. Although inter-rater agreement is calculated, the subjective nature of humor might introduce variability in the evaluation results.
paper_summary: paper_summary: This study has illuminated the intricate dynamics of humor in the multi-modal realm of memes, emphasizing on the impact of multi-images on meme captioning. The innovative XMECAP framework achieved state-of-the-art performance in meme caption generation as well as multi-modal humor detection.
strength_weakness: strength_weakness: 1. The XMECAP methodology features a novel reward model that integrates both global and local similarities between visuals and text through supervised fine-tuning and reinforcement learning for meme caption generation.

2. Superior Performance Validated: Extensive experiments have confirmed the superiority of the XMECAP approach over current benchmarks in both single and multi-image meme captioning, along with promising results in conventional multi-modal humor detection tasks.

3. The research underscores the ability of the proposed XMECAP to discern intricate associations between memes and their corresponding captions through visualization analysis, paving the way for future advanced meme-related research.
limitations: limitations: 1. In Table 1, the first row, I assume it includes data from all four categories, yet the max/min tokens are 24 and 6 respectively. The max tokens in captions of praise others in memes is 25, and the min tokens in captions of self-mockery in memes is 5. This seems to contradict my understanding.

2. Section 3.2 seems to be confusing and does not correspond with Figure 1. Specifically, as follows:
- The "shared projection layer" in the middle of Figure 1 is described at the beginning of Section 3.2 as two trainable linear layers, and I am not clear about the meaning of 'shared'.
- In line 268, "self-attention mechanism of the LLM," it appears to me to be just a regular self-attention mechanism, and I'm not clear on the significance of emphasizing LLM.
- The symbols for global attention and token-level attention seem to contradict Figure 1, which makes it difficult for me to understand.
- In the last paragraph, there are two instances of  $S_{I_i,j}$, and I suspect there might be a typographical error with one of them.

3. Lines 367 and 414 describe these parameters as trainable weights (with a singular/plural grammatical error in line 367), yet in lines 427 and 430, these parameters are set to specific numerical values, which is confusing to me. Assuming these parameters are indeed trainable weights, not constraining them could be fatal because the model might rely on these trainable weights to achieve a falsely minimized loss.

4. There is a lack of description for the baselines, and it is unclear whether all baselines have been fine-tuned on this dataset.

5. Ablation studies should differentiate between single-image and multi-image scenarios, as I am more interested in the impact of each component on the performance of multi-image settings.

6. For Section 4.5, more examples should be presented to comprehensively illustrate the role of the proposed method.

7. In line 453, "Detailed" should be in lowercase as it is not the beginning of a sentence. Additionally, the author refers to Appendix A, which I have not seen, and to my knowledge, ACM MM prohibits the use of appendices.