Can Language Models Laugh at YouTube Short-form Videos?
Authors: Dayoon Ko
Sangho Lee
Gunhee Kim
Venue: EMNLP 2023 Main (2023)
Keywords: video
humor
explanation
youtube
short-form videos
funny

Abstract: As short-form funny videos on social networks are gaining popularity, it becomes demanding for AI models to understand them for better communication with humans. Unfortunately, previous video humor datasets target specific domains such as speeches or sitcoms, and mostly focus on verbal cues.
We curate a user-generated dataset of 10K multimodal funny videos from YouTube, called ExFunTube. Using a video filtering pipeline with GPT-3.5, we verify both verbal and visual elements contributing to humor. After filtering, we annotate each video with timestamps and text explanations for funny moments. Our ExFunTube is unique over existing datasets in that our videos cover a wide range of domains with various types of humor that necessitate a multimodal understanding of the content.
Also, we develop a zero-shot video-to-text prompting to maximize video humor understanding of large language models (LLMs). With three different evaluation methods using automatic scores, rationale quality experiments, and human evaluations, we show that our prompting significantly improves LLMs' ability for humor explanation.

Can Language Models Laugh at YouTube Short-form Videos?
Dayoon Ko!
Sangho Lee²
Gunhee Kim
1 Seouul National University
2 Allen Institute for Artificial Intelligence
crrvoonoo@visioncco snu. snu. ac.
sanghol@allenai. org gunhee gunhee.kimdeee kimmksu. ac.kr
https://gtt thub. com/daaoon-ko/Exfuuue
Abstract
As short-form funny videos on social networks gaining popularity, it becomes demanding for Al models to understand them for better communication with humans. Unfortunately, previous video humor datasets target specific domains such as speeches or sitcoms, and mostly focus on verbal cues. We curate user- generated dataset of 1OK multimodal funny videos from YouTuub,, called ExFunTube, Us- ing video filtering pipeline with GPT-3.5, verify both verbal and visual elements contribut- ing to humor. After filtering, we annotate each video with timestamps and text explanations for funny moments. Our ExFunTubo is unique over existing datasets that our videos cover a wide range of domains with various types humor that necessitate multimodal under- standing of the content. Also, develop zero-shot video-to-text prompting maximize video humor understanding of large language models (LLMs). With three different evaluation methods using automatic scores, rationale qual- ity experiments, and human evaluations, we show that prompting significantly improves L.LMs' ability for humor explanation.
f Like dandetin
KY DANDELIO
you dis the dancelon
trying gisa fem flwer.
Timestamps Explanations of the funnyy moments
25 45
It's funny because the white dog shown dandalion and then the dag the dandelio unespre tedly.
funny the black white dog shown dondelion does dandalion
stmply give dog flower, not dog man's intentian
dog turns notioss danlelon from man's hand, It's funny becausse the men's exogger st reaction.
goes over
An example from the ExFunTube dataset. We curate funny short-form videos in various domains through filtering pipeline that verifies both verbal and visual elements contributing to humor. Each video is annotated with timestamps and explanations funny moments. In this example, three funny moments are identified.
Figure I:
retrieving information from the joke, cartoon, the perceiver's own knowledge. Since a sufficient understanding of the context required to perceive and further resolve the incongruity, understanding humor can be challenging. Nevertheless, if Al models can understand humor, they could interact more effectively with humans by providing empa- thetic responses based users' sense of humor. Furthermore, if the models understand short-form funny videos, they can recommend videos based on users' preferences or even generate witty titles based on vidco contexts.
1 Introduction
Today, huge number of short-form funny videos are popularly circulated social media platforms. Although humor often triggers instant laughter, un- derstanding humor a straightforward process. Numerous studies (Hazlitt, 1845; Kant, 1786; Ner- hardt, 1970; Jones, 1970; Shultz, 1972; Suls, 1972, 1983) have explored the cognitive process of hu- mor appreciation. For instance, Hazlitt (1845) and Kant (1786) propose the incongruity theory, assert- ing that incongruity provokes laughter. Nerhardt (1970) further develops the idea by defining the discrepancy between expectation and content, such as punchlines or cartoons. Suls (1972) suggests the incongruity-resolutio theory, positing that humor arises only when the incongruity is resolved by
Several studies (Hasan et al., 2019; Castro et al., 2019; Patro et al., 2021; Kumar et al., 2022) have collected humorous video datasets to investigate whether models understand if vidco is funny or not. However, the datasets have been gathered from limited domain, such as speeches or sitcoms. For example, Hasan et al. (2019) collect videos from TED, where there single speaker, and visual cues are restricted to gestures or facial ex-

---

#Data Points
Dataset
Modality
Type
Data Coutig
Exp
Task
EXPUN
T
Pun
2K
(Pun, Keywords, Up to 5 scores & explanations
Pun Exp
Abstratt Scene
(A funny image, An unfunay image, 10 funniness ratings)/
Imuge
AVH/ FOR
3K 15K
[A counterpart (object replaced) image)
Scoring & Altering
(Cartoon, Three fimalist captions. annocations locations. descriptions, uncanny descriptions, relevant eatities, and explanations) )
NYCC
LT
Cartoon
0.7K
Cartoon Caption Exp
MORB
LT
Posts
3K
(Image, Cuption, explunation)
Image Surcasm Exp
MUSIARD
V.A.T
Sitcom
6K
(Video, Binary (Lumay/unfueny) label)
Video Sarcasm BC
WITS
V.A.T
Sitcom
2.2K
[Video. One Explanation)
/ Dialogue Sarcasm Exp Video Humor BC
FUNNN
V.A.T
Spesch
8K
(Video, Binary (lunny)unfunny) lahel)
MHD
V.T
Sitcom
11K
(Vidco, Binary (fumuy/unfunny) label)
Vidco Humer BC
Shart-form
ExFunTube
V.A.T
Youtube vidcos
1OK
(Video, Up timestamps & explanations]
/ Video Humor Exp
Table I: Comparison of our ExFunTube with previous humor datasets: ExPUN (Sun et al., 2022), AVH&FOR (Chandrasekaran et al., 2016), NYCC (Hessel et al., 2022), MORE (Desai et al., 2022), MUSIARD (Castro et al., 2019). WITS (Kumar al., 2022), UR-FUNNY (Hasan et al., 2019), and MHD (Patro et al., 2021). In the Modality column, I, V, A, and T denote image, video, audio, and text, respectively. The #Data Points column shows only the number of positive (humorous) data points. The Data Config column specifies the composition of each data point. The Exp column indicates the presence of annotated explanations. In the Task column, Exp and BC are abbreviations of explanation generation and binary classification task each.
pressions. Castro al. (2019) build the MUSIARD dataset from four sitcoms, mainly from "Friends" and "Big Bang Theory," and Patro et al. (2021) col- lect the MHD dataset from the sitcom "Big Bang Theory." However, in sitcoms, the fixed actors fol- low a predetermined script on constructed set, and the punchline plays crucial role, the vi- sual elements may have less contribution to humor. Moreover, the aforementioned video datasets only have binary labels indicating whether the content is humorous or not. As binary classification may not evaluate whether a model truly understands the humor in a video, Kumar et al. (2022) collect WITS with annotated text explanations. However, this dataset is limited to sarcasm, a specific form of humor, and focuses sarcasm explanation in dialogue. It highlights need for humor expla- nation dataset that considers visual elements more and covers general humor.
plaining humor present in text to some extent (Chowdhery et al., 2022). Inspired by the recent research on multimodal-informee prompting (Zeng et al., 2022), we convert video content into text, leveraging various zero-shot models on diverse modalities of the video. We provide LLMs with the text prompt linguistic summary of video content. Specifically, we consider two modalities of the video content: visual and audio. From the visual modality, we obtain dense video descrip- tions. From the audio modality, we acquire speech transcripts and sound labels. Finally, we chrono- logically integrate them into a text prompt that can maximize LLMs' ability for humor explanation. Since evaluating model's ability to explain hu- mor is challenging, report our results in three different ways: model-based automatic scores, ra- tionale quality metrics with the moment localization task, and human evaluation. First, we report model- based metrics instcad of those using word overlap. Second, we conduct a rationale quality experiment, which assesses the quality of explanations from the accuracy of predicting gold labels (Wiegreffe et al., 2021). Finally, we carry out human evaluations with sampled test examples. Through these three different results, our prompting approach consider- ably improves the humor explanation performance of three important LLMs, including one zero-shot GPT-3.5 and two finetuned T5 (Raffel et al., 2020) and BART (Lewis et al., 2020).
To this end, we curate ExFunTube, a dataset of funny, short-form videos with explanations. These videos are collected from user-generated YouTube videos, which are shared on the "r/youtubehaiku" subreddit. In this subreddit, users upload short- form funny videos, typically up to 30 seconds long. We develop a video filtering pipeline with GPT-3.5 (Ouyang et al., 2022), designed to exclude the videos with minimal visual impact on humor. Then, we annotate the collected videos with timestamps and text explanations of funny moments, exemplified in Figure 1.
To summarize, our key contributions are:
Recent LLMs show great performance for ex-
We curate ExFunTube, a dataset consisting

---

of 10,136 user-generated, funny short-form vidcos. Each video is annotated with times- tamps and explanations of funny moments. As compared in Table 1, our ExFunTube is unique over existing datasets that our videos cover a wide range of domains with various types of humor that necessitate multimodal understanding of the content.
et al. (2021). As word-overlap scores may be insuf- ficient for the evaluation of explanation, Wiegreffe et al. (2021) propose a rationale quality metric that calculates the difference of prediction scores for gold labels when rationales are provided or not: Ace (IR - 0) Acc (I - O), where I, R, and O denote input, rationale, and gold label, respectively, In addition, Sun al. (2022) evaluate explanations by comparing the accuracy of joke classification with and without explanations: Ace (IE O) Acc (I - O) where E. denotes explanation. We introduce a moment localization task to compute the rationale quality score of the video explanation.
2. We design zero-shot video-to-text prompting that converts video content into text to maxi- mize LLMs' ability to explain video humor.
3. With three different evaluation methods of model-based lexical scores, rationale quality scores, and human evaluations, we verify that our prompting improves LLMs' performance on humor explanation.
Modular Vision-Language Learning. As pre- trained models become larger and are trained with extensive datasets, various multimodal comprehen- sion tasks have been tackled by composing these pretrained models. One approach is to transform visual information into discrete text words (Zeng et al., 2022; Yang et al., 2022: Wang al., 2022b). Zeng et al. (2022) propose a modular framework that leverages L.LM to construct the input text for the sub- sequent model based on the output of multimodal models in the previous stage. They demonstrate performance improvements in image captioning and visual question answering (VQA) tasks. An- other approach connects pretrained models through continuous feature embeddings (Patro et al., 2021; Alayrac et al., 2022; Tiong al., 2022). Li et al. (2023a) pretrain additional lightweight modules that bridge the frozen image encoder and LLMs to eliminate the modality gap between the two frozen pretrained models. Tewel et al. (2022) connect the frozen image encoder with the frozen language decoder and evolve additional pseudo tokens during inference time to perform the video captioning task. Recently, there have been efforts integrate these two different approaches. Li et (2023b) intro- duce VideoChat, chat-centric video understanding system consisting of two modules: VideoChat- and VideoChat-Embed. The former generates text descriptions from the video and the latter encodes the video as embeddings. These text descriptions and embeddings are combined with received ques- tion to form a prompt, based which the LLM generates response.
2 Related work
Humor Understanding. It has been long- standing question whether AI models under- stand humor in text, images, videos. Early studies focused on classifying whether text (Annamorad- nejad and Zoghi, 2020), images (Chandrasekaran et al., 2016), or videos (Hasan et al., 2019; Castro et al., 2019; Patro et al., 2021) are humorous or not. Some studies, such as Chandrasekaran et al. (2016). also rate the degree to which abstract scenes are perceived humorous. However, binary classi- fications or ratings do not fully evaluate whether a model understands humor in detail. Recent hu- mor studies have shifted towards having models explain humor. Sun et al. (2022) augment the SemEval 2017 Task 7 (Miller et al., 2017) with funniness ratings and explanations. Hessel et al. (2022) augment the New Yorker cartoon captions with explanations. Desai et al. (2022) propose dataset of explanations for sarcastic captions, and Kumar et al. (2022) collect sarcastic videos from sitcom with explanations.
Natural Language Explanation. As tasks of interest become increasingly complex, predicting la- bels may not be enough to evaluate the models' true understanding. Thus, some works make models explain their decisions an alternative. For in- stance, FLUTE (Chakrabarty et al., 2022) augments e-SNLI (Camburu al., 2018) to curate figurative texts with labels for natural language inference (NLJ) tasks and evaluate model-generate explana- tions. To evaluate model explanations, they utilize a rationale quality metric suggested by Wiegreffe
In our work, we combine vision-language pre- trained models with LLMs through text for two uses: (i) video filtering for collecting multimodal funny videos and (ii) video-to-texi generation to provide LLMs with a prompt of video content.

---

3 The ExFunTube Dataset
(a) Generate a video caption (V) & transcript (T)
Captioning
The ExFunTube dataset comprises 10,136 videos, each annotated with timestamps of funny moments and corresponding explanations describing why each moment is humorous. The purpose of this dataset is to evaluate the models' ability to ex- plain why a given video is funny as measure of understanding video humor.
Speech-to-Terl
Exclude if funny utterance w/
In this task, will sne description and transcript of video, funny utterance the given transeript. there is no funny utterance, answer 'No."
Description:
Transcript:
Funny utterance:
GPT-3.5
(c) Include if no funny utterance w/o V
3.1 Video Collection and Filtering
In this task, you will see a transcript of a video. Find furny utterance in the given transcript. If there funny utterance. answer 'No.'
We initially crawl all 220K videos shared on the subreddit "r/youtubchaiku," "1 where people share humorous short-form YouTube videos lasting up 30 seconds. To ensure multimodal humor in videos, we design a four-step filtering pipeline that selects videos with both visual and verbal elements contributing to humor, as shown in Figure 2.
Transcript:
Funny utterance:
GPT-3.5
(d) Include if two explanations are dissimilar
In this task, will descri ption and a transcrint of video Explain why the video funny Description: Transcrip: Explanation in one sentence:
In this task, will trans cript of video. Explain why the video funny.
Transcript:
Video Caption and Transcript. In the first step (Figure 2 (a)), we obtain transcript and video caption to describe the verbal and visual elements of a video clip, respectively. We extract a video caption using a zero-shot video captioning model (Tewel et al., 2022). Since our dataset contains diverse videos such as animations and edited videos not present in previous video datasets, we choose a model that utilizes both CLIP (Radford et al., 2021) and GPT-2 (Radford et al., 2019), which are pretrained on huge Web-sourced data. We transcribe audio from the video clip using speech- to-text model Whisper (Radford et al., 2022). We remove vidcos with no speech or in languages other than English.
Explanation in one seriencee
Explanation w/ V
GPT-3.5
Explanation w
SentBERT
Score
0.8
0.8
Figure 2: The video filtering pipeline selects multimodal funny videos. Red boxes display the actual prompts provided to GPT-3.5. See the details in $ 3.1. (a) We generate transcript and caption from the input video. (b) Via GPT-3.5 prompting, we filter out the video that not funny from the transcript and caption. (c) The video is accepted if it is funny from both the transcript and caption but not from the transcript only, since its humor is multimodal. (d) GPT-3.5 generates humor explanations with or without the video caption. We remove the videos if they are too similar since their humor not multimodal. Examples for each are presented in the Appendix.
Multimodal Humor. Our goal is to collect the videos that are funny from both verbal and visual elements, instead of funny from only one modality. Thus, as shown in Figure (b), we first verify that the video is verbally funny; we do this by whether GPT-3.5 can find a funny utterance given a pair of the video caption and the transcript. If GPT-3.5 detects no funny utterances, we filter out the video. Next, shown in Figure 2 (c), again prompt GPT-3.5 to find funny utterance with only transcript (i.e., no video caption). If no funny utterance detected, then we accept this video. The rationale that the humor of this video is multimodal; the visual caption is required to identify the fun in the video. Otherwise, if GPT-3.5 can find a funny utterance in this case, we perform a further inspection as follows.
Difference in Explanations. In the last step (Figure 2 (d)), GPT-3.5 is prompted to generate explanations in one sentence for the two cases: when given both a video caption and transcript and when given only a transcript. We then measure the similarity between the two explanations using the SentBERT score (Reimers and Gurevych, 2019), which embeds each sentence and calculates the cosine similarity of their embeddings. The reason for adopting the SemtBERT score is that it can reflect the semantics of the entire sentence. If the score is higher than the threshold, we exclude the video since the video caption does not contribute to the humor explanation. Otherwise, the video is accepted.

---

Rationale of Our Pipeline. There has yet to be a method to gauge the extent and manner in which visual elements contribute to humor. In other benchmarks, the multimodality of datasets has been validated by analyzing the performance gap when visual information is either provided or not (Hasan et al., 2019; Patro et al., 2021; Kumar et al., 2022). Similarly, we collect videos that exhibit differences in the assigned task (i.e., iden- tifying humorous utterances by GPT-3.5) with or without visual information. In the field of NLI, previous works (Liu et al., 2022; Wiegreffe et al., 2022; Chakrabarty et al., 2022) leverage the power of LLMS such as GPT-3 (Brown et al., 2020) in cre- ating figurative language examples or explanations for them. Likewise, we use GPT-3.5 to check the difference between generated explanations. To the best of our knowledge, this is the first approach that employs explanations for curating dataset. Thanks to the pipeline, we can collect 21K high-quality multimodal humorous videos.
Out of 219 workers, only 60 pass the qualification test, indicating our thorough selection. For cach video, we instruct one worker first to identify up to three funny moments within a video (up to 30 seconds long) and then annotate why each moment is funny. To make workers explain both humor elements and justifications, we provide recommended format: "[What is funny]. It is funny because [Why funny]". We only accept responses including both descriptions (What) and justifications (Why) and reject those that lack either. Given the difficulty of the task, we offer detailed feedback to the workers, helping them improve their performance with high annotation standard.
As a result, we obtain 1 11,166 explanations, each paired with start and end timestamps of the moment. They consist of 44.3 words on average. Out of 10, videos, 9,222 contain one funny moment, 798 contain two, and 116 contain three. Most videos contain a single funny moment since videos are typically shorter than 30 seconds. However, given the varied content in each video, there can be any number of funny moments.
Postprocessing. To ensure that our dataset does not contain any disrespectful or harmful content towards individuals or animals, we conduct a thor- ough manual review of all 21K videos. We filter out the videos using the five criteria based on the safety objectives outlined by Thoppilan et al. (2022): (i) Discrimination: videos displaying discrimination based race, gender, sexual orientation, age, or disability. (ii) Animal crueity: videos depicting acts of animal cruelty, such as cat falling. (iii) Dangerous goods, services, activities, or self-harm: videos featuring dangerous content like drugs, vio- lence, or bullying. (iv) Obscenities or profanities: videos containing explicit language or sexual ac- tions. (v) Shocking content: videos that include shocking content, such as gunshots explosions, After the filtering about 50% of the videos are removed, and we are left with 10,136 videos.
Approach
We explore an approach to explain video humor. Our idea is first to convert the video content into fine-grained text and then take advantage of recent powerful LLMs in zero-shot manner. We design to extract as much information from videos into text as possible. Figure 3 shows a zero-shot video- to-text prompting that converts the vidco content into a text input to LLMs.
4.1 Fine-grained Text Prompts
Videos contain visual and audio modalities. The audio is further split into speech and sound. For each component, we initially generate text descrip- tions using state-of-the-un zero-shot models. Then, we arrange text descriptions in chronological order and use them as a prompt.
3.2 Data annotations
Visual. In order to populate high-qualit text descriptions about the visual, we first (i) segment the video, (ii) generate multiple frame captions, and (iii) retrieve the best-matching caption with the video-to-text model.
We crowdsource via Amazon Mechanical Turk (AMT) to annotate start and end timestamps of funny moments and provide text explanations for each moment. To participate in our dataset anno- tation, workers must meet the following criteria: HIT approval rate of 99% or higher, total of more than 10,000 approved HITs, and be located in one of the countries of AU, CA, GB, NZ, or US. We conduct a qualification test for these workers, selecting those who can effectively explain humor.
explaamte
First, we employ PySceneDetect? to divide a video into a set of N segments based on visual changes. During the filtering pipeline ($3.1). the speech-to-text model model generates timestamps

---

Prompt
BLIP-2 Captioning
Oaptoes Opplnee
Please gene ate esp lepaatotion of why video is funny, given vibus descrit. and dakigu for monokg and an audoo intion for the video, Explain be
Segmert
the video. Visual descriptions
chronologcal
Video
Video-to-
Caption
Visual descriptions and utterances cchronologicallyy::
Parting,
BLIP-2 Imagi Captioning
Cupton Options
Speakkr I: "Hey Luke, sit. like, dandelion АНHННН
white dog with blue being some kind flower.
Segmanti
InternVideo Video-to- Tort Retneval
Scone:
Speaker look what you dandelion"
NP Caption
person foods a flower to white husky.
Whisper Speech-to- Text
ChatGPT Speaker Diarization
paw.
Audio
)
Scene:
Spealier 1: "AYYE MY DANDELIONN"
EfficientAT Audio lagging
Sound Tags
person holding out thumb to husky.
Explanation(Up to sentences):
oz gran-g
Configuration
Figure 3: (a) A zero-shot video-to-text prompting for converting video content into fine-grained text ($ 4.1). For the visual modality, the video is first divided into segments, for each of which many possible captions are generated, and the best one chosen finally. For audio modality, a transcript with speaker separation and sound tags are obtained. (b) The fine-grained text is configured an input prompt to LLMs ($ 4.2).
for each utterance. We also use them to split the segments further, resulting in more fine-graid mee-graieed and semantically meaningful video segments.
4.2 Prompt Configuration and LLMs
After extracting text from visual, speech, and sound, we configure the prompt like an example of Figure 3. The prompt starts with predefined text "Please generate to instruct LLMs to explain as they are watching the video. We then include sound tags enclosed in parentheses and arrange the extracted text of speech and visuals for each video segment chronologically. To distinguish between video seg- ments, we begin each segment with "Scene: Finally, we ask LLMs to generate an explanation of up to three sentences.
Next, we extract frames at rate of 5fps from each of the N video segments. We generate (== 20) cap- tions per frame using the image captioning model BLIP-2 (Li et al., 2023a) with a "Who is doing what?" prompt, which enhance action detection. We then have frame caption corpus (# Frames K captions) per segment. Subsequently, we use the video-to-text model Intern Video (Wang al., 2022a) to retrieve the caption that best matches each video segment from the respective frame corpus. Finally, we obtain one caption per segment, result- ing in a total of N captions, which are fine- ine-grained descriptions of the visual component.
LLMs. Although any LLMs can be adopted, we use three different ones: finetuned T5 (Raffel et al., 2020) and BART (Lewis et al., 2020), and zero-shot GPT-3.5 text-davinci-003.
Speech. We transcribe audio with Whisper (Rad- ford et al., 2022) as done in our video filtering pipeline. We then predict the number of speak- ers and assign speakers to each utterance utilizing ChatGPT (OpenAI, 2023). This speaker separation helps deep understanding of dialogue.
5 Experiments
We experiment with different models to see how well they explain the humor in the ExFunTube videos. We evaluate the models in three different ways of model-based automatic scores, rationale quality experiments, and human evaluation.
Sound. We extract sound tags to provide more context. We use an audio tagging model (Schmid et al., 2022) to classify the entire audio stream. We select the top predicted tags that have higher confidence value than the threshold (0.3). We concatenate the tags and insert them at the beginning of the prompt. This can provide the model with an overall atmosphere of the video.
5.1 Experimental Setup
Baselines. We evaluate four types of explanation models. (i) Text-only LLMs generate explanations when only a transcript is provided (i.e., no use of visual). We use T5 Large and BART Large with finetuning and GPT-3.5 as zero-shot model. (ii)

---

Automatic Score
Rationalc Quality Score (})
Human Evaluation (
SentBERT (T)
ROSCOE (RA) (T)
@0.7
@00.5
@0.4
Mean
@0.88
@0.7
Mean
0..3
Rating
T5
0.154
0.355
0.585
0.795
0.534
0.406
0.871
0.780
10.3
21.9
Text-Only
BART
0.169
0.388
0.617
0.807
0.545
0.440
0.875
0.785
13.7
30.1
0,178
GPT-3.5
0.149
0.310
0,556
0.774
0.529
0.371
0.841
0.772
18.8
22.5
0.385
MAP
0.149
0.375
0.604
0.809
0.541
0.438
0.880
0.785
13.1
25.3
0.131
VideoChat-Text
GPT-3.5
0.115
0.345
0.618
0.839
0.539
0.414
0.900
0.783
13.9
26.5
T5
0.230
0.483
0.719
0.887
0.584
0.543
0.932
0.804
2.9
12.5
Prompting
BART
0.238
0.500
0,730
0.886
0.588
0.554
0.935
0.805
23.9
0.282
GPT-3.5
0.214
0.541
0.806
0.945
0.602
0.639
0.971
0.817
5.5
9.3
0.523
Gold
0.792
Table 2: Humor explanation results in terms of automatic scores (SentBERT and ROSCOE), rationale quality scores, and human rating. In the automatic scores, @ shows the proportion of test explanations of which scores are higher than K, and the mean column the average score of each metric. For rationale quality scores with funny moment localization, adopt two IoU thresholds, 0.3 and 0.5; lower scores are better. For human rating, five workers rate each of 100 randomly selected test videos from No (0), Weak No (0.25), Neutral (0.5), Weak Yes (0.75), to Yes (I). After excluding the highest and lowest scores, the remaining scores are averaged.
MAF (Kumar et al., 2022) is a multimodal end-to- end model designed for video sarcasm explanation. It generates explanations by receiving features of the three components (visual, specch, and audio). We train the model on our dataset. (iii) VideoChat- Text (Li et al., 2023b) multimodal prompting framework that textualizes video information into text, including video/clip captions, objects con- tained in the video and a transcript. Given the prompt, GPT-3.5 generates explanations in zero- shot manner. (iv) LLMs with our prompting generate explanations given prompt created by zero-shot video-to-text prompting, using the same LLMs (i) T5, BART, and GPT-3.5. Note that T5 and BART models are finetuned gener- ate explanations given generated prompts, while GPT-3.5 generates in a zero-shot manner.
5.2 Results of Model-based Automatic Scores
Since the metrics based word overlaps may fail reflect faithfulness and plausibility as highlighted by Sun al. (2022), we evaluate explanations using two model-based scores: SentBERT Score and ROSCOE (Golovneva et al., 2022). ROSCOE a suite of metrics designed to evaluate the reasoning process within a chain-of-thouge prompting (Wei et al., 2022). It is suitable for our explanation tasks since goal is to uncover the reason for laughter (ie.,, why is the video humorous?) Among the various scores provided by ROSCOE, we use the reasoning alignment (RA) score, which computes the contextual similarity between the hypothesis and reasoning.
Table reports the model-based automatic scores of different methods. We show not only the mean metric values but also the proportions of the test set with scores higher than various thresholds; @ K represents the proportion of data points with scores equal to or greater than K.
Explanation Generation. For all finetuned models on our dataset, we employ K-fold cross- validation as follows. We divide the entire dataset of 10,136 videos into five equal-sized subsets. In each iteration, we train the model on three subsets, use subset for validation, and test on the remaining subset. We repeat this process five times, rotating the test subset in each iteration. Finally, obtain predicted explanations for the entire set.
The results show that, except for SentBERT @0.7, GPT-3.5 with our prompting reaches the best per- formance. Especially, the SentBERT and ROSCOE scores with our prompting are higher than those with text-only baselines in all cases. In addition, our method outperforms the multimodal end-to- end baseline MAF and the multimodal zero-shot prompting baseline VideoChat-Text.. The compar- ison of @K metries shows even more significant differences, particularly for SentBERT @0.5 and
Evaluation. To compare the predicted explana- tion with the gold explanation for each video, we concatenate explanations for each moment into a single, unified explanation. For more details on experiments, please refer to the Appendix.

---

ROSCOE @0.8, where the performance margin ranges from 0.1 (BART) to 0.27 (GPT-3.5) com-
100
80
60
pared to the text-only baselines. This means that using transcripts alone may not be sufficient to understand the humor in our videos.
40
20
GPT-
Figure 4: Results of human preference: comparing GPT- 3.5 with our prompting to text-only GPT-3.5, MAF, and Gold, respectively.
5.3 Results of Rationale Quality Scores
We conduct rationale quality experiment following Wiegreffe et al. (2021) and Sun al. (2022). Since our dataset consists of videos, unlike theirs, we adapt the experimentation scheme by evaluating the rationale quality through a moment localization task, which aims at predicting funny moments defined by their start and end timestamps in video given the text explanation.
5.4 Results of Human Evaluations
For human evaluation, we employ 10 AMT workers using the same criteria as in the dataset annotation but excluding the ones who already participated in the annotation. We randomly select 100 vidcos and evaluate explanations generated by all models ex- cept baselines using T5 and VideoChat- which show worse automatic scores than other text-only or multimodal basclines. We obtain human evalua- tions with two methods: rating and comparison.
We use QD-DETR (Moon et al., 2023) as localizer and divide the entire dataset into 8:1:1 splits for training (8, 10), validation (1,013), and testing (1,013). During the training, the localizer is learned to predict the gold timestamp given gold explanation.
inference, we compute the rationale quality as the prediction difference of the localizer between when given a explanation and when given gold explanation.
For the rating, workers are asked to rate each explanation according to No (0), Weak No (0.25), Neutral (0.5), Weak Yes (0.75), and Yes (1) and check any shortcomings. We ask five workers for each explanation, exclude the highest and lowest scores, and take the average. For the comparison, workers compare GPT-3.5 with our prompting (1) Text-only GPT-3.5, (2) MAF, and (3) Gold explanations and choose the better explanation. We ask five workers for each pair of comparisons.
Let M be moner-geeerated explanation, G be a gold explanation, and r be a threshold. For each test data point, we calculate the maximum IoU from the top 5 candidates given M or G, respectively denoted as IoUM or loUG. We use the top 5 since there be at most three funny moments in single video and the localization predictions can overlap with each other. We compute the difference when loUM M т. The final score S is the sum of differences for all test data:
The rating results are presented on the far right of Table 2. The scores of BART and GPT-3.5 increase by about 0.1 when our prompting is in- cluded. The comparison results are presented in Figure 4. The number of votes for text-only GPT- 3.5 significantly lower than that of GPT-3.5 with our prompting, indicating that visual information valuable, and our prompting helps convey visual information effectively. In both rating and com- parison, MAF shows lower performance than the text-only models despite being multimodal model. This suggests that providing visual information text to LLMs could be more effective than train- ing the multimodal model end-to-end. Moreover, GPT-3.5 with our prompting, which shows the best results, still scores lower than Gold, indicating that understanding and explaining the humor in our dataset still remains unsolved.
S Σ (lOUG, - IoUM,) . 1(loUM, > T),
where n the number of test data points, and 1(.) is the indicator function.
Table 2 shows the results when the IoU threshold T is set to 0.3 and 0.5. A lower score is better as it is closer to the gold standard. In each LLM,, the performance improves when our prompting is included compared to corresponding text-only ones. In particular, our approach improves GPT-3.5 the most, with the threshold at 0.3 resulting in score gap of 13.3, and 0.5, a score gap of 13.2. Again, the performance all LLMs with our prompting is better than MAF and TieeoChat-Textt.

---

0.65
BART CPT-335
VidesChat-Text WP
GPT-3.5
Performence
0.55
0.45
Sarpite
Abourdiy
Inery
Pan
Wor
Clowns
Desopan
Obanmatoona
Mitundesstanii
Safa
Shunlooa
Parsonifictio
Mopromunc
Emuaprati
Sorcoon
Joko
Visua
salf solf-dooroooceeeee
Toconmy
Dark
Figure 5: Explanation performance according to humor taxonomy. We categorize all videos into 20 humor classes and compare the performance of cight different baselines terms of the SentBERT score. The humor taxonomy is arranged in descending order of proportion in our dataset.
5.5 Analyzing LLMs with Humor Taxonomy
GPT-3.5 w/ Prompting
We classify our dataset into total of 20 humor categories referring to Martin and Ford (2018) and Buijzen and Valkenburg (2004), and observe the performance of baselines by the humor taxonomy. We provide ChatGPT with 20 categorics along with brief description and one example (i.e., one- shot learning) and instruct ChatGPT to classify the video based the given explanation. Thanks to ChatGPT's powerful in-context learning capability, we effectively classify 10,136 videos based on their corresponding explanations.
w/o
w/o
wlo A
w/ V.
SemtBRRT
0.512
0.497
ROSCOE (RA)
0.778
0.763
0.801
0.817
Table 3: Ablation results of 3.5 with our prompting measured by SentBRRT and ROSCOE scores when each modality component is removed. T,, and denote visual, speech, and sound, respectively.
which suggests that all three components are cru- cial for understanding and explaining humorous videos in our dataset. Additional ablation studies are presented in the Appendix.
Figure 5 shows the models' performance by hu- mor categories. Excluding the Jokes and Self- deprecating classes, the performance increases with our prompting in all categories. In particular, the performance significantly increases in Clownish humor, Visual gags, and Slapsticks, which heavily reflect visual elements. This indicates that our zero- shot video-to-text prompting effectively conveys visual elements to the LLM.
Conclusion
We introduced ExFunTube, dataset consisting of 10,136 user-generated videos annotated with timestamps and explanations of funny moments. Our dataset aims to assess how well AI models understand and explain video humor. We devised a zero-shot video-to-tex prompting to make existing LLMs better explain the video content. With three different evaluation methods, we demonstrated that the humor in our dataset multimodal, and our prompting maximized LLMs' ability to generate explanations.
5.6 Ablation Study
We compare the importance of each modality in humor explanation. Table 3 presents the results of SentBERT and ROSCOE scores when visual, speech, and sound components are not included in the prompt one by one. In GPT-3.5 with our prompt- ing, the performance without the visual component drops much as when the speech is removed, indi- cating that the visual component plays an important role in our dataset. Moreover, the performance de- creases when either of the components is removed,
However, the performance still falls short of human levels, our dataset remains sufficiently chal- lenging and calls for future research. Furthermore, we can consider the training of the model using user feedback for personalized humor understanding.

---

Limitations
for quality enhancement of casual videos and their 3D metaverse transformation), and Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govern- ment (MSIT) (No. 2021-0-01344 Artificial Intel- ligence Graduate School Program (Scoul National University)). Gunhee Kim is the corresponding author.
Since the copyright remains with the original own- ers of our dataset videos, will only distribute URLs instead of videos.
Our method relies on the performance of existing state-of-the-ard models, as we used them in a zero- shot composition. Also, our approach composes models through text, so it could also be explorable to use adaptor-based method for prompt tuning during inference.
References
We measured the videos by dividing them into three modalities, but we did not consider the tem- poral information of sound. As timing can play a role in humor, analyzing the sound in accordance with the timeline could be helpful.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems, volume 35.
Lastly, humor is subjective, which means that our collected explanations may be subjective, too.
Issa Annamoradnejad and Gohar Zoghi. 2020. Col- BERT: Using BERT Sentence Embedding for Humor Detection. arXiv preprint arXiv:2004. 12765, 1(3).
Ethics Statement
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models Are Few-Shot Learners. Advances in neural information processing systems, 33:1877-1901.
We put much effort into ensuring that our dataset contains no inappropriate videos that may raise eth- ical issues. Based on the safety rules of Thoppilan al. (2022), authors manually viewed each video entirely from start to end and filtered the video if there was any content that corresponded to the filtering criteria presented in the dataset postpro- cessing. Although we carefully reviewed all the videos, there could still be some videos that are not comfortable for someone. If such inappropriate videos are found, will remove them in the future. Also, since we only recruit workers in AU, CA. GB, NZ, and US as mentioned in the Appendix, the cultural and geographic biases may influence humor explanations.
Moniek Buijzen and Patti M Valkenburg. 2004. Devel- oping a Typology of Humor in Audiovisual Media. Media psychology, 6(2):177-167.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018 e-SNLI: Natural Language Inference with Natural Language Explanations. In Advances in Neural Information Processing Systems, volume 31.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez- Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards Multimodal Sarcasm Detection (An Obviiousl__ Perfect Paper). In Pro- ceedings of the 57th Annual Meeting of the Associa- tion for Computational Linguistics.
Acknowledgments
We sincerely thank Jaekyeom Kim, Jaewoo Ahn, Soochan Lee, Wonkwang Lee. Yeda Song, and Jaehyeon Son for their valuable comments. We would also like to thank AMT workers for their commitment to building the ExFun dataset. This work was supported by the SNU-Global Excel- Ience Research Center establishment project, Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00274280) Institute of Information & communications Technol-
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. Flute: Figurative Lan- guage Understanding through Textual Explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Arjun Chandrasekaran, Ashwin K Vijayakumar, Stanis- law Antol, Mohit Bansal, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2016. We Are Humor Be- ings: Understanding and Predicting Visual Humor. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi,
ogy Planning& & Evaluation (ITTP) grant funded by
the Korea government (MSIT) (No. 2022-0-00156, Fundamental research on continual meta-learning

---

Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyzz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellstee,,,,,m,, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways.
Immanuel Kant. 1786. Kritik der Urteilskraft und Schriften aur Naturphiosophie, volume 5. Insel- Verlag.
Shivami Kumar, Atharva Kulkarni, Md Shad Akhtar, and Tanmoy Chakraborty. 2022. When did you become so smart, oh wise one?! Sarcasm Explanation Multi-modal Multi-party Dialogues. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers).
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sesguence-to-Squennee Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. pages 787 1-780.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. BLIP-2: Bootstrapping maaguugge-maage Pre-training with Frozen Image Encoders and Large Language Models. arXiv preprint arXiv:2301. 12597.
Poorav Desai, Tanmoy Chakraborty, and Md Shad Akhtar. 2022. Nice Perfume. How Long Did You Marinate in It? Multimodal Sarcasm Explanation. In Proceedings of the AAAI Conference on Artificial Intelligence.
KunChang Li. Yinan He, Yi Wang. Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. 2023b. Videochat: Chat-centric video un- derstanding.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computa- tional Linguistics.
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. WANLI: Worker and Ai Collabora- tion for Natural Language Inference Dataset Creation. arXiv preprint arXiv:2201.052 .0595..
Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Con- ference on Learning Representations.
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam ZarandZa and Asli Celikyilmaz. 2022. ROSCOE: Suite of Metries for Scoring Step-by-Ste Reasoning, arXiv preprint 222.0922....7919.
Rod A Martin and Thomas Ford. 2018. The Psychology of Humor: An Integrative Approach. Academic press.
Tristan Miller, Christian F Hempelmann, and Tryna Gurevych. 2017. Semeval-2017 Task 7: Detection and Interpretation of English Puns. In Proceedings of the I 11th International Workshop on Semantic Eval- uation (SemEval- 2017).
Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Ifiekhar Tanveer, Louis- Philippe Morency, and Mohammed Ehsan Hoque. 2019. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing DEMNLP-JCCNLP
WonJun Moon, Sangeek Hyun, SangUk Park, Dongchan Park, and Jae- Heo. 2023. Query-Dependent Video Representation for Moment Retrieval and Highlight Detection. arXiv preprint arXiv:2303.I 13874.
William Hazlitt. 1845. Lectures on the English Comic Writers. 28. Wiley and Putnam.
Göran Nerhardt. 1970. Humor and Inclination to Laugh: Emotional Reactions to Stimuli of Different Diver- genee from Range of Expectancy. Scandinavian Journal of Psychology, 11())185-1155
Jack Hessel, Ana Marasović, Jena D Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2022. Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest. arXiv preprint arXiv:2209.06293.
OpenAL. 2023. ChatGPT. Generated with GPT-3 tech- nology.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-277444
James McCoy Jones. 1970. Cognitive Factors in the Ap- preciation of Humor: A Theoretical and Experimental Analysis. Yale University.

---

Badri N Patro, Mayank Lunayach. Deepankar Srivastava, Hunar Singh, Vinay P Namboodiri, et al. 2021. Mul- timodal Humor Dataset: Predicting Laughter Tracks for Sitcoms. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng. Alicia Taylor Bos, Leslie Baker, Yu Du, et 2022. LaMDA: Language Models for Dialog Applications. arXiv preprint arXiv:2201.022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry; Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning Transferable Visual Models from Natural Language Supervision. In International conference on machine learning. PMLR.
Anthony Meng Huat Tiong. Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. 2022. Plug- and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training. arXiv preprint arXiv:2210.08773
Yi Wang, Kunchang Li. Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao. Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. et al. 2022a. Intern Video: General Video Foundation Models via Generative and Discrimina- tive Learning. arXiv preprint 222.02221..01111
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-scale Weak Supervision. arXiv preprint arXiv:2212.04356
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language Models are Unsupervised Multitask Learners. Ope- nAI blog, 1(8):9.
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang. Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng 2022b. Language Models with Image Descriptors are Strong Few-Shot Video- Language Learners. In Advances in Neural Informa- fion Processing Systems, volume 35.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the Lim- its of Transfer Learning with Unified Texx- Transformer. Journal of Machine Learning Research, 21:1-67.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, 2022. Cfain-of-Thought Prompting Elicits Rea- soning in Large Language Models. arXiv preprint arXiv:220..11903
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence Embeddings using Siamese BERT- Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNPP).
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta. Mark Riedl, and Yejin Choi. 2022. Reframing Human- Collaboration for Generating Free-Text Explana- tions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 632-658.
Florian Schmid, Khaled Koutini, and Gerhard Wid- mer. 2022. Efficient Large-scale Audio Tagging via Transformer-to CN-CNN Knowledge Distillation. arXiv preprint 221.2221..0477.
Sarah Wiegreffe, Ana Marasović, and Noah A Smith. 2021. Measuring Association Between Labels and Free-Text Rationales. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing.
Thomas R Shultz. 1972. The role of incongruity and resolution in children's appreciation of cartoon humor. Journal of Experimental Child Psychology, 13())456- 477.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An Empirical Study of GPT-3 for Few-Shot Knowledge- Based VQA. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3081-3089.
Jerry Suls. 1983. Cognitive Processes in Humor Appre- ciation. Handbook of Humor Research: Volume 1: Basic Issues, pages 39-57.
Jerry M Suls. 1972. A Two-Stage Model for the Ap- preciation of Jokes and Cartoons: An Information- Processing Analysis. The psychology of humor: The- oretical perspectives and empirical issues, 1:81-100.
Andy Zeng, Maria Attarian, Brian lchter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sind- hwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. 2022. Socratic Models: Composing Zero- Shot Multimodal Reasoning with Language. arXiv.
Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Alessan- dra Cervone, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022. ExPUNations: Augment- ing Puns with Keywords and Explanations. arXiv preprint 22 135..3
Yoad Tewel, Yoav Shalev, Roy Nadler, Idan Schwartz, and Lior Wolf. 2022. Zero-Shot Video Caption- ing with Evolving Pseudo-Tokens, arXiv preprint arXiv:2207.11100

---

A Experimental Details
SemtBRRT ROSCOE (RA)
Video Filtering Pipeline. In the video filtering pipeline, we utilize a zero-shot video captioning model from Tewel et al. (2022), speech-to-text model Whisper (Rafford et al., 2022), and GPT-3.5 (Ouyang et al., 2022). For the video captioning model, we optimize pseudo tokens for 25 iterations at inference time to guide the pretrained GPT-2 (Radford et al., 2019) with the CLIP ViT-L/14 image encoder (Radford et al., 2021). We use A optimizer (Loshchilov and Hutter, 2019) with a learning rate of 0.008 and an L.2 weight decay of 0.003. For Whisper, we use the large-v2 model. For GPT-3.5, we use text-davinci-O00 and set the temperature to 0 for funny utterance detection and 0.3 for explanation generation.
w/o
0.540
0.783
w/o T
0.463
0.753
TS w/ Prompting
w/o A
0.801
w/ V. T. A
0.584
0.804
W/o V
0.551
0.788
W/ T
0.497
0.767
BART w/ Prompting
w/ A 0.587
0.805
w/ V. A 0.588
0.805
Table 4: Ablation results of T5 and BART with our prompting measured by SentBERT and ROSCOE scores when each modality component is removed. V, T. and A denote visual, speech, and sound, respectively.
encoder. We sample video frames at rate of 1 fps. Except for the aforementioned hyperparameters, we use the default values for all models.
Video-to-Text Prompting. During the prompt- ing stage, we use BLIP-2 (Li et al., 2023a), Intern- Video (Wang et al., 2022a), Whisper, ChatGPT (OpenAI, 2023), and an audio-tagging model from Schmid et al. (2022). We use the coco-pretrained BLIP-2 model with nucleus sampling. For Intern- Video, we use CLIP ViT-L/14 as the image encoder. We set the temperature to 0.3 for ChatGPT, and we use the mn40_ model for audio tugging.
B Additional Ablation Study
We conduct ablation experiments on BART and T5 with our prompting as well, and the results are as shown in Table 4. Similar to the results of GPT-3.5 with our prompting, using all modalities achieves the best performance, and there is certain degree of performance decrease when the visual component is removed.
Explanation Generation. To generate explana- tions with baseline models, we finetune T5 (Raffel et al., 2020) and BART (Lewis et al., 2020) with a batch size of 4 for 5 epochs. We use the AdamW op- timizer with a learning rate of 2e-5 and an L2 weight decay of 0.01. Additionally, we train MAF (Kumar al., 2022), multimodal end-to-end model with an aduptor added BART, with batch size of for 20 epochs. We use the AdamW optimizer with an L2 weight decay of Ic-4, and the learning rate is set to 5e-8 for BART parameters and 5e-7 for the remaining parameters. We use BART Large for all models.
C Crowdsourcing Details
We use three different user interfaces of Amazon Mechanical Turk (AMT) for (i) annotating the times- tamps and explanations of funny moments, and the human evaluation of (ii) rating and (iii) comparison, as shown in Figures 6-8, respectively. We guarantee AMT workers receive fair wages of approximately $18 per hour. Additionally, we allocate about $2 as compensation for each data point and grant addi- tional wages to workers contributing extended time and effort.
Rationale Quality Experiments. For the ratio- nale quality experiments with moment localization, we train QD-DETR (Moon al., 2023) with a batch size of 128 for 200 epochs. We use the Adam W optimizer with a learning rate of le-5 and an L2 weight decay of le-4. We optimize with the mo- ment retrieval loss consisting of the LI loss, the cross-entropy loss and the generalized IoU loss. We use the loss balancing terms of 10, and for each of them, respectively. We do not use the saliency loss. We use the hert-base-uncasee model (Devlin et al., 2019) as the text encoder with the max query length set to 400 and CLIP ViT-L/14 as the video
D Case Study
Figures 9-12 show representative videos accepted or excluded by our video filtering pipeline. Fig- ures 13-18 provide several examples to demonstrate humor explanations that our baseline models actu- ally generate. We color-code relevant (blue) and irrelevant (red) information contained in generated explanations. LLMs with our prompting, especially GPT-3.5, correctly explain the funny moments in Figures 13-16 while text-only LLMs and MAF fail to. All the models fail to explain humorous moments in Figures 17-18.

---

Temporal localization & Explanation
Please enter the shortest possible time range between second seconds, and explain why the video funny including visual information funny remarks speech.
than moments, please input field them Individually. Filling fields a ian't necessarlly good thing. Write only what makes you laugh particularty.
R
Pass the butter robot from Rick
Watch later
Share
Watch on YouTube
your convenience, provide video subtitles. This could be wrong.
Video subtitles
What purpose? Pass the butter. Thank you. What my purpose?. You passed butter. Oh god.
Start time (secon-
End time (sacondss
Explanation
20
24
The robot realizes his purpose 'life' to "pass butter' and reacts with verbal "oh my God" while slumping dismay. It's funny, because it's reaction we'd normally attribute to human and not robot.
Start time (s...
End time (se...
Explanation
Add field
your answer being submitted, please check requirements below.
1. Start time and end time entered mustt be less than length of the video.
2. Start time must be less than end time.
3. Enter the time range short as possible. (The maximum limit removed by reflecting the feeaback.)
The explanation
least 100 characters.
(Optional) Please let us know if anything was unclear, if you experienced any issues, or you have any feedback for us.
Submit
Figure A user interface annotating timestamps and explanations humorous moments. Workers are asked to watch video, identify up to three funny moments, and provide the start/end timestamps along with the explanation for each moment.

---

[Your Work]
Coffee Slam
Rate Explanations
Please read each explanation and rate as 'No", "Weak No', "Neutral", "Weak Yes", "Yes" based the criterion and then check shortcomings each explanation. More details on the shortcomings are provided below.
Descriptions of Shortcomings
Missed Humor: The explanation fails to identify the key aspect of the humor in the video.
Too Vague: The explanation isn't specific about the funny parts of video.
Incorrect Details:
The explanation has wrong information or misinterprets the funny elements.
LLacking Context explanation misses important context or background information.
Too Verbose: The explanation is overly lengthy, making it harder to understand the humor.
Explanation #1.
The delivery of line "I got the coffees!! delivered with exaggerated enthusiasm, which funny Itself. The response of "Ahl" is delivered with surprised and relieved expression, which adds to the comedic effect. The combination of the two lines creates humorous moment that elicits laugh from the viewer.
Q1. Does the explanation clearly desoribe the reasons why funny including video content?
Weak No
Neutral
Weak Yes
Yes
Q2. What are the shortcomings of the explanation? (Check all.)
Missed Humor
Too Vague
Incorrect Details
Lacking Context
Too Verbose
None
Figure 7: A user interface for human evaluation through rating. Workers are asked the explanation scale of No. Weak No, Neutral, Weak Yes, to Yes, and to choose any shortcomings if present.

---

[Your Work]
Coffee Slam
Choose a Better Explanation
Please read the following three pairs of explanations and for each pair choose the one that explains better why the video is funny.
Explanation #1.
The delivery of which is funny in itself. The response of "Ah!" is delivered with surprised and relieved expression, which adds the comedic effect. The combination of the lines creates humorous moment that elicits laugh from viewer.
got coffees!* delivered an exaggerated enthusiasm,
Explanation #2.
video starts a man on skateboard holding onto pile of cups, and then the audio caption of skateboard breaking in the background of speech is heard. The man then falls the ground by trash cans, and bandana. The last scene shows the same skateboarder walking next to truck, which implies that he was the who broke the skateboard.
shows guy skateboard wearing
Q. Which one provides more clear explanation of why it is funny, including video content?
Exp #1
Exp #2
Figure 8: A interface for human evaluation through comparison. Workers are asked compare GPT-3.5 with our prompting to text-only GPT-3.5, MAF, and Gold, respectively. and select the superior one.

---

Video showing hostage taker detonating crowded mall during the game stealth gameplay mode of multiplayer.
Video in the insect world, and it's.
Is that magnifying glass? Yep.
Are you feeling warm sudden?
We've done Hey.
Funny Utterance w/ V
Funny
Are you feeling warm all sudden?
Utterance w/ V
Funny
X
Utterance w/o V
Figure 9: An example of a video excluded in the second step (Figure 2 (b)) of the filtering pipeline.
Figure 11: An example of video accepted in the third step (Figure (c)) the filtering pipeline.
Video newspaper ads from the time of century ago.
Video shows the brain's response to cartoon,
Hey, why did they call it Mordor? That place seemed like it had a reasonable number doors. You know you can't get rid me. That won't stop me from trylig"
Alright, what's up everybody? just randomly stumbled upon article in old issue of the Delta Colorado Independent. This from October 19th, 1923, Someone dreamed the other night that he was living in the year 2023 and people were going strike because they only got $125 day while the price of eggs had gone up to $10 dozen, Ooh, spooky, right?
Funny Utterance w/ V
You know you can't get rid of me.
Funny
You know you can't get rid of me.
Funny Utterance
Utterance w/o
Ooh, spooky, right?
The video funny because the brain's response the cartoon unexpected and the subtitles are humorous.
Explanation w/V V
Funny
Ooh, spooky, right?
Utterance w/o
The video funny because is humorous take on classic fantasy setting Mordor, with protagonist's attempts escape being thwarted by the sheer number doors.
The video is funny because it a humorous take the absurdly high prices of eags in future compared to the wages the
Explanation w/ V
Explanation w/o V
The video is funny because it absurd to think that people would go on strike for only getting $125 day in the future, when the price eggs has gone up to $10 dozen.
SentBERT score 0.49
Explanation w/o
SentBERT 0.83
Figure 10: An example of a video accepted in the fourth step (Figure 2 (d)) of the filtering pipeline.
Figure 12: An example of video excluded in the fourth step (Figure (d)) the filtering pipeline.

---

Hey Luke, sit, Luke, dandelion. AHHH! Luke, look what you did dandelion. Gus, look dandelion AH-HA- HA-HA! Look what he did to the dandelion. was trying to give him flower. AYE! MY DANDELION!
Get the muscles developed, And the muscles, She pushed a lot more.
T5
The video is funny because it presents a comical moment where a man is attempting to push his body to the point where he is unable to do so. Furthermore, the man's remark is comical because is attemptiin to make joke about how hard he is pushing his body to the point where he unable so.
T5
A man is sitting in chair and he says "look what you did to the dandelion" and then he screams, It is funny because it is unexpected and because he is screaming at the same time.
BART
The video is funny because presents a close up capture of a dandelion that has been knocked over by man that attempting give a flower. Furthermore, this scene comical because the man featured seems to be comically flailing his arms and legs in such a way that makes it appear as if he is trying to dance to beat the song playing the background. The video also funny when the man says, "I was just trying to give him a fiower," because it clear that he is not in the mood to dance, but instead merely trying to get the attention of his companion.
BART
The video is funny because it presents a close up capture of female individual that appears to be attempting to "push lot more." Furthermore, this scene is comical because clear that the individual featured attempting demonstrate
"push-a-thon" in order "get the muscles developed," which is why it is laughable to see that the woman featured is actually attempting push herself to the extent that she is actually "pushing" herself.
GPT- 3.5
The video is funny because of the contrast between speaker's 's expectation Luke's reaction to dandelion and his actual reaction. The speaker's shock and dismay at Luke's destruction of the dandelion is comically exaggerated, and the speaker's surprise is punctuated by the loud 'Aye!' when the dandelion is destroyed. The speaker's over-the-tor reaction to the destruction of the dandelion is what makes the video funny.
GPT-
The speaker's enthusiasm and repetition of the phrase "muscles" comical, as if they are trying to emphasize the importance of the muscles, The speaker's exaggerated emphasis "pushed" also adds to the humor as it implies that the person they are referring to is putting in a lot of effort. The speaker's 's enthusiasm and exaggeration makes the video funny.
3.5
the word
MAF
The video funny because it presents a clip of man saying. "She pushed lot more." It is comnccl because is unexpected to see a man say this, and also because the video is edited to make it seem as the man is implying that the woman pushed him harder than she actually did.
MAF
The video is funny because presents clip man attempting to give a dandelion to another man, but instead of giving him a flower, the man throws it to the ground, which is unexpected. Furthermore, it is comical because it is unexpected see man throw flower another man.
T5
man holding dandelions and dog shown eating them. funny because the dog so aggressive and the man trying to get him to sit, but he is not doing so well.
T5
A man is riding lawnmower and he is talking about getting his muscles developed, and then he says 'she pushed lot more." It is funny because he is talking about pushing his muscles, and also because he is talking about pushing his muscles.
w/P P
w/ P
BART
man gives dog dandelion licking It is funny because unexpected, and also because the dog seems to enjoy it,
dog starts
P
BART
A is riding lawnmower next to bunch sheep, and he says, "She pushed a lot more." funny because it is unexpected to see someone push a mower like that, and also because the man says it in a way that sounds like he is bragging.
w/ P
GPT-
The video is funny because the white dog. Luke, is initially obedient when told to sit, but then proceeds to eat the dandelion instead of just sniffing it. This causes another person to scream in shock, leading to chain reaction of laughter and more dogs being led dandelions. The unexpected behavior of the dogs and the exaggerated reactions of the people make the video humorous.
3.5
P
GPT-
The visual of a person riding lawnmower with herd of sheep humorous itself. The dialogue between the two speakers adds to the comedic effect as they discuss the man's muscles and how much he has pushed, The absurdity of the situation makes it even funnier.
3.5
w/ P
Gold
It's funny because the white dog is shown dandelion and then the dog eats the dandelion unexpectedly. It's funny because the black and white dog is shown a dandelion and does the same thing as the dog and eats the dandelion. Also, the man's intention was simply give the dog flower, not for the dog to eat it.
Gold
It's funny to because the implication is that lawnmower is sheep powered, because the sheep pushing it, instead of horse powered. It's also weird to see sheep doing that.
The dog turns and notices the dandelion, then goes over and eats the dandelion from man's hand, It's funny because of the man's exaggerated reaction.
Figure 13: (Correct) An example of explanation gen- eration results. GPT-3.5 with our prompting correctly describes the unexpected behavior of dogs and the exag- geration of the people that provoke laughter.
Figure 14: (Correct) An example of explanation gen- eration results. LLMS with our prompting describe the verbal ("She pushed a lot more") and visual elements ("a person riding a lawnmower with herd of sheep") contributing to humor.

---

OREGON COAST AQUARIUM NEWPORT
Hey, Mr. Bumblebee. Thanks for the honey. High five, Oh, yeah, there.
like the color. You like the color? Yeah. I had mixed reviews on the calor, Some people said they like it, some people said they don't like it. And said, well what color is your baguette?
T5
The video is funny because it features a man who is attempting to enter an aquarium by yelling Aquuriuum, Aquarium, Aquarium, Aquarium,"
T5
The man is describing a baguette that he bought and he says he liked the color, and then he says "some people said they like it, some people said they don't ' like it." It is funny because he is implying that people don't like the color of the baguette.
BART
It's funny because the narrator keeps saying want to go to the equarium" over and over again, as if he's trylnn to convince the audience that they should go to an aquarium.
GPT-
The speaker the video is saying the same phrase over and over again silly, sing- song voice. The repetition of the phrase is humorous, and the speaker's enthusiasm for wanting to go to the aquarium is endearing, The way they keep asking "somebody" to take them to the aquarium especially funny.
BART
A man asks woman what color is her baguette. She replies, "I had a mixed reviews on the color, some people said they don't like it." The man then asks her, "And said, well what color?' It is funny because is unexpected.
3.5
GPT- 3,5
The speaker's response to the question of whether people like the color or not is hilarious,
MAF
It's funny because the man is talking about going to the aquarium, but he says he wants to go to someone else's aquarium.
The unexpected and absurd question of what color their baguette is catches the audience off guard and elicits a burst of laughter, The speaker's deadpan delivery of the line only adds to the humor.
T5
man holding goldfiih in his hand and says "Aquarium, aquarium, wanna go the aquarium." The fish then starts squirm and he says "Okay, aquarium, somebody." It funny because the man is trying to get the fish to go to the aquarium, but the fish is not interested in going.
w/
MAF
A man says "I had a mixed reviews on the color of my baguette," It is funny because it is unexpected,
T5
The man the cartoon says "I like the color, you like the color?" and then the car is towed away. It's funny because the car is very bright color and the man in the cartoon is a very nerdy looking guy.
w/ P
BART w/ P
It's funny because the narrator keeps saying "aquarium"
and over again, but the fish doesn't seem to care.
BART A man says "I had mixed reviews the color" while looking at car. This funny because it's play on the word mixed,
GPT-
The video is funny because the goldfish is portrayed as character that wants to go to aquarium, The goldfish is seen swimming around the tank and looking up the camera with its big eyes, if it is asking for help. The video ends with an ad for the Oregon Coast Aquarium in Newport, making it even funnier.
3.5
w/
GPT-
The video is funny because the speaker is making joke about the color of the car, asking what color the person's baguette is comparison The visuals add to the humor, showing the man sunglasses and orange shirt next the orange sports car, and the tow truck driver with his tool, The sound of the car at the beginning also adds to the comedic effect.
3.5
w/
Gold
The goldfish swimming around in his bowl saying. "I want to go to the aquarium." funny because he goldfish and he's already in tank full of water.
Gold
The video funny because features an animated clip featuring man that towing a luxury while the presence of its presumably wealthy owner that makes the sly comment in stating. "I had mixed reviews on the color. Some people say they like it, some people said they don't like it. I said well, what color is your Baguette?" This remark is laughable because it implies that the man is rich and the person that questioning the car color not and is likely not ownership of Baguette, so therefore, their opinion is irrelevant.
Figure 15: (Correct) An example of explanation gener- ation results. Except for LLMS with our prompting, the presence of a goldfish (or fish) is not mentioned. Partic- ularly, GPT-3.5 with our prompting accurately describes the character and behavior of the goldfish, and even mentions the content of the concluding advertisement.
Figure 16: (Correct) An example of explanation gener- ation results. Text-only LLMs do not mention a car that has a similar color to a Baguette. Meanwhile, LLMs with our prompting provide details about the car in the scene. Note that GPT-3.5 with our prompting can ex- plain the sarcasm related the baguette like in Gold.

---

Hey man, raining today?
Hey, Mr. Bumblebee, Thanks for the honey, High five. Oh, yeah, there.
Uh... Yeah, it's sprinkling.
T5 It's funny because the man is asking if it's raining and the weatherman says it's sprinkling, which is unexpected.
T5
The video is funny because it presents a comical scene of man attemppting to eat a bumblebee, which is surprisingly common occurrence in nature. Furthermore, the man's attempt to eat bumblebee is comical because it is a slapstick comedy that is comically portrayed in the clip.
BART
It's funny because the asks his friend if it's raining and the friend replies, "Yeah, it's sprinkling" and then the camera zooms on the ground where it is actually raining.
BART
The video is funny because it presents a close capture of male individual that is attempting to high five a "Bumblebee" character, Furthermore, this clip is comical because the clip is edited to make it appear as though the male individual is actually attempting to 'high five" the Bumblebee character, which is a silly notion because absurd.
GPT-
The video is funny because the man asking the question standing outside the rain, yet he still has ask if it's raining. The person he's asking looks at him incredulously and simply replies "Yeah, it's sprinkling". The contrast between the man's obliviousness and the other person's deadpan response is humorous.
3.5
GPT- 3,5
The video is funny because the speaker is talking to bumblebee as if it is a person. The speaker then gives the bumblebee high five, which is a humorous gesture as bumblebees are not able reciprocate the gesture. The speaker then says "Oh, yeah, there" satisfied tone, making the situation even funnier.
MAF
It's funny because the asks the other man if it's raining today, and the man replies "yeah, it's sprinkling".
T5
The man asks the other man it's raining, and he replies, 'uh, it's sprinkling." It is funny because he implying that is bad and he is implying that bad.
w/ P
MAF
The video is funny because it features clip of man giving a high five to a bee, which is unexpected. Furthermore, this dlip comical because man's high-fiving the bee is unexpected, and also because the bee's reaction to the high-five is not what you would expect from a bee.
BART
A man asks his friend if it's raining today and his friend replies, "yeah, it's sprinkling" " It is funny because it is unexpected that someone would say that.
w/ P
GPT-
The humor in this video comes from the absurdity of the situation, The man is driving while eating an ice cream cone, which is already a bit ridiculous, but then the question about the weather adds to the humor. The deadpan response "yeah, it's sprinkling" adds to absurdity and makes the situation even funnier.
3.5
A man says "hey Mr. Bumblebee, thanks for the honey, High five." It funny because he is pointing at bumblebee and he pointing at flower.
w/ P
w/
BART
The video is funny because it captures a clip of male individual attempting to high five a bumblebee that appears to attempting to get away from Furthermore, this clip comical because male individual attempttnn capture the attention of the bee but is comically unsuccessful his attempt to do
w/ P
Gold
What is funny is that the man in the car puts his plain white icecream cone outside his window and when he brings it back inside the car you can see that it is covered with sprinkles. Another man had asked him if he thought is was raining today and he uses his sprinkled "sprinkling". Making joke the two meanings for the word sprinkles creates the humor and the unexpected site of the icecream cone suddenly covered with candy sprinkles.
show that is
GPT-
The scene is funny because the man is trying to high five a bumblebee, which is an impossible task. The bumblebee ablivious the man's attempts and is instead facusing on the daisies. The juxtaposition of the man's excitement and the bee's obliviousness is humorous.
3.5
w/
Gold
It's funny because you expect the bee to sting the man but the bee gives the man a high five instead and then puts his arm back.
Figure 17: (Incorrect) An example of explanation generation results. Unlike both text-only BART and GPT-3.5 say it's raining, GPT-3.5 with our prompting correctly mentions man eating an ice cream cone in his car. However, it fails to explain the use of the "sprinkled cone" for pun.
Figure 18: (Incorrect) An example of explanation gen- eration results. Unlike text-only LLMss, both MAF and LLMs with our prompting correctly identify Bumblebee as a bee, not a character. However, they incorrectly generate explanations saying that Bumblebee fails to perform a "high-five." which differs from Gold.


Reviews:
[{"review_id": "BNQcQ512yc", "reviewer": "Reviewer_3agr", "summary": "", "questions": "1.\tAlthough binary classifications or ratings do not fully evaluate whether a model understands humor in detail, have you made any relevant attempts? And whether the LLM-labelled humor samples exhibit LLM-specific preferences?\n2.\tYou believe that a flaw of the dataset in MAF is the reliance on visual information (at line 79). Why do you spend much resource to mine visual modality with BLIP-2?\n3.\tWhat are the prompt demonstrations you used in the experiments?\n4.\tWhat is the difference between speech and sound contexts? Which is more important?\n5.\tAre there other prompt-based baselines for textual description generation?\n6.\tWhat is the time complexity? Where is the implementation details?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback and address all comments below.  \n\n\n**Reasons to reject 1. & Question 2. Disagreement on the reliance on visual information in the dataset of MAF.**      \n\nWe feel very unfortunate that there is a big misunderstanding about our description of the MAF dataset. In line 79, we mentioned that “However, this dataset is limited to sarcasm, a specific form of humor, and focuses on sarcasm explanation in dialogue. It highlights a need for a humor explanation dataset that considers visual elements and covers general humor.” We never said that there is a flaw in the MAF, but pointed out that it only dealt with sarcasm as a specific form of humor, and thus a dataset that covers general humor is needed like our ExFunTube dataset. Also, we never said that it relies on visual information, but simply said it focuses more on dialogue. In fact, the MAF paper named its target task as the “Sarcasm Explanation in Dialogue” task.   \n     \n\n**Reasons to reject 2. Loss of modal-specific information due to conversion into text description.**      \nOur paper title is “Can LLMs understand video humor?” Since most powerful LLMs, including GPT-3.5 and 4, can receive only text input, we have no choice but to convert the video content into text. GPT-4 is supposed to be multimodal by enabling an image as input; as of the paper submission, it could not accept images as input. Even if we use feature encoding and model fusion methods, in the end, we have to convert it into text to exploit LLM’s incredible ability of explanation generation. In both cases, information loss is inevitable when we have to provide the video content as text input to LLMs. Therefore, we explored a prompting method that can minimize such information loss.     \n\n\n**Reasons to reject 3. & Question 3. No prompt demonstration and misplaced emphasis on performance improvement.**       \nThe prompt demonstration is shown in Figure 3. (b). The reason for mentioning performance improvement in line 334 is to hint that the effectiveness of our prompting would be validated in the experiments in Section 5.    \n\n\n**Reasons to reject 4. & Question 6. Concerns about the resource intensity of the proposed approach.**      \nThis work focuses on evaluating whether LLMs can understand the humor in the videos. Given that we were going to use existing LLMs, our methodological focus was how to generate and configure a text prompt that can maximally extract LLM’s capability for humor explanation. Since the multimodal information in a video should be converted into text as an input to LLMs with minimal information loss, our choice was to use state-of-the-art (SOTA) models, even if it requires some amount of computing resources. In the case of BLIP-2, we extracted time (sec) * 5 frames/sec * 20 captions/frame captions for each video, so video captioning took about 48 hours for our whole dataset using 8 A6000 GPUs. Also, speaker diarization with GPT-3.5-turbo took 48 hours, and explanation generation with GPT-3.5 text-davinci-003 took 24 hours. Considering the size of the dataset, this amount of computing resources and time is not unrealistic. Please see Appendix A for more implementation details asked in Question 6.    \n\n\n**Reasons to reject 5. & Question 5. Lack of comparisons with various prompt baselines.**      \nThe results of various prompt baselines using the same LLM are presented in Table 2 and Table 3. The Text-only in Table 2 indicates the prompt baseline using only transcripts obtained from speech. Additionally, Table 3 shows the results of our approach excluding one component each from visual, speech, and sound.   \n\nDuring the rebuttal period, we conduct additional experiments with another multimodal prompt baseline VideoChat* [1]. It extracts information from videos using two modules: (i) VideoChat-Text generates text descriptions from the video, and (ii) VideoChat-Embed encodes the video as embeddings. The text descriptions and embeddings are combined with the received question as a prompt given to the LLM to generate a response. VideoChat uses the same  LLM (GPT-3.5 text-davinci-003) as our method, and the question given to the LLM is “Explain why the video is funny in up to 3 sentences.” The results can be seen in the table below. Our prompting method scores significantly higher than VideoChat by 0.150 in SentBERT Mean and 0.069 in ROSCOE (RA) Mean.   \n\n*The VideoChat-Text module adopts Tag2Text [2], GRiT [3], and T5, in addition to InternVideo and Whisper that are also used in our prompting. Moreover, the VideoChat-Embed module includes StableVicuna [4] and the temporal modeling of InternVideo, in addition to BLIP-2 used in our prompting. That is, the VideoChat baseline is much heavier than our prompting with worse performance.     \n \n\n\n||SentBERT @0.7|SentBERT @0.6|SentBERT @0.5|SentBERT @0.4|SentBERT Mean|ROSCOE (RA) @0.8|ROSCOE (RA) @0.7|ROSCOE (RA) Mean|\n|---|---|---|---|---|---|---|---|---|\n|Videochat w/ GPT-3.5|0.020|0.115|0.344|0.671|0.452|0.197|0.783|0.748|\n|MAF|0.149|0.375|0.604|0.809|0.541|0.438|0.880|0.785|\n|Our-prompting  w/ GPT-3.5 |0.214|0.541|0.806|0.945|0.602|0.639|0.971|0.817|\n\n\n**Reasons to reject 6. Missing ablation studies.**      \nWe conducted ablation experiments; please see Table 3 in the main draft for the results of GPT-3.5 with removing each of the visual, speech, and sound components. We also discussed these results in Section 5.6. Moreover, the results of using T5 and BART as the backbone LLM were presented in Table 4 in Appendix and discussed in Appendix B.  \n\n\n**Reasons to reject 7. LLM with prompting naturally surpasses Transformer.**      \nThe table above shows that the VideoChat prompting with GPT-3.5 underperformed MAF in terms of SentBERT and ROSCOE (RA) scores. It implies that simply adopting a strong LLM does not guarantee higher performance, and our prompting method accurately converts the video content into text and makes GPT-3.5 better explain the humor in the videos.\nFurthermore, Table 2 already showed that the BART with our prompting performed better than MAF. Please note that MAF also used BART-Large as a backbone with additional modules. In Table 2, BART with our prompting had a higher SentBERT Mean score by 0.047, ROSCOE (RA) Mean by 0.02, Rationale Quality score by 1.4, and Human Evaluation Rating by 0.151.   \n\n\n**Reasons to reject 8. Low reproducibility.**      \nThe implementation details are provided in Appendix A, and the proposed prompting does not require additional training. We will also release the code for dataset collection and our prompting method.  \n\n\n\n**Question 1. Attempt at binary classification and presence of LLM-specific preferences.**      \nSince our target task is humor explanation in natural language, we did not try binary classifications or ratings. Recent studies ([5], [6]) also argued that moving forward from identification is necessary. Our dataset is fully labeled by humans; the funny videos were verified by AMT workers, and explanation sentences were also written by human workers.  \n\n\n**Question 4. Difference between speech and sound components.**      \nSpeech refers to utterances, while sound refers to other non-verbal auditory effects. For instance, in the example of Figure 1, the utterances represented by the blue bubble are speech, while the sound is the barking of a dog or the sound of panting. Figure 3. (b) shows how speech and sound are used in our prompt, where speech includes utterances like “Hey Luke, sit. Luke, dandelion. AHHHH!”, and sound includes sound tags like “Speech, Panting, Dog”. The ablation results in Table 3 show that speech is more important than sound, as the w/o T baseline (excluding speech) led to a more significant performance drop than the w/o A baseline (excluding sound).   \n\n\n\n**References**   \n\n[1] Li, KunChang, et al. \"Videochat: Chat-centric video understanding.\" arXiv preprint arXiv:2305.06355 (2023).  \n[2] Huang, Xinyu, et al. \"Tag2text: Guiding vision-language model via image tagging.\" arXiv preprint arXiv:2303.05657 (2023).  \n[3] Nguyen, Van-Quang, Masanori Suganuma, and Takayuki Okatani. \"Grit: Faster and better image captioning transformer using dual visual features.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.    \n[4] StableLM contributors. Stablelm: Stability ai language models. stableLM, 2023.    \n[5] Kumar, Shivani, et al. \"When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.   \n[6] Chakrabarty, Tuhin, et al. \"Flute: Figurative language understanding through textual explanations.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tConstruct a multi-modal humor video dataset ExFunTube from YouTube with high quality control and support from large language model(LLM), the dataset cover a wide range of domains with various types of humor, and contains annotate start and end timestamps of funny moments and provide text explanations for each moment. This seems the only contribution.", "reasons_to_reject": "1.\tThe author believes that a flaw of the dataset in MAF is the reliance on visual information (at line 79). However, my opinion is exactly the opposite. Since it is a multimodal data set, it is reasonable to rely on any modality, and each modality has its importance.\n2.\tDue to the conversion of visual and acoustic modalities into text descriptions, the final explanation of humor is entirely dependent on text modality, which leads to the loss of some important modal-specific information in the intermediate steps. More appropriate feature encoding and modal fusion methods could be improved.\n3.\tIn section 4.1, I didn’t see any prompt demonstration. And I didn’t understand why you emphasize the performance improvement (Line 334) in the section 4.1, which is the part of approach, rather than experimentation.\n4.\tThe proposed approach relies on tedious preprocessing steps for three modalities. For example, BLIP-2 needs much computing resource and time. And simultaneously with GPT3.5, the application of this approach is very unrealistic.\n5.\tSome comparisons with various prompt baselines (same LLM) are missing.\n6.\tMissing the experiments and analysis about the importance of visual, speech and sound in your task.\n7.\tWith LLMs, it seems natural for your proposed approach performing better than MAF with only traditional Transformer. \n8.\tThis paper is hard to follow and the experiments are difficult to re-produce.", "questions_to_authors": "1.\tAlthough binary classifications or ratings do not fully evaluate whether a model understands humor in detail, have you made any relevant attempts? And whether the LLM-labelled humor samples exhibit LLM-specific preferences?\n2.\tYou believe that a flaw of the dataset in MAF is the reliance on visual information (at line 79). Why do you spend much resource to mine visual modality with BLIP-2?\n3.\tWhat are the prompt demonstrations you used in the experiments?\n4.\tWhat is the difference between speech and sound contexts? Which is more important?\n5.\tAre there other prompt-based baselines for textual description generation?\n6.\tWhat is the time complexity? Where is the implementation details?", "paper_topic_and_main_contributions": "For processing multi-modal video humor explanation, this paper constructs a multi-modal humor video dataset ExFunTube from YouTube with high quality control and support from large language model (LLM). This dataset ExFunTube cover a wide range of domains with various types of humor, and contains annotate start and end timestamps of funny moments and provide text explanations for each moment. Then, this paper proposes a zero-shot prompt approach to explain video humor via LLM’s Excellent semantic understanding ability. Three different evaluation methods seem to demonstrate the approach improves LLM's performance on humor explanation.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "“curate” doesn't seem like a common word."}, "extra_text": {}}, {"review_id": "AVwfpIHeHc", "reviewer": "Reviewer_k1U2", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and for addressing all concerns below.\n\n**Reasons to reject 1. Unbalanced distribution of the number of humor points and need for negative samples.**  \n\nIn our dataset, the distribution of humor intervals over videos is not balanced, but we believe this is not a big issue. Each video includes varied content, so it can have any number of humor intervals. Furthermore, the videos in our dataset are short-form, often less than 30 seconds, it is more common for each video to have one humor interval. The purpose of collecting multiple intervals per video is to ensure that annotators capture all the humorous moments and document all of them to avoid missing.   \n\nWe thank the reviewer for the suggestion about negative samples. During the dataset collection, MTurk workers also identified the videos that had no funny moments. The number of such videos is 453 (about 4%), and they can be used as negative samples.    \n\nOr, one can use the parts of videos that are not included in the ‘funny moments’ as negative samples. We will update our manuscript to discuss this negative sample issue clearer.    \n   \n\n\n**Reasons to reject 2. Concerns over unguaranteed dataset quality.**  \n\nAs stated in Appendix C, MTurk workers must meet high standards to participate in our dataset annotation: a HIT approval rate of 99% or higher, a total of more than 10,000 approved HITs, and be located in one of the countries of AU, CA, GB, NZ, or US. In our qualification test, we only selected the workers who could explain humor well in funny videos. Only 60 workers out of 219 passed the qualification test, indicating our rigorous and careful selection.   \n\nIt is challenging to guarantee the quality of free-form annotations. Therefore, we let the workers first annotate timestamps to identify what content is funny, and then write a detailed answer of at least 100 characters about the moment they found funny. Furthermore, as mentioned in Section 3.2, we asked each annotator to write in the format of “[What is funny]. It is funny because [Explanation of why funny]” to ensure they not only describe the funny moment but also elaborate on the reason why it is funny. We removed the responses that do not follow this direction. Workers who continued to disregard this standard were excluded from further annotation.\nResponses that include both the description and the reason were individually reviewed by the authors to verify whether the reason for humor was valid, and if not agreed upon, the responses were collected again. The human evaluation score of the annotated explanation was 0.792, as shown in Table 2, which fell between weak yes (0.75) and yes (1), demonstrating the appropriateness of the annotation.  \n\nFurthermore, when we performed human evaluation, we asked workers to check all shortcomings of the explanation. The shortcomings include Missed Humor, Too Vague, Incorrect Details, Lacking Context, and Too Verbose; please see Figure 7 in Appendix for descriptions of these shortcomings. For randomly sampled 100 videos, we asked five annotators per video to select all shortcomings that existed and select None if there was no shortcoming. The presence of each shortcoming was determined by a majority vote.  \n\nThe results are shown below. For Gold explanations, instances of Missed Humor or Incorrect Details are much less than model-generated explanations. Also, instances of Too Vague and Lacking Context are less than model-generated explanations except GPT-3.5 w/ Prompting. It seems that GPT-3.5 w/ Prompting includes much information to explain humor, which causes the trade-off between vagueness/lacking context and verbosity (19 instances of Too Verbose for GPT-3.5 w/ Prompting). Lastly, there are significantly more instances of None for Gold explanations than for model-generated explanations and about three times more than GPT-3.5 w/ Prompting. This suggests that Gold explanations show high-level quality.  \n   \n||Missed Humor|Too Vague|Incorrect details|Lacking Context|Too Verbose|None|\n|---|:---:|:---:|:---:|:---:|:---:|:--:|\n|Text-only BART|75|17|69|27|24|0|\n|Text-only GPT-3.5 |56|25|40|33|2|4|\n|MAF|76|21|72|47|4|0|\n|BART w/ Prompting|62|18|64|34|7|1|\n|GPT-3.5 w/ Prompting|29|7|48|15|19|11|\n|Gold|10|15|3|20|2|38|\n\nWe additionally conduct human evaluations for randomly sampled 100 videos to verify the quality of annotation. Based on the criteria of good explanations that [1] suggested, which are Circularity, Relevance, and Coherence, we ask three annotators per video to respond with either Yes or No. The results are shown below. High Circularity, Relevance, and Coherence scores show considerable quality of annotations. Since the rebuttal period is short, we evaluate only 100 examples, but we will extend it to the whole dataset in the final version.\n\n||Circularity|Relevance|Coherence|\n|---|:---:|:---:|:---:|\n|Score|0.94|1.0|0.98|  \n\n\n\n**Reasons to reject 3. Uncomfortable content in the dataset.**  \n\nWe manually reviewed videos based on the safety objectives proposed by [2] from Google Research, as mentioned in the Ethics statement. We rigorously filtered out any videos with ethical issues; as a result, we chose only 10K videos out of the 21K ones that passed through our pipeline (50% were filtered). The second example that the reviewer felt uncomfortable has no ethical issue according to our filtering criteria. In the video, three characters emotionlessly repeated “That’s so funny” in irregular timings. However, we totally understand that it might be uncomfortable for someone since the characters may look somewhat bizarre. We will recheck all the videos again and see if there are any videos that might cause some discomfort.\n\n**References**    \n\n[1] Von Wright, Georg Henrik. Explanation and understanding. Cornell University Press, 2004.  \n[2] Thoppilan, Romal, et al. \"Lamda: Language models for dialog applications.\" arXiv preprint arXiv:2201.08239 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The article released a large-scale multimodal humor video dataset that provides novel multi-humor interval annotations, which can provide new data support for community understanding of videos.\n2. Although there are numerous similar frameworks, the article provides a simple and feasible method for converting videos to text. It can capture videos according to their plot/shot.\n3. The article provides a feasible method for understanding humorous videos, which can achieve the SOTA understanding performance.", "reasons_to_reject": "1. The article provides humor interval annotations, but the distribution of the number of humor points is not very balanced, and I believe that videos without humor intervals should be added (negative samples can assist the model in understanding the true meaning of humor).\n2. The annotation of the released dataset was not fully evaluated for quality (or not displayed in the article), so the quality of the dataset cannot be guaranteed.\n3. There is uncomfortable content in the released dataset. For example, the second example in the attachment not only has formatting issues, but its content also makes me feel very uncomfortable while watching.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The article released a multimodal humor video dataset. It uses mature tools to convert visual and audio signals into text signals, and is manually annotated. The videos in this dataset contain one or more humor points, and experiments have shown that although existing large models can effectively identify humor points, there is still room for improvement.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "There is some uncomfortable content in the dataset."}, "extra_text": {}}, {"review_id": "fOyPWR3L2v", "reviewer": "Reviewer_gSbC", "summary": "", "questions": "A. Were there instances where videos were identified by the MTurk annotators as non-humorous but weren't eliminated by the pipeline? This could provide a quantitative assessment of the pipeline's efficiency.\n\nB. Can you specify the number of annotators per video? Was there only one annotator for each video, or were multiple involved?\n\nC. How adaptable do you believe your framework is to other datasets of different topics? Could you shed some light on potential difficulties or nuances that might arise during such generalization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all feedback and positive comments. We address all concerns below.  \n\n**Reasons to reject 1. Underutilization of audio information.**  \n\nWe divided the audio information into speech and sound components. From the speech component, we extracted transcripts, and from the sound component, we obtained sound tags. We temporally aligned the speech with the visual component using timestamps, but we did not with the sound tags, which could be seen as not fully utilizing the audio. However, it was based on our experiments that putting sound tags for the whole video led to better performance. We tested temporally segmenting the whole audio input and adding sound tags per segment, but it was unstable for training and led to information loss and hallucinations. \n\n**Reasons to reject 2. Lack of detailed justification for model selection and insufficient discussion on potential pitfalls.**  \n\nOur prompting method aims at extracting as much information as possible from YouTube videos (diverse user-generated videos) into text in a zero-shot manner. Therefore, we apply SOTA zero-shot models for each modality. For example, in the dataset collection pipeline, we used a Zero-Shot Video Captioning model [1] for video captioning, which adopted GPT-2 [2] and CLIP [3] that were pretrained on large-scale datasets. In prompting, for speech understanding, we labeled the speaker using ChatGPT with the transcript obtained from Whisper. We also tried other models, for example, mPLUG-2 [4] and Swinbert [5] for zero-shot video captioning, OFA [6] and mPLUG [7] for image captioning, GRiT [8] and GLIPv2 [9] for object detection, CLIP4Clip [10] for video-to-text retrieval, Pyannote [11] for speaker diarization, and CAV-MAE (audio-only) [12] for sound tagging, but they suffered from more information loss and hallucination than the models we chose.   \n\nAdmittedly, there were other issues when we used our chosen models. The Zero-Shot Video Captioning model took 10 minutes per video because it repeated the process of CLIP loss guidance to GPT-2 at inference time. In total, it took one week with 8 A6000 GPUs to caption all candidate videos before passing through the pipeline. Also, BLIP-2 took 48 hours with 8 A6000 GPUs to extract captions from video frames. For speaker diarization, ChatGPT sometimes made errors because it is given only the transcript, but it worked better than other models. Finally, unlike other models, EfficientAT was not robust to our dataset because it was not pretrained on sufficiently large-scale datasets.  \n\nIn the final draft, we will add the detailed rationale for choosing each model and the limitations that could occur.  \n\n\n**Question 1. Non-humorous instances for assessment of pipeline’s efficiency.**  \n\nDuring the dataset collection, MTurk workers also identified the videos that had no funny moments. The number of such videos is 453 (about 4%), which indicates that our pipeline is effective.  \n\n\n**Question 2. Number of annotators per video.**  \n\nEach video was annotated by a single worker. We did not obtain multiple annotations per video, because it would take a lot of time and cost for large-scale video sets. Instead, we carefully selected annotators who could provide high-quality explanations, and we verified every annotation by ourselves. As stated in Appendix C, MTurk workers must meet high standards to participate in our dataset annotation: a HIT approval rate of 99% or higher, a total of more than 10,000 approved HITs, and be located in one of the countries of AU, CA, GB, NZ, or US. In our qualification test, we only selected the workers who could explain humor well in funny videos. Only 60 workers out of 219 passed the qualification test, indicating our rigorous and careful selection. More crowdsourcing details can be found in Appendix C.  \n\n\n**Question 3. Adaptability and potential challenges of the framework in generalization.**  \n\nOur prompting framework, designed to extract as much information from videos into text as possible, can be applied to multimodal datasets of different topics, including three types of TV shows, sitcoms, medical dramas, and crime dramas, in TVQA [13], and short clips from movies in LSMDC [14].   \n\nHowever, it may have some difficulties in certain types of videos. To extract visual information, we divided the video into multiple segments using utterance timestamps and scene boundary detection. Therefore, videos with little utterances and infrequent scene changes (e.g., Kinetics [15]) may not be easily segmented by our method. Additionally, we obtained sound tags from the entire audio stream for considering non-verbal auditory effects. This may not be suitable for datasets that require detailed information about sound (e.g., MUSIC-AVQA [16]).  \n\n\n**References**    \n\n[1] Tewel, Yoad, et al. \"Zero-shot video captioning with evolving pseudo-tokens.\" arXiv preprint arXiv:2207.11100 (2022).\n[2] Radford, Alec, et al. \"Language models are unsupervised multitask learners.\" OpenAI blog 1.8 (2019): 9.\n[3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.\n[4] Xu, Haiyang, et al. \"mPLUG-2: A modularized multi-modal foundation model across text, image and video.\" arXiv preprint arXiv:2302.00402 (2023).   \n[5] Lin, Kevin, et al. \"Swinbert: End-to-end transformers with sparse attention for video captioning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.  \n[6] Wang, Peng, et al. \"Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.\" International Conference on Machine Learning. PMLR, 2022.  \n[7] Li, Chenliang, et al. \"mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.  \n[8] Nguyen, Van-Quang, Masanori Suganuma, and Takayuki Okatani. \"Grit: Faster and better image captioning transformer using dual visual features.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  \n[9] Zhang, Haotian, et al. \"Glipv2: Unifying localization and vision-language understanding.\" Advances in Neural Information Processing Systems 35 (2022): 36067-36080.  \n[10] Luo, Huaishao, et al. \"Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.\" Neurocomputing 508 (2022): 293-304.  \n[11] Bredin, Hervé, et al. \"Pyannote. audio: neural building blocks for speaker diarization.\" ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.  \n[12] Gong, Yuan, et al. \"Contrastive audio-visual masked autoencoder.\" The Eleventh International Conference on Learning Representations. 2022.  \n[13] Lei, Jie, et al. \"TVQA: Localized, Compositional Video Question Answering.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.   \n[14] Rohrbach, Anna, et al. \"Movie description.\" International Journal of Computer Vision 123 (2017): 94-120.  \n[15] Carreira, Joao, and Andrew Zisserman. \"Quo vadis, action recognition? a new model and the kinetics dataset.\" proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.  \n[16] Li, Guangyao, et al. \"Learning to answer questions in dynamic audio-visual scenarios.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has many strengths that make it worthy of acceptance, including the following:\n\n1. The authors have devised an innovative and inspiring method for dataset creation, seamlessly capturing both textual and visual nuances.\n2. By leveraging current state-of-the-art (SOTA) models and techniques, the paper showcases its relevance and modern approach.\n3. The team has conducted thorough experiments and evaluations.\n4. The research's modular framework stands out, designed to easily integrate or replace components with upcoming SOTA models, ensuring long-term adaptability.", "reasons_to_reject": "The paper exhibits promising strengths; however, there are certain aspects that could be improved:\n\n1. Audio information is not fully utilized. A more extensive exploitation of this modality could enhance the system's comprehension of the data and potentially improve the overall results.\n2. The choice of selected models and methods in the study could use more detailed justification. Understanding why these specific models were chosen over others and their relative advantages would be beneficial. Additionally, the potential pitfalls or errors introduced by these models are not sufficiently addressed. ", "questions_to_authors": "A. Were there instances where videos were identified by the MTurk annotators as non-humorous but weren't eliminated by the pipeline? This could provide a quantitative assessment of the pipeline's efficiency.\n\nB. Can you specify the number of annotators per video? Was there only one annotator for each video, or were multiple involved?\n\nC. How adaptable do you believe your framework is to other datasets of different topics? Could you shed some light on potential difficulties or nuances that might arise during such generalization?", "paper_topic_and_main_contributions": "This paper utilizes LLMs to understand the humorous content within videos by harnessing multimodal features. The main contributions of this study are as follows:\n\n1. The authors have curated the ExFunTube dataset, where both textual and visual features play a crucial role in conveying humor. This compilation was achieved through an innovative data selection pipeline.\n2. The authors have developed a zero-shot video-to-text prompting framework specifically designed to optimize the LLM's comprehension of humor within videos.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciated the dataset introduced by this paper, finding it "wide-domain", "large-scale", and created through an "innovative data selection pipeline". Reviewers generally agreed that the work is sound, with some concerns about possible data quality issues from reviewer k1U2 addressed in the author response. Reviewer 3agr gave a much lower excitement score than other reviewers, however I tend to agree with the other reviewers that the combination of model generations and human selection, and evaluation, will likely make this a high quality resource, and that the experiments done here look very thorough. 

To address the points related to dataset discomfort raised by reviewer k1U2, I strongly recommend that the ​​authors ensure that details of their manual review process and use of the Lamda guidelines are clear in the paper, and also recommend (but less strongly) that they write a content warning for readers of the paper, and users of the dataset, about possible discomfort caused by some of the videos in the dataset.
Decision: Accept-Main