Teaching Human Behavior Improves Content Understanding Abilities Of VLMs
Authors: Somesh Kumar Singh
Harini S I
Yaman Kumar Singla
Changyou Chen
Rajiv Ratn Shah
Veeky Baths
Balaji Krishnamurthy
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
human behavior
content understanding
behavioral science
vision language models
large language models
memorability
video understanding
image understanding
language understanding
persuasion strategy
marketing
advertising
datasets
behavior in the wild
Primary area: foundation or frontier models, including LLMs

Abstract: Communication is defined as "*Who* says *what* to *whom* with *what* effect." A message from a communicator generates downstream receiver effects, also known as behavior. Receiver behavior, being a downstream effect of the message, carries rich signals about it. Even after carrying signals about the message, the behavior signal is often ignored while training vision-language models. We show that training VLMs on receiver behavior can actually help improve their content-understanding abilities. We demonstrate that training VLMs to predict receiver behaviors, such as likes, comments, and replay graphs, which are available at scale, enhances the VLM's performance across a broad range of downstream content understanding tasks. We show this performance increase over 6 types of behavior, 46 different tasks covering image, video, text, and audio over 26 benchmark datasets across both zero-shot and fine-tuning settings, outperforming many supervised baselines on diverse tasks ranging from emotion recognition to captioning by up to 150%. We note that since receiver behavior, such as likes, comments, and replay graphs, is collected by default on the internet and does not need any human annotations to be useful, the performance improvement we get after training on this data is essentially free lunch.  We also release **BLIFT**, our **Behaviour-LLaVA IFT** dataset comprising 730k images and videos with their receiver behavior collected from multiple platforms on which we train our models to achieve this. The dataset and code are available at [behavior-in-the-wild.github.io/behavior-llava](https://behavior-in-the-wild.github.io/behavior-llava).

Published as a conference paper at ICLR 2025
TEACHING HUMAN BEHAVIOR IMPROVES CONTENT UNDERSTANDING ABILITIES OF VLMs
A Yaman K Singla
L III
Somesh Singh*
Harini S I
Rajiv Ratn Shah IID
Changyou Chen
Veeky Baths
Balaji Krishnamurthy
Adobe Media and Data Science Research (MDSR)
HITD.
SUNY at Buffalo,
CNRL and APPCAIR at BITS Pilani
behavior-in-the wwwtoo--ii-ttewwwdddeeee www.:/ooo----eeeeedeeeeeeeerrrr.
ABSTRACT
Communication defined as "Who says what to whom with what effect." A message from a communicator generates downstream receiver effects, also known as behavior. Receiver behavior, being a downstream effect of the message, carries rich signals about it. Even after carrying signals about the message, the behavior signal is often ignored while training vision language models. We show that training VLMSs on receiver behavior can actually help improve their content- undeessaddn abilities. We demonstrate that training VLMs to predict receiver behaviors, such likes, comments, and replay graphs, which are available at scale, enhances the VLM's performance across a broad range of downstream content understanding tasks. We show this performance increase over types of behavior, 46 different tasks covering image, video, text and audio over 26 benchmark datasets across both 0-shot and fine-tuning settings, outperforming many supervised basclines on diverse tasks ranging from emotion recognition to captioning by upto 150%. We note that since receiver behavior, such likes, comments, and replay graphs, is collected by default on the internet and does not need any human annotations be useful, the performance improvement we get after training on this data is essentially free-lunch. We also release BLIFT, our Behaviour LLaVA IFT dataset comprising 730k images and videos with their receiver behavior collected from multiple platforms which we train our models to achieve this. The dataset and code are available at behavior- inn--in-thhee the-theeeeweeee wild.. d.ddd..g.. gi thub. 10/ www./ehaavvorlllee
(Whatt)
Content
YouTube
Channel:
twith effect)
Receiver Effect
Amnidt and bonny ore alugg beaun fut sumn.
uniiing
Temporal Understarding
Charscter Understanding
YouTubo Subscribers
Cognitvee Undersanding
Views: Boyk
Won't make me change insrance companes,
Iit a graat commercaa..
uper Opidoon Undeestnnin
Speater (Who says): State Farm Insurance
Comments:
Replay graph
Oud Sooee firm longer mansusin
WNighta Papsh, Baach Laban Betirmaaa".. Lell Unécrstandin
Say find
crowtedge
Arrie hust can't get thet
Mast regingee mocevve In uip
Figure 1: The diagram depicts the five factors of communication in the context of an example YouTube video https: //www//www/// youtube. com/watch 2 Th04e22 2 TM and where lies the free lunch. The receiver effect is not used while training Large Vision and Language Models. However, contains many important signals that help in understanding the content. The figure shows several comments containing temporal, cognitive, character. context, and user opinion information useful for understanding the video.
INTRODUCTION
Communication defined by five factors: sender, message, channel, receiver, and behavior (Shannon & Weaver, 1949; Lasswell, 1948; 1971; Kumar, 2024). Lasswell (1948) encoded these five factors in the phrase, "Who says what to whom with what effect.." " Human behavior occurs as a downstream artifact in the process of communication. Behavior is produced by the receiver response the

---

Published as a conference paper at ICLR 2025
Simullate Susee cestmrnts
Simutaca of ststaus, liker, and comments-on this Sibeo
1. Ljst lest my jrandpu a lew wooko ago, Ewobled ut 00-5011 would parforms for him every time coulda's make is. milon
video
Who dors the glrl ln siden
2. The smile en that glrl's Iace alter the fisul stop aheays makes me cry. Like sha's proud che could put a show Yor only
him
porform foe at the and of the vileo?
audienca that matters, A momant that even money can't buy Prondly built In GEORGEAM
Behavior LLaVA
Whichh peimary sport activilty beinng shorwn the video?
Views 2013.
Likes 1.8s
levakaainn
Commentce 200
Emelion: Allectien, Bay, Pride
Persunsion
What depicted
Stratogo Social
Ideniity, Emetiin
Weike L line for this video.
W hat In the persuasion steatogg uned the vidte??
hazrescrniing giefs father Kis thoough grandlather, bringing joy
porferm her wheelchiir
Figure 2: Behavior-LLaVe is trained to answer behavioral questions like simulating user comments and likes on the video. The model, once trained, shows superior performance than LLaMA-Vid and other VLMs on content-related tasks like emotion recognition. action recognition, question answering, persuasion strategy classification, etc. The original video was showcased in SuperBowl-2024 and is posted on YouTube on the comm//atch/==UU7BJB
URL httpp://www. commercial featuring the 2024 Kia EV9" by Kia America.
The video is titled "Perfect 10: The Kia big game
message sent by the sender. Being downstream effect, behavior help infer important signals about the message itself. These signals, if properly harnessed, should be able to increase performance on the message understanding tasks popular in NLP and CV, like question answering, sentiment analysis, topic classification, etc. Despite this, behavior data is considered noise and is ignored while training large language models (Biderman et al., 2022; Penedo et al., 2023) and also large vision and language models (Liu et al., 2023a; Zhu et al., 2023)". In this paper, we explore this line of thought more.
Humans produce two kinds of behavioral signals upon observing message (Bertenthal, 1996; Prinz, 1997): perceptual signals and actions as behavior. Perceptual signals, like seeing, touching, and hearing, help receiver primarily sense the world around her, ultimately guiding her actions. Actions are how a receiver acts on the outside world. The signals produced by the human receiver upon receiving message carry information about the message itself (Fig. 1). For instance, if a person's 's heartbeat rises upon watching a movic scene, it can help us infer that perhaps the scene was an exciting scene (Dzedzickis et al., 2020). Similarly, regressing while reading is indicative of important or confusing phrases (Bicknell & Levy, 20.1). In these cases, perception behavior helps us derive inferences about content. In a similar vein, the actions a person performs after watching a movie, such as comments and likes, carry signals about the movie (Fig. 1, 2).
Expanding on these ideas, prior literature has shown that harnessing perceptual signals, like eye movements, saliency, keystrokes, mouse movements, and FMRI, by modeling them together with content understanding tasks can improve both NLP and CV tasks. For instance, integration with perception signals causes performance improvement in tasks like visual and natural language question answering (Patro & Namboodiri, 2018; Khurana et al., 2023; Sood et al., 2020), text and image sentiment analysis (Khurana et al., 2023; Barrett et al., 2018; Fan et al., 2018), natural language inference (Khurana et al., 2023), part-of-speech identification (Barrett et al., 2016,;)), named entity recognition (Hollenstein et al., 2019; Hollenstein & Zhang, 2019), syntactic parsing (Plank, 2016), image captioning (Cornia et al., 2018), and visual object detection (Wang et al., 2018b; Kruthiventi et al., 2016).
While the initial studies show that perceptual signals have much promise for improving downstream content understanding, they have a few significant issues due to which integrating human perception has not seen wide adoption in training LLMs. These perceptual signals can only be collected in lab settings requiring specialized lab equipment and are thus expensive to collect and thus are also limited in number. For example, the largest datasets containing the human processing signals are SALICON (Jiang et al., 2015) and Cheng al. (2014) for visual saliency (10k images each), CELER (Berzak et al., 2022) and Dundee corpus (Kennedy et al., 2013) containing eye movements over 28k sentences and 20 news articles respectively, and Dhakal et al. (2018) containing keystroke patterns over 1.5k
'For instance, Pile a popular dataset used for training LLMs. Biderman et al. (2022) mention that they remove all metadata (containing behavioral data) while creating Pile and subsequently training Pythia. Similarly, hehavioral data was removed while creating LA- -nn--A-nrtutt11155555 150k (Liu et al., 2023a), MiniGPT (Zhu et al., 2023), and RefinedWeh (Penedo et al., 2023) datasets.
2

---

Published as a conference paper at ICLR 2025
Nemornbiltty
Vidco Parsassies
Sehaviuus
video Harrarah
LLMAAVVID
Undarsanding
Video
ALVV
Nomeez Berrieuod
Image
Audio
lighliht Detection
Audio Sarmonsaatton
Dosse Caption
Image Emrotion
Figure 3: Behaviour-LLarr achieves much higher zero-shot performance compared Ad-LLaVA and the hase model LLaMA-VID across diverse suite of image, video, and audio henchmarks.
sentences. Clearly, these datasets, while making important contributions, do not scale the level at which today's large language models are trained (trillions of natural language and image tokens).
On the other hand, actions (the other type of behavioral signals produced by a human receiver) are collected at large scale in the form of digital analytics. Examples of this kind of data are likes, views, shares, comments, and purchase histories on images, tweets, videos, webpages, and other kinds of media. Action data has much broader representation than is possible in lab settings, is available on more diverse content, and is much cheaper to collect than using specialized lab equipment. At the same time, actions have not been much investigated in the literature for their potential to improve downstream content understanding. Most of the prior literature tries to predict the receiver behavior (Gao et al., 2020; Wang & Torres, 2022; Zhou et al., 2018; Wei al., 2016; Wang et al., 2018a) rather than using the receiver behavior to improve the content understanding capabilities of models.
Therefore, in this paper, make initial efforts collect and understand digital analytics at scale with the aim of integrating them with VLMs to improve their downstream content understanding capabilities. We introduce methods for filtering and cleaning behavioral data and then propose tasks for large language and vision models, leading to improvements in language and visual content understanding tasks. For this, look to Reddit and YouTube as two major sources of visual content and human behavior in the form of viewer comments, likes, replay graphs, and upvotes. From Reddit, we collect 5 million images and videos along with their upvotes and top-upvoted comments from two major subreddits (r/pics and r/videos). Similarly, from YouTubbe, we collect 2.2 million videos from 30000 channels along with their likes, views, replay graphs, and top user comments. After extensive filtering and cleaning, we are left with 730k samples of videos and images across the two platforms which we use for the next steps.
After collecting user behavior over image and video content, we design tasks to teach large vision and language models ( (VLMs) to simulate user behavior. For this, we use an instruction fine-tuning image and the other metadata like time of post and channel, we ask the model to simulate user behavior of likes and comments. See Fig 2, Listing for examples. We choose LLaMA- Vid (Li et al., 2023b) resultant model Behavior-LLav. (Large Language and Vision Assistant) (Liu et al., 2023a). We test Behavior-LLaVA on a diverse variety of tasks, evaluating its capabilities on image, video, text, and audio understanding tasks. We compare Behavior-LLa'e and other supervised baselines. Further, to show the impact of behavior, we train another version of LLaMA- Vid, where we train it on the same set of videos and images as Behavior- VL but do not include behavior information. We call this model Ad-L.aVA.
format. Given video
base model to teach it the user behavior. We call the
against its base model, LLaMA-Vas Vid,
We make the following contributions with this work:
1) Behavior-LLava VL Instruction Fine-Tuning: We explore the idea of learning human behavior, resulting in better content understanding. We test this for action-level behavior data such as receiver comments, likes, and replay graphs. We collect dataset called BLIFT, consisting of 400k images and 330k videos, along with their receiver behavior. Then, LLaMA-Vis is trained for the task of predicting receiver comments and upvotes given a media (a video or an image) (Listing I). We show that using this simple task formulation over behavioral data collected in the wild, results in performance improvement over a hierarchy of tasks. We get improvements over the base LLaMA- Vid across 46 tasks over 26 benchmark datasets in both zero-shot and fine-tuned settings shown in Figure 3. We show this over low-level content understanding tasks like object and activity recognition and also over high-level tasks like topic and emotion detection. Through this, propose scalable
3

---

Published as a conference paper at ICLR 2025
approach to increase the content understanding abilities of VLMs, requiring minimal cost and no architectural changes.
2) AdLLa VA: Disentangling the effect of content and behavior: To disentangle the effect of training LLaMA- Midd-iiddooo on additional image and video data from the effect of training on behavior data, train LLaMA- Vid on BLIFT's videos and images without including behavior. We call this model Ad.LLaVA. We show that Ad-LLaVA shows equivalent performance as its base model LLa Vid; however, Behavior-LLaYs performs better than both Ad-LLa and LLaMA-Vid, thus highlighting the importance of behavior data and instruction fine-tuning on behavior data.
3) Perception Action: We also show an ablation of Rehavior-LLaaa across different kinds of behavior. We try out the perception behavior of saliency prediction over images and five types of action-level behavior over images and videos. We find that perception-level behavior does result in significant performance improvements; however, action-level behavior shows improvements across all the tasks. We posit that one reason for this could be due to the scale for which action-level data is available (Table 10). While perception behavior is mostly collected in lab settings, action-level behavior data is diverse, can be collected in a scalable manner automatically and cheaply.
2 METHODOLOOGY
In this section, we introduce our approach to train Behavior-LLLaA, Since no publicly available corpus consists of behavior together with image and video content, first introduce our instruction fine-tuning dataset, "Behavior-LLaVA Instruction our methodology to train Behavior-LLaVA, Finally, we report the results of testing Behavior-LLa capabilities on hierarchy of tasks. The tasks cover low-level media understanding tasks like object and activity detection, high-level media understanding tasks like emotion, topic, and persuasion strategy classification. Figures 14, 15, 17, and 16 illustrate the topic distribution in the BLIFT dataset, highlighting the diversity of image and video themes, as well as comments, identified through BERTopic and GPT-4o-mint clustering.
dataset" (BLIFT). Next, we introduce
2.1 BLIFT DATASET
Given the abundance of media and behavioral data and its accessibility, our data collection relies on primary sources: Reddit and YouTube. These platforms share similarities in terms of hosting media content (images and videos) and providing user engagement metrics in the form of Reddit upvotes and comments, and YouTube likes, views, comments, and replay graphs. Here, we delineate the process involved in constructing the instruction fine-tuning dataset, which we term as the Behavior-LLave LLaVA Instruction Fine-Tuning Tuning (BLIFT) dataset.
2. DATA FROM REDDIT
To collect substantial corpus of diverse images and videos, targeted two specific subreddits, namely r/pics and r/videos. Established over 15 years ago, these subreddits had a user base exceeding 20 million during the data collection period, with an average of over 5,000 online users concurrently, Notably, due to stringent content moderation guidelines (Reddit, Inc., 2024; red, 2024a;bxc) and the exclusive focus of these subreddits on media content, they offer a rich variety of content devoid of thematic biases. Our data collection spans until January 2022, during which the activity on these subreddits witnessed notable decline following several policy adjustments and user protests (Hern, 2023; Economist, 2023).
To ensure data quality and relevance, we executed series of filtering steps on the posts and comments from these subreddits, illustrated in Figure 13. Initially, we excluded posts predating February 2015 from r/pics, coinciding with the implementation text (red, 2024a;b). This filtering step resulted in the exclusion of 3.1 million images and 2 million videos. Subsequently, considering the sustained popularity of both subreddits, with rankings within the top 20 since 2017 and consistent membership exceeding 20 million. confined our dataset to posts from January 2018 onwards. This selection process yielded 1.4 million images and 1.1 million videos.
rule requiring images without digital/overlay
4

---

Published as a conference paper at ICLR 2025
Listing I: Behavior Instruction fine-tuning template for the video: https: //MMw, youtube con/ vatch22v-KKQKKRFF4
<SYSTER Al visual assistant. You given detailed description of media, felloved by the actual redia. Answer all questions as if you are seeing the media.s/
SYSTEM
<USER>
The video advertisesent
for the brand Gatorade. The autio in ad says "DASR HERE v Analyze this video deeply, then write szene by scene description of the video and answer the following questions What percentape vievers would like this video, and what would be the top-5 popular coments video? shat would the replay gr grash values each scene be? <vidego
"Satorade
Footsteps with
</videe
</UBER>
<NSSISTANT>
The scene by scere descriptions are:
Scere
looking off into the distance with orange 1ine going screen. foreground colors of scone are black, green, gray, dark gray, and the background colors are dark brown, black, dark gray. The dominent tone of the scene is neutral. This scene is categorized by the tags: cosretic ponytall,
The shows
around outside
Scene
shows wonan balancing on skateboard the scene are black, nud green, dark gray, olive, and the background colors are black, dark gray, gray, dark brown The doninant tone of the scene is neutral. This scene is categorized the
yard. foreground
athletic, balance,
girl, house
hurdle,
jog. logging, plank, rail, sessas, wonan, yard.
Scone 3: The scene shows a girl jurping over a wooden ramp in the backyard. The foreground colors of the scene are black, dark gray, gray, dark blue, and the background colors purple, dark pink,
dark brawn,
200 BEHAVIOR 665
The video will be 1iked by 2.80 viewers,
popular comrents could be:
"how. Love it. She'ss such an inspiration to the next ganeration as well as everyone.
2. "Inspiring and great story behind this comnercial. Builds reaning and purpose in the hearts have been
since good, meaninghul "She"s ss insptration the world. Thanks to ny nicce learning gysnastics.
youth.
TOMORROW
Hopefully soneday, she is an inspiration to others as Suni is an to everyone" "Chillis watching this, Such Inspiration."
"Yooooo, this is
The replay values for each scane would be:
Scone T: 0.86
0.23
Scana 0.38
</ASSTSTANT>
Figure 4: Behavior Instruction fine-tuning template for the video: https://www.yottble cwm/watcht BRPQKJRF4YY
Further refinement of the dataset involved removing posts and comments marked as NSFW (through community moderators or Ilama-guard- -oard-3-1b lb (Llama Team, 2024) filters), BOT-generated, or [deleted], along with eliminating duplicate images and videos. This curation step reduced the dataset to 876,000 images and 983,000 videos. To address redundancy in comments, we excluded those comprising fewer than three words and employed TF-IDF-based deduplication with a similarity threshold of 0.6, determined through manual observations.
Following these steps, posts with fewer than two comments were filtered out, resulting in a dataset comprising 631,000 images and 397,000 videos. Additionally, videos exceeding duration of 500 seconds were omitted, leaving 221,000 videos for analysis. Notably, images not directly hosted on Reddit were excluded due to scraping and copyright limitations. Similarly, for r/videos, only videos hosted on YouTube were considered. It is pertinent to mention that approximately 51 % of YouTube videos collected during this period were either made private/unlisted removed, resulting in 400,000 images and 80,000 videos, accompanied by 1.5 million and 312,000 comments, respectively. These comprehensive filtering steps ensured the construction of diverse and relevant dataset for fine-tuning instruction-basee models.
1.2
DATA FROM YOUTUBE
Our data collection from YouTube begins with querying Wikidata (Vrandečić & Krötzsch, 2014) for YouTube IDs to compile list of channels. Wikidata, derived from Wikipedia, provides curated selection of renowned channels, automatically filtering out noisy videos commonly found in datasets collected from diverse sources like user-gene videos. This initial step yielded dataset of 2.2
5

---

Published as a conference paper at ICLR 2025
million videos spanning the period from 2018 to 2023, sourced from approximately 6,000 channels collected from Wikidata.
To refine the dataset, manual filtering was employed exclude certain categories deemed less relevant for our purposes. These categories included music and songs, gaming content, non- Englisnl videos, sports commentary, anime, memes, channels with disabled comments sections, and news-related content. Furthermore, and only videos with a substantial viewership, defined as greater than 10,000 views, were retained. We observed that these videos usually have less noisy comments and likes.
Subsequently, the top comments from each video, as ordered by Youeube (i.e., the most liked comments), were selected for inclusion in the dataset. To address redundancy in comments, TF-IDF filter was applied with a threshold of 0.7, which proved effective in removing duplicate comments prevalent in YouTube data alongside llama-guard-3-1 (Llama Team, 2024) filters.
Comments were further filtered to include only those with minimum of four words and a maximum of 100 words, ensuring a balance between relevance and conciseness. Additionally, to mitigate the presence of NSFW content, a vocabulary specific to NSFW terms (Idn) was employed to filter out inappropriate posts. On average, we finally get 3.1 comments per video, providing a substantial corpus of user-generated content for analysis. After applying these filtering steps, the dataset was reduced to 250,000 videos, ensuring a curated and relevant collection for subsequent analysis and model training.
2.2 INSTRUCTION FINE-TUNING LLAMA-VID
After collecting Reddit and YouTube media and user behavior, we formulate instruction fine-tuning tasks for training LLMMA-Vid. In the training instruction, given the media content and automatic speech recognition if available, we ask the model to simulate the scene-bys scene-by-scem. by-neene description and user likes/views and top-5 comments. This instruction training template is given in Listing 1. To generate the instruction data, first, frames are sampled using the 30-degree rule (SI et al., 2023; Arev et al., 2014; Friedman & Feldman, 2004), then the scene-by-scene description are obtained by concatenating automatically generated captions and tags from LLaVA-1 1BB (Liu et al., 2023a), colors and tone through Qin et al. (2020). This instruction format keeps the instructions similar to the instruction format for other VLMs like LLaVA (Liu et al., 2023a), MiniGPT- (Zhu et al., 2023), BLIP (Li et al., 2022), LLaMA- Vid (Li et al., 2023b), etc., while additionally teaching the model to learn behavior. We keep the instruction fine-tuning template similar for both YouTTub and Reddit. The complete instruction given in Listing 1.
We start with the trained LLaMA- model. The LLaMA- Vid model uses two tokens to represent each frame in the video, which they call content and context tokens. While the context token encodes the overall image context based on user input, the content token encapsulates visual cues in each frame. For learning context tokens, the model uses attention queries that interact with previously generated image features in the designed attention module. To generate content tokens, the image features are average pooled. This dual-token strategy significantly reduces the number of tokens needed represent videos, thus enabling the model scale to longer (hour-long) videos. To better support hour-long videos, LLaMA- Vid was trained on a 9k movie-level conversation instruction set containing plot reasoning and detail understanding questions.
Taking the base LLaMA-Vid model, we finetune it further on behavioral data. Notably, our focus is different from research works such as (Xie et al., 2024; Li et al., 2023c; Chen al., 2023b) which try to design mechanisms for LLM and VLMs to learn specific behaviors like trust and coordination in game-like like synthetic environments. Rather, we teach the VLMs to learn real human behavior and show that it improves downstream performance on content understanding tasks. Therefore, we teach the model that given a media file (image or video), has to simulate the likes and comments on the media (Listing l). In our experiments we observe that the sampling ratio of BLIFT and IFT datasets is an important hyerparrameter. We track 4 zero-shot metrics, likes/views, comments perplexity, and empirically find the best results with 1:1 ratio for 2.2 epochs (see Fig 18 and Table 11). For the best checkpoint, the perplexity on comments reduces from 6.22 to 3.05, and the R2 on likes/views goes from -5,1 to 0.45. All experimental hyperparameters are provided in Section E.
We combine 730k instruction pairs from BLIFT with the original instruction tuning dataset consisting of 40K text conversations from ShareGPT, 625K single multi-turn visual QA pairs, and 98K video

---

Published as a conference paper at ICLR 2025
Model
socne
way speaking
relationship
like rutio
view count
director
genre
wriler
year
Video41%%-GP1133555 generated story
Plan
60.2
39.07
64.1
0.061
12.84
69.9
58.1
52.4
75.6
Vibo44166-CP1--55 generated story
clessifer
32.95
75.26
50.84
32.16
LLaMA-Vid GPT-35 Geserated Stocy
58.12
35.5
60.6
0.314
10.34
66.
49.77
34.23
72.12
Ad-LLaVA
59.05
37.07
61.2
0,319
10.37
66.84
55.13
35.33
77.34
Rehavior LLMVA GPE35 Genernted Story
66.43
41.03
64.21
0.17
5.12
71.12
63.45
39.4
79.3
Improvement
LLaMA-Vid
LLaVA
10.48%
8.855
27.495
15.19
Table 1: Comparison of various models on the Long Video Understanding benchmark (Wu & Krahenbuhl, 2021) consisting of 9 VQA tasks. We that Behavior-LLaVA improves on LLaMA-Viss 9/9 tasks with average improvement of 21.49%. Further, it outperforms the state-of offtth--artttt 5/9 tasks.
QA pairs; all the modules except the Visual Encoder kept frozen. We ablate on multiple sampling ratios from BLIFT. We train the LLaMA- Vid checkpoints with their original SFT mix along with BLIFT. We ablate different sampling ratios and found 1:1 be empirically performing the best. We train the model for 2.2 epochs, keeping track of the (-shot evaluation metrics and perplexity on comments in the eval set Fig 18 and Table show the ablations on different sampling ratios and epochs of training. For the best checkpoint, the perplexity on comments reduces from 6.22 to 3.05, and the R2 on likes/views goes from -5.1 to 0.45.
AdLLa VA to show the impact of behavior data: To disentangle the effect of training on additional data samples from the effect of training on behavioral data, we train LLaMA- Mid on BLIFT with the video and image verbalization and do not include receiver behavior. Then, the overall instruction template consists of scene-by scene-by-scene by-seene automatically generated verbalization similar to Listing without the likes and comment simulation. We call the LLaMA-Vid fine-tuned this data, Ad-LLaVA. We compare Behavior- VA with Ad-LLaVA and LLaMA- -id along with other state-of-the-art literature benchmarks on various tasks (Tables 1-3).
We use various filtering steps in our data pipeline, including NSFW filters, time filters, bot filters, and gaming and news video filters. To quantify the impact of our filtering steps on the final performance, we compare the performance from training on unfiltered data with filtered data (Figure 18). The figure shows the benefit of our data filtering process. While training on unfiltered BLIFT also improves the result over baseline performance of Llama-Vid but data filtering adds more improvement on top of
Impact of filtering steps on performance:
Ablation with perceptual behavior: As an ablation experiment, we also try teaching the Behavior- LLaVA perceptual signals. For this, we take the largest perception signal dataset in the literature Salicon 10k (Jiang et al., 2015). It consists of 10,000 MS COCO images (Chen et al., 2015) with free-viewing eye gaze data collected through novel mouse- bosse interface. The dataset has been widely used in many studies. We formulate two tasks using this data, (1) Salicon [Object]: estimating the saliency over the objects in the image and (2) Salicon [Region]: estimating the saliency over a region, where the regions tiles formed by breaking the image into 3x3 grid. For both tasks try to model two objectives, ranking and predicting, we found ranking to be much more effective. The instruction are given in Listings 6 and 7.
3 RESULTS AND DISCUSSION
In the experimental results, we aim to showcase the diverse and emergent capabilities our Behavior- LLaVA model through quantitative numbers various tasks and qualitative examples. These abilities include generating detailed image and video descriptions, emotion and sentiment analysis, question answering, video understanding tasks like scene and action detection. Additionally, we present the ability of Sehavior-LLaVa to transfer learn on other behaviors like memorability of a video both short-term and long-term.
3.1 EVALUATION
To test the effectiveness of Behavior-LLava conduct experiments involving 46 distinct tasks across 26 benchmark datasets. The diversity of tasks and datasets allows us to evaluate the perfor- mance and capabilities of Behavior-LLLaA thoroughly. Each of them is covered briefly next:
7

---

Published as a conference paper at ICLR 2025
1. Visual Question Answering (VQA): We evaluate the performance of visual question answering on the following benchmark datasets:
The Long-Video Understanding (LVU) benchmark by Wu & Krahenbuhl (2021) comprises nine distinct tasks aimed assessing long video comprehension, incorporating over 1000 hours of video content. These tasks encompass diverse aspects such as content understanding (including relationship, speaking style, scene/place), prediction of user engagement (YouTube like ratio, YouTube popularity), and movie metadata (director, genre, writer, movie release year).
The Holistic Video Understanding (HVU) dataset by Diba et al. (2020) stands as the largest dataset for long video comprehension, comprising 572,000 samples. Encompassing broad spectrum of semantic elements within videos, HVU tasks involve the classification of scenes, objects, actions, events, attributes, and concepts. Performance evaluation on HVU tasks is conducted using the mean average precision (mAP) metric on the validation set.
We also use MSVD-QA, MSRVTT-QA (Chen & Dolan, 201 1; Xu et al., 2016b), and ActivityNet- QA (Caba Heilbron et al., 2015) datasets. Their description is given in Appendix D.
2. Video and Image Understanding Benchmarks: We use a wide variety of tasks to evaluate video and image understanding: topic, emotion, and persuasion strategy classification, action and reason retrieval and generation, and emotions. We briefly introduce the benchmarks:
The advertisements dataset by Hussain et al. (2017) contains 3,477 video advertisements and the corresponding annotations for emotion and topic tags and action-reason statements for each video. There are a total of 38 topics and 30 unique emotion tags per video. Further, we have 5 action-reason statements for each video for the action-reason reason generation task. Persuasion strategy dataset (Bhattacharyya et al., 2023) is dataset consisting of 1002 video advertisements from popular brands and their persuasion strategy labels like social identity, anchoring and comparison, reciprocity, foot-in-the-door, etc.
For emotion analysis, we use VideoEmotion-8 (Asur& & Huberman, 2010), Ekman-6 (Xu et al., 2016 CAER (Lee et al., 2019), IAPSa (Mikels et al., 2005), Emotion6 (Peng et al., 2015), EmoSet (Yang et al., 2023), and Abstract (Machajdik & Hanbury, 2010) datasets. A brief description for each of them is given in Appendix D.
3. Image Dense Captioning: Literature image captioning datasets such MS-COCO (Chen et al., 2015) reduce the inherently rich information and fine-grained semantics to simplistic captions, with very brief statements focussing only on salient objects. Behavior data such as user comments help a model learn much more information such as object and material properties, world knowledge, emotion, character understanding, spatial relationships, aesthetics, etc. (see Fig. 1), enhancing the model's 's captioning capability. Therefore, we design a captioning task to test this capability and compare it with respect to LLaMA-Vid and LLaVA-34B (a 2.5x larger model). Since we do not have ground truth for this task, following the LLM-as-a-juee for all the models. GPT-4V is asked to evaluate the dense captions on three metrics: Correctness (Listing 2) evaluating the factuality and model hallucinations, Detail (Listing 3) evaluating the number and depth of details captured by the generated captions, and Quality (Listing 4) measuring the subjective quality of the concepts chosen to be highlighted by the captioning model and the arrangement, coherence, and the linking of various concepts.
paradigm, we use GPT-4V as the judge
4. Image and Video Memorability Simulation: Behavior-LLa'a the media. To check if training on behavior helps in solving other behavior tasks (Khandelwal et al., 2024), we test it over image and video memorability simulation. For this, we select seven benchmark datasets covering long-term and short-term memorability over images and videos: LaMem (Khosla et al., 2015), SUN (Isola et al., 2011), and MemCat (Goetschalekx & Wagemans, 2019) for images and Memento OKeNenmnn et al., 2020), VideoMem (Cohendet et al., 2019), MediaEval (Kiziltepe et al., 2021), and LAMBDA (I et al., 2024) for videos. We briefly cover each of them Appendix D. 5. Modalities other than videos and images: Rehavior-LLave fine-tuned using BLIFT, is pretrained and fine-tuned on image and video datasets. To test if behavior data can improve the results on other modalities as well, we test Behavior-LLaVA performance on two tasks across audio and text modalities (Table 12). For audio, we evaluate on the audio summarization task (Han et al., 2023) and for text, evaluate on the IMDB sentiment benchmark (Maas et al., 2011).
is trained on behavior along with
built on top of LLaMA- Vid and
For Tables 1, 5, and we follow the evaluation protocol of Video-4096 (Bhattacharyya et al., 2023), for Table 6 we follow the evaluation protocol of LLaVA and LLaMA-VIL For Tables 2, 3, and 7, for O-shot evaluation results, we use the logits of the next token from the given task vocabulary. For Table 3, we use the evaluation protocol of SI et al. (2023).
8

---

Published as a conference paper at ICLR 2025
Training
Dataset
Video Emotion-8
CAER
Ekman-6
Random
Random
12.5
14.28
16.67
O-Shot
LLaMA- Vid
29.7
27.2
37.33
Behavior-LLav,
41.35
51.0
49.33
AdA
29.8
27.3
37.66
Improvement of Behavior-LLaVA over LLaMA- Vid
39.22%
84.19%
32.14%
Finetuned
Zhao et al. (2020)
54,5
78.3
55.3
Zhang et. (2023c)
57.3
80.1
58.2
eMOTIONS eWu 2023)
53.12
Arevalo et al. (2017)
$3.7
77.3
54.2
ei al. (2020)
53.3
57.3
Xu cl. (2016a)
52.6
77.9
55.6
ILaMA-Vid
53.8
75.6
Ad-LLaVA
54.1
76.1
57.8
Behuvior-LLavA
56.9
79.3
58.4
Improvement of Behavior- VÁ over LLaMA- Vid
5.70%
3.57%
0.80%
Table 2: Comparison of various models on three video emotion understanding benchmarks (Video Emotion8 (Jiang et al., 2014), CAER (Lee et al., 2019), Ekman-6 (Xu et al., 2016a)). The main goal of comparing on these benchmarks is to demonstrate Behavior-LLa VA's understanding of complex tasks like video emotions of long-form vidcos. We see that Behavior-LJ LLVVA improves on LLaMA-VEL Vid on 3/3 benchmarks with average improvement score of 51 51.85% in zero-shot and 3.39% in fine-tuned settings. Further, it outperforms the current statc-of-the-ane on 3/3 benchmarks in zero-shot and 1/3 in fine-tuned settings.
Training
Models
Lamem
Imsaee Daitasets Memcat
SUN
Memento10k
Datasets VideoMem
MediaEval LAMBDA
Hummn Consusiency
0.68
0.78
0.75
0.73
0.61
0.61
Finetumed
10-shot coniest GPT:33
0.29
0.18
0.15
0.07
0106
0,06
0.0b
ViIMem (Hagen Espesett, 20231
0.63
0.51
0.08
Henry
2004)
Henuy
2024)
Henry trained with 75% duta (1 al., 2024)
0.71
0.75
0.73
0.74
0.63
0.49
0:47
Henry trained all (combinned dasasets al. 2024)
0.72
0.79
0.76
0.72
0.00
0.48
0.52
Ad-LTaVA trained with 50% dain
0.67
0.85
0.66
0.69
0.36
BA3
D:47
Debaviour LLaVA trained with 25% data
0.67
0.72
0.69
0.68
0.53
0:44
0.50
Belaviour LLaVA trained with 50S data
0.72
0.77
0.73
0,71
0.39
0:46
0.51
Behaviour LLaVA trained with 75% data
0.27
0.70
0.60
0.47
Rehavior-1ILA miined datasets
Improvementt Bebanioe-LLVV
daia)
19,57'
9.88%
37.5%
Improvement
Bebavioe-LLLVAA over ML-d (50% data)
10.77%
13.26%
8.96%
2.00%
7.27%
4.54%
27.5
O-shet
LLaMA-
0.13
0.11
0.05
0.03
0.05
0.02
0.05
As-LLaVA
0.14
0.13
0.06
0.06
0.07
0.04
0.13
Belavior-LLaVA
0.21
0.17
0.13
0.12
0.08
0,077
0.16
Imprrvvment of Behavior-LLLaA over LLaMA-Vid
61.58
$4.5%
160%
300%
160%
3509
219%
Table 3: Comparison of various models on seven video and image memorability benchmarks (Memento10k (New-
man et al., 2020). VideoMem (Cohendet et al., 2019), LaMem (Khosla et al., 2015), SUN (Isola et al., 2011) MemCat (Goetschalekx & Wagemans, 2019), MediaEval (Kiziltepe et al., 2021), LAMBDA (I el., 2024)). The main goal of comparing and high-level tasks like memorability simulation. We see that Behavior- LLaVA improves on LLaMA- M-Viiiiddd on 7/7 benchmarks with an average improvement score of 186.4% in zero-shot and 39% fine-tuned settings after seeing 25% train data. Further, it performs similarly to the current state-of-the on 7/7 benchmarks in the fine-tuned settings while still seeing only 25% data.
these benchmarks to demonstrate Behavior-LLaVe VA'ss's understanding of complex
3.2 DISCUSSION
Table 13 shows the average O-Shot performance improvement of Bchaviour-LLaVA over Llama- Vid across all tasks. Tables 1, 6, and 9, contain the results for the visual question answering tasks, Tables 2, 5, contain the results for video and image understanding tasks, Tables 8 contains the results for dense-captioning. Table 3 contains the results for image and video memorability benchmarks, and Table 12 contains the results for the audio and text tasks. From the results, we observe the following trends:
Behavioral data along with content data causes improvement in content understanding: A common trend we observe across all the experiments is that Behavior-LLaVA performs better than the base model LLaMA- Vid and the finetuned model Ad-LLaVA on all tasks, especially in zero-shot settings. In fact, Ad-LLa VA performs very similar to LLaMA- Vid itself. This shows that BLIFT adds meaningful signals on average (rather than noise) to the model. Interestingly, the performance gains remain even after fine-tuning on the task dataset (Tables 2, 5, 7).

---

Published as a conference paper at ICLR 2025
Model
Summarization Shot)
Moment Retrieval mAP (avg)
Highlight Detection mAP
BLEU
ROUGE
METEOR
Rehavioo-LLaaVAARppllyyCCC-
11.2
19.1
25.3
35.2
34,9
11.9
20.3
26.9
34.9
33.1
Behavior-LLaVA
10.6
18.3
22.7
33.1
32.7
LLaMA-VID Video-ChatGPT
9.1
16.5
20.3
30.3
32,4
5.0
14.0
19.7
Table 4: Improvement downstream content understanding tasks by introducing more behaviour signals. Brackets [] denote the new behaviour that we include. Replay graphs (Khandelwal et al., 2024). Nem-Recalls et al., 2024) Evaluation done on Multi-shot video summarization (Han et al., 2023) and MomentDETR (Lei et al. 2021)
Behavior data leads to more improvements higher-level tasks: The performance gains are relatively smaller for low-level tasks of action and object recognition (Tables 6 and 9), but much higher for the more high-level tasks of emotion understanding, sentiment analysis, persuasion strategy classification, and memorability simulation, long form-vide understanding and other sub-tasks of Table 9. This indicates that receiver behavior has richer signals for higher-level tasks; in fact, fine-tuned Behaviour-LJ LA models outperform GPT4-V in image emotion recognition. The gains observed across both image and video benchmarks.
Video verbalization leads to more improvement than just using video:
We also observe that classification using a story generated by GPT-3.5 (following Bhattacharyya et al. (2023)) results in better performance than only using the video (Tables 5, 9).
Improvement in content understanding capabilities generalize other modalities: To eval- uate the generalizability of behavior data across modalities, we extend our evaluation to audio summarization and text sentiment analysis and observe improvements of 19.5% (Table 12).
Behavior data from BLIFT provides insights into elements like style, topic, emotional content, themes, aesthetics, objects, and other descriptive details (Figs 11,12): Figures 6, 7, 10, and 5, 8, 9, show several randomly sampled qualitative examples for dense captions generated by Behavior- LLaVA over images and videos respectively. It can be noticed that despite not being explicitly trained for the task of caption generation, the model performs quite well, picking up various artistic, cognitive, and object and material properties. Table 8 contains the results of the quantitative evaluation for the task of dense captioning. The table shows that while Behavior-LLa VA shows decrease in correctness over LLaMA- Vid, it shows significant improvement in other aspects, including detail and quality. On these aspects, it even comes close to 2.5X larger models (LL. (34B)).
Comparison of improvements observed from perception and action as behavior: Next, in Table 10, we compare the signals from behavioral data of perception and action. For this, we compare Behavior-LLaVA trained on BLIFT and Behaviour-LLLaVA trained on Salicon salient regions and objects. Further, within BLIFT, we compare the performance from predicting singled out behaviours including likes/views, titles, comments. It can be noted that training just Salicon results in performance decrease for the lower-level task of action recognition (MSRVTT-QA) but improves on the higher-level task of Emotion recognition. However, the gains are smaller than those observed with training on BLIFT.
4 CONCLUSION
In this paper, we explore the idea of learning behavior leading to learning content better. Humans produce behavior in response content. Hence, logically, behavior should contain signals about content, which, used as a training task, should help in learning content better. We follow this line of thought and show that training large vision and language models on user behavior data of comments and likes collected from Reddit and YouTube leads to performance improvements across a wide variety of tasks. The gains are higher on higher-level tasks such as emotion recognition, persuasion strategy classification, and question answering and smaller on lower-level tasks like action and object recognition. Further, the gains remain even after tne-ttnnig the VLMs on those benchmarks, thus demonstrating the importance of learning behavior in understanding content better.
5
ACKNOWLEDGEMENT
Rajiv Ratn Shah is partly supported by the Infosys Center for Al and the Center of Design and New Media HIT Delhi.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
List of dirty, naughty, obscene, and otherwise bad words. https://gitthhn wtt. -iitt-ff-Diry-Naughhle Obsc-ne--ceeeeee wwwpeeeeeennd-ceeeeeeeee-eaa--ee 13th April, 2024.
Reddit pics: Rules. http:///s//....eeeccccoco//o///a 2024a. Accessed: April 11,
2024. 4
Reddit pics: Wiki index. April 11, 2024. 4
Reddit videos: Rules. http:s//eeerrrrddd..ccooor/////t 2024b. Accessed: 11, 2024. 4
2024c. Accessed: April
Ido Arev, Hyun Soo Park, Yaser Sheikh, Jessica Hodgins, and Ariel Shamir. Automatic editing of footage from multiple social cameras. ACM Transactions on Graphics (TOG), 33(4):1-11, 2014.
John Arevalo, Thamar Solorio, Manuel Montes y Gómez, and Fabio A. González. Gated multimodal units for information fusion, 2017. 9
Sitaram Asur and Bernardo A Huberman. Predicting the future with social media. In 2010 JEEE/WIC/ACM international conference on web intelligence and intelligent agent technology. volume 1, pp. 492 499. IEEE, 2010. 8, 30
Maria Barrett, Joachim Bingel, Frank Keller, and Anders Søgaard. Weakly supervised part-of-speech tagging using eye-tracking data. In ACL (Volume 2: Short Pupers). ACL, August 201 6a. doi: 10.18653/v1/P16 2094. URL httpss:/aclaathol....
org/p16-2094 2094-
Maria Barrett, Frank Keller, and Anders Søgaard. Cross- transfer of correlations between parts of speech and gaze features. In Proceedings of COLING 2016, the 26th International Conference Computational Linguisties: Technical Pupers, pp. 1330-1339, Osaka, Japan, December 6b. The COLING 2016 Organizing Committee. URL https: htt.tttsss///caatttiin /ww.aathoaaaaa.y.....oo. wtw.eeannnnooooooooooocceee wrgg///-//111 6-126..
Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. Sequence classification with human attention. In CNLL. Association for Computational Linguisties, October 2018. URL https::s///ee h/a.aattoo///ll org/ 81 1030. 2
Bennett Bertenthal. Origins and early development of perception, action, and representation. Annual review of p 47(1):431-453 1996. 2
Yevgeni Berzak, Chie Nakamura, Amelia Smith, Emily Weng, Boris Katz, Suzanne Flymn, and Roger Levy. Celer: A 133366-ptttt 365-paatit corpus of eye movements in and 12 english reading. Open Mind, 2022. 2
Aanisha Bhattacharyya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn Shah, and Changyou Chen. A video is worth 4096 tokens: Verbalize videos to understand them in zero shot. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9822-9839 Singapore. December 2023. Association for Computational Linguistics. doi: 10. 1006:/3////2223.mmmmmmaaa 8. 10, 25
URL httpssss//acaaanhoooog/// org/ 2023, emnlp-main. 608.
Klinton Bicknell and Roger Levy. Why readers regress to previous words: statistical analysis. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011. 2
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arXiv preprint
arXiv:2201.07311, 2022. 2
Breakthrough. Pyscenedetect: Video scene detection and analysis tool, 2023. URL https:
//github. http:/htt..om///raaaaaaaaaaaaaae
31
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015. 8, 30

---

Published as a conference paper at ICLR 2025
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th anmual meeting of the association for computational linguistics: human language technologies, pp. 190-200, 2011. 30
Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang, and Jing Liu. Valor: Vision- audion-uddio-aae -adio-langagge unni-perception pretraining model and dataset, 2023a. URL pttts://arke wrg.obs/
abs 2204. 08345. 27
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors, 2023b. URL pattts://areee wwg. 2308. 108488..888....... 6
Xinlei Chen, Hao Fang. Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00335,, 2015. 7,8
Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE transactions on pattern analysis and machine intelligence, 37 (3):569-582, 2014. 2
Romain Cohendet, Claire-Hélène Demarty, Ngoc QK Duong, and Martin Engilberge. Videomem: Constructing, analyzing, predicting short-term and long-term video memorability, Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2531-2540, 2019. 8, 9, 30
Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. Paying more attention to saliency: Image captioning with saliency and context attention. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 14(2):1-21, 2018. 2
Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. Observations on typing from 136 million keystrokes. In Proceedings of the 2018 CHI conference on human factors in computing systems, pp. 1-12, 2018. 2
Ali Diba, Mohsen Fayyaz,, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Lue Van Gool. Large scale holistic video understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 590-610. Springer, 2020. 8, 26
Andrius Dzedzickis, Artüras Kaklauskas, and Vytautas Bucinskas. Human emotion recognition:
Review of sensors and methods. Sensors, 20(3):592, 2020. 2
The Economist. Why reddit users are protesting against the site's leadership. The Economist Ex- plains, June 2023. URL https: //wpp::://www.///p economi st, com/ wttp:/eeoeiisseeeepppllaaeeeaa wetpeeepaaai//////eeee weweinnsttte--iitsseseeeeeee
2023/002
Accessed: April
11, 2024.
Shaojing Fan, Zhiqi Shen, Ming Jiang, Bryan L Koenig, Juan Xu, Mohan S Kankanhalli, and Qi Zhao. Emotional attention: A study of image sentiment and visual attention. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 7521-7531, 2018.
Doron Friedman and Yishai A Feldman. Knowledge-basee cinematography and its applications. In ECAI, volume 16. pp. 256. Citeseer, 2004. 6
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. Dialogue response ranking training with large-scale human feedback data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2020. doi: 10. 1111136/1122200.mmmll cammlp-main.28 main.28. URL http: //t:://xxxxo/ www.ddo...oooooooooooooooooooo org/10. 1tt../////...1153////220000 18653/v1/2020. emnlp-main. 28.
Lore Goetschalekx and Johan Wagemans. Memcat: new category- image set quantified memorability. PeerJ, 7:e8169, 2019. 8, 9,30
Thomas Hagen and Thomas Espeseth. Image memorability prediction with vision transformers. arXiv preprint ar110:2001.08647, 2023. 9
12

---

Published as a conference paper at ICLR 2025
Mingfei Han, Xiaojun Chang, Heng Wang, and Linjie Yang. Shot2story20k: A new benchmark for
comprehensive understanding of multi-shot videos. arXiv preprint arXiv:2312.10300, 2023. 8, 10, 27
Alex Hern. Reddit moderator protest spreads across thousands of communities. The
Guardian, December 2023. URL https: /wwww theguardian. com/ http///hhhhoooyl/eeeeeee /2tee///22///// /2223//dc/////
al-ammimmaaa... Accessed: April 11, 2024. 4
Nora Hollenstein and Ce Zhang. Entity recognition at first sight: Improving NER with eye movement information. In NAACL: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. 2
Nora Hollenstein, Maria Barrett, Marius Troendle, Francesco Bigiolli, Nicolas Langer, and Ce Zhang. Advancing nip with cognitive language processing signals. arXiv preprint arXiv: 19X4::904.02662,, 2019.
Zaeem Hussain, Mingda Zhang. Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, and Adriana Kovashka. Automatic understanding image and video advertisements, 2017. 8, 25
Harini S I, Somesh Singh, Yaman K Singla, Aanisha Bhattacharyya, Veeky Baths, Changyou Chen, Rajiv Ratn Shah, and Balaji Krishnamurthy. Long-term ad memorability: Understanding and generating memorable ads, 2024. 8, 9, 10,
Phillip Isola. Jianxiong Xiao, Antonio Torralba, and Aude Oliva. What makes an image memorable? In CVPR 2011, pp. 145-152. IEEE, 2011. 8, 3,
Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1072-1080, 2015. 2,7
Yu-Gang Jiang, Baohan Xu, and Xiangyang Xue. Predicting emotions in user- iser-generated videos. In Proceedings of the AAAI conference on artificial intelligence, volume 28, 2014. 9
Alan Kennedy, Joël Pynte, Wayne S Murray, and Shirley-Anne Paul. Frequency and predictability effects the dundee corpus: An eye movement analysis. Quarterly Journal of Experimental Psychology, 66(3):601-618 2013. 2
Ashmit Khandelwal, Aditya Agrawal, Aanisha Bhattacharyya, Yaman Kumar, Somesh Singh, Uttaran Bhattacharya, Ishita Dasgupta, Stefano Petrangeli, Rajiv Ratn Shah, Changyou Chen, and Balaji Krishnamurthy. Large content and behavior models understand, simulate, and optimize content and behavior. In The Twelfth International Conference on Learning Representations, 2024. URL
https://opent ew. dofrumm?iddrrka/wwwwe 8, 10, 25
Aditya Khosla, Akhil S Raju, Antonio Torralba, and Aude Oliva. Understanding and predicting image memorability at large scale. In Proceedings of the IEEE international conference on computer vision, pp. 2390-29002222 2398. 2015. 9, 30
Varun Khurana, Yaman Kumar, Nora Hollenstein, Rajesh Kumar, and Balaji Krishnamurthy. Syn- thesizing human gaze feedback for improved NLP performance. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1895-1908, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.eadd 1.ainn...9 139, URL. http://///caaahhhooogoooooo orgg/0233 http:/22....cmmaaaaaeeea eacl-main. 139.
Rukiye Savran Kiziltepe, Lorin Sweeney, Mihai Gabriel Constantin, Faiyaz Doctor, Alba Garcia Seco de Herrera, Claire-Helene Demarty, Graham Healy, Bogdan Ionescu, and Alan F. Smeaton. An annotated video dataset for computing video memorability. Data in Brief, 39:107671, dec 2021. doi: 10. 10.1006/j..ibl 202..di..2..11.. 107671. 91
Srinivas SS Kruthiventi, Vennela Gudisa, Jaley H Dholakiya, and R Venkatesh Babu. Saliency unified: A deep architecture for simultaneous eye fixation prediction and salient object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5781-5790, 2016.77 2
13

---

Published as a conference paper at ICLR 2025
Yaman Kumar. Behavior a Modality. PhD thesis, State University of New York at Buffalo, 2024. 1
Yaman Kumar, Rajat Jha, Arunim Gupta, Milan Aggarwal, Aditya Garg, Tushar Malyan, Ayush Bhardwaj. Rajiv Ratn Shah, Balaji Krishnamurthy, and Changyou Chen. Persuasion strategies in advertisements. Proceedings of the AAAI Conference on Artificial Intelligence, 2023. 25
Harold D Lasswell. The structure and function of communication in society. The communication of ideas, 37(1):13-139, 1948.
Harold D Lasswell. Propaganda technique in world war I. MIT press, 1971.
Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10143-10152, 2019. 8, 9,
Jie Lei, Tamara L. Berg, and Mohit Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language queries, 2021. URL hwwph:tt///// /ttp///www..gaas////////////a//a /ttp///b///177.0000 09609. 10
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image
pre-training for unified lision-language understanding and generation. In ICML, 2022.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding, 2023a. 25
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311 1. 1043, 2023b. 3, 6
Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behaviors for Ilm-based task-oriented coordination via collaborative generative agents, 2023c. URL https: //aaxivv.rg//aaeeeeeeeee /210 06500. 6
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08488: 2023a. 3,
Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas Hi, and Ge Li. One for all: Video conversation is feasible without video instruction tuning. arXiv preprint arXiv:2309. 15785, 2023b. 25
AI Meta Llama Team. The llama 3 family of models. https: //gi thtb. www/metam con/mmetallama// hamplelllaam/bbb/maaaaaaa LaaiLLLama-Gaa Laaallaaaaaaaaaa
.nd 2024. 5,
Andrew I.. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www.aclwedt org/ htt///hthoog//11-1/////1 8,
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05422 2023. 25
Jana Machajdik and Allan Hanbury. Affective image classification using features inspired by psychology and art theory. In Proceedings of the 18th ACM international conference on Multimedia, pp. 83-92, 2010. 8, 26, 30
Joseph A Mikels, Barbara L Fredrickson, Gregory R Larkin, Casey M Lindberg, Sam Maglio, and Patricia A Reuter-Lorenz. Emotional category data on images from the international affective picture system. Behavior research methods, 37:626-636 2005. 8, 26, 30
Anelise Newman, Camilo Fosco, Vincent Casser, Allen Lee, Barry McNamara, and Aude Oliva. Multimodal memorability: Modeling effects of semantics and decay video memorability. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. Springer, 2020. 8, 9, 30
14

---

Published as a conference paper at ICLR 2025
Badri Patro and Vinay P Namboodiri. Differential attention for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7688, 2018. 2
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: outperforming curated corpora with web data, and web data only, arXiv preprint arXiv:2306.01 116, 2023. 2
Kuan-Chuse Peng, Tsuhan Chen, Amir Sadovnik, and Andrew C Gallagher. mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 860-868. 2015. 8, 26, 30
Barbara Plank. Keystroke dynamics signal for shallow syntactic parsing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 609-619, 2016.
Wolfgang Prinz. Perception and action planning. European journal of cognitive psychology, 9(2):
129-154, 1997.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern recognition, 106:107404, 2020.
Haonan Qiu, Liang He, and Feng Wang. Dual focus attention network for video emotion recognition. In 2020 IEEE International Conference Multimedia and Expo (ICME), pp. 1-6. IEEF, 2020. 9
Reddit, Inc. Reddit content policy. https://www. reddi tinc. wttphhc.ccoooooocccccccoo htt.://www.....cc.ooooooooooc/oe 2024. Accessed: April 11, 2024. 4
Claude E. Shannon and Warren Weaver. The mathematical theory of communication. The mathemati- cal theory of communication. University of Illinois Press, Champaign, IL, US, 1949.
Harini SI. Somesh Singh, Yaman Singla, Aanisha Bhattacharyya, Veeky Baths, Changyou Chen, Rajiv Ratn Shah, and Balaji Krishnamurthy. Long-term ad memorability: Understanding and generating memorable ads. arXiv preprint arXiv:2309.00378. 2023.
Ekta Sood, Simon Tannert, Philipp Müller, and Andreas Bulling. Improving natural language processing tasks with human gaze-guided neural attention. NewrIPS, 2020. 2
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. 2022. 25
Denny Vrandečić and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Commun.
ACM, 57(10):78-85 sep 2014. ISSN 0001-0782. doi: 10.11 10.1144//62948 URL https://doi. org/10. 1145/2629489. 5
Ke Wang, Mohit Bansal, and Jan-Michael Frahm. Retweet wars: Tweet popularity prediction via dynamic multimodal regression. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 1842-1851. IEEE, 2018a. 3
Wenguan Wang, Jianbing Shen, Xingping Dong, and Ali Borji. Salient object detection driven by fixation prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1711-1720 2018b.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint 222 2022. 25
Zhilin Wang and Pablo E. Torres. How to be helpful online support forums? In Elizabeth Clark, Faeze Brahman, and Mohit lyyer (eds.), ). Proceedings of the 4th Workshop of Narrative Under- standing (WNU2022), pp. 20-28, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2.33//1/1222///// wnu- 1.3. URL https://aclanthol.oo org/2022 wnu-1.
15

---

Published as a conference paper at ICLR 2025
Zhongyu Wei, Yang Liu, and Li. Is this post persuasive? ranking argumentative comments in online forum. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 195-200, 2016.
Chao- Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1884-1894, 2021. 7,
Xuecheng Wu, Heli Sun, Junxiao Xue, Ruofan Zhai, Xiangyan Kong, Jiayu Nie, and Liang He. emotions: A large-scale dataset for emotion recognition in short videos, 2023. 9
Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye, Shiyang Lai, Kai Shu, Jindong Gu, Adel Bibi, Ziniu Hu, David Jurgens, James Evans, Philip Torr, Bernard Ghanem, and Guohao Li. Can large language model agents simulate human trust behavior?, 2024. URL htttt:://arxiiii org. httrrg/abs/2422.. 04559.
Baohan Xu, Yanwei Fu, Yu-Gang Jiang, Boyang Li, and Leonid Sigal. Heterogeneous knowledge transfer in video emotion recognition, attribution and summarization. IEEE Transactions on Affective Computing, 9()):255-2)) 270, 2016a. 8, 9, 30
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5296, 2016b. 8, 3030
Liwen Xu, Zhengtao Wang, Bin Wu, and Simon Lui. Mdan: Multi-level dependent attention network for visual emotion analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9479-9488, 2022. 26
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 141,,, 2022. 25
Jingyuan Yang, Jie Li, Xiumei Wang, Yuxuan Ding, and Xinbo Gao. Stimuli-aware visual emotion analysis. IEEE Transactions on Image Processing, 30:7432-7445, 2021. 26
Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Emoset: A large-scale visual emotion dataset with rich attributes. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pp. 20383-20394, 2023. 8, 26, 30
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tunel audio-visual language model for video understanding. arXiv preprint Ttt::23000028558 2023a. 25
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learming Representations, 2023b. 25
Zhicheng Zhang, Lijuan Wang. and Jufeng Yang. Weakly supervised video emotion detection and prediction via cross-modal temporal erasing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18888-18899 2023c. 9
Sicheng Zhao, Yunsheng Ma, Yang Gu. Jufeng Yang, Tengfei Xing, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. An end-to-end visual-audio attention network for emotion recognition in user-generated videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020.
pp. 00.3033311 2020.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma. Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for tickkthroouhh rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pp. 1459-1068, 2018. 3
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023. 2,
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A QUALITATIVE EXAMPLES AND DATASET ANALYSIS
The Volkswagen ad titled begins with a group of three women seated excitedly Volkswagen Jetta. the driver sponing wide smile she grips the steering wheel. Thhe voiceover nostalgic likening the Jetin to unforgettable first experiences like first kiss or hearing indic rock for the first time-a symbol of newfourd freedom and excitement.
As the scenes unfold, the camera captures the trio cruising down winding reads, their laughter blending with music the and adventure, with the Jetta serving the backdrop to their shared moments joy and spontaneity. Trunsitioning to wider shot of the Jetta gliding along a scenic highway, surounded by lush greenery. the evokes cruising with vintage footage of classic Volkswagen vehicles, reflecting on the brand's is 75-year history America. starting with beloved Boetle.
tousling
hair. atmosphere inside
camaraderie
exploration and the open road. The visuals seamlessly blend modern day
The ad concludes with the Volkswagen logo and the tagline "An American Love Story,, encupsulating enduring relationship between Volkswaaeen
its drivers across generations. This phrose serves tribute to Volkswagen's year history in America, beginning with the iconic Type I vehicles fondly known as "The Beetle" Through its nosialgic narrative and captivating visuals, the leaser promises viewers immersive journey into the essence of Volkswagen timeless icon that has been part countless cherished memeries on the American road.
Figure 5: Dense caption generated by Behavior- LeaaV for the video of a Volkswagen ad. The original video is posted at URL: https:/ youtube com/watch wwwp//attatvvkkon----- Te. The red-colored text highlights the most important aspects the video captured by Behavior-LLaVA, demonstrating an understanding aesthetics. characters, world knowledge, emotion, and spatial relationships. More such examples are given in Figs. 6, 7, 10, and Figs. 8, 9 for images and vidcos respectively.
In this captivating painting, majestic takes center stage against stunning hackdrop. The sky above is poetic split between night and day deep, starry blue on one side with crescent moon. warm, radiant blend yellows and pinks on the other, suggesting either sunrise sunset. tree itself embodies the essence of transition, with one side adomed in lush green foliage and the other stripped bare, symbolizing
timeless dance of scasons and life's 's perpetual transformations.
This artwork not only captures the but also speaks deeper truths. inviles contemplate the delicate halance between light and darkness, growth and dormancy, woven into the fahric of existence. The artist's brushstrokes, rendered with tranquility and introspection. This painting beautifully reminds us to embrace the flecting beauty of each passing moment, whether bathed daylight or cloaked quiet mystery night.
palette and gentle abstraction, evoke
Figure 6: Dense caption generated by Behavior-LLaVA for an artistic drawing of tree.
17

---

Published as a conference paper at ICLR 2025
The image features pair of Nike Air Force sneakers, which are classic and iconic design. The sneakers are positioned in the center of the image, with the left shoe slightly in front the right one, giving a
depth. They predominantly white with green accents, including the Nike swooch logo on the side and the outsole. The laces are white, and the midsole appears to be a light beige color. The background
plain, light color, which contrasts with the sneakers and makes them Overlaid on the image is stylized text that reads "JUST DO IT" in cursive, flowing script. The text is in a light green color that matches the secents on the sacakers, creating a cohesive color scheme, Below smaller line of lext that says "New collection." indicating that these sneakers
our,
the main lext, there are part of new line or series from Nike.
The overall style of the image clean and modern, with focus the product. The of the "JUST DO IT slogan, which is a well known tagline for Nike, adds layer of branding and recognition to image. The text is designed to be eye-entching and draw attention to the sneakers, which are main subject of the image. The composition balanced, with the sncakers contrally placed and the text evenly distributed around them.
Figure 7: Dense caption generated by Behavior- for a Nike ad. The red-colored text highlights the most important aspects of the vidco captured by Behavior-LLa LLaVA, demonstrating an understanding of aesthetics, characters, world knowledge, emotion, and spatial relationships.
In heart-pounding and visually stunning trailer for Red Dead Redemption. we are thrust into the grint world of the American frontier. The trailer opens with voicoover, chilling warning delivered with calm certainty: "Listen to me, we don't want to kill any of you... But trust me, we will." Scenes flash by in quick succession, each more intense a lone figure, silhouetted against a setting sun, riding magnificent horse through a sprawling, golden field. The rigeed beauty of the landscape contrasts sharply with the impending sense of danger.
last. see
Cut to a dimly lit saloon where a group of hardened men sit around a table, cards in hand, tension thick in the The voicoover continues. "This whole thing is pretly much done. We' 'e more ghosts than people."
flurry of actionn unfolds: quick through the air with deadly precision. The vniceover reminisces, "Good old Dutch my best friend... You know how we met? A pair of hucksters trying rob each other... Back in '78 or thereabouts."
in a darkened nom. bullets slicing
The visuals intensify as we witness robbery in progress, chaos crupting masked figures burst into bank. "Ladies and gentlemen, this a rubbery," " declares one of the oullaws, setting the stuge for a clash between lawlessness and order.
Amidst the chaos, snippets of camaraderie emerge: "Sons of Dutch. Makes us brothers." But looming shadows betrayal and regret cast doubt on these bonds. "Sometines,, brothers make mistakes." acknowledges the voicoover, acknowledging the complexities of loyalty and survival in this unlorgiving world.
The trailer crescendos with a crescendo of gunfights, horsehack chases, and impassioned speeches. "You'll Illl never change. know that," declares a voice, capturing the immutable nature of the human spirit amidst adversity. Throughout, thematic motil resonates: "You have to love yourself lire." The elemental symbolism underscores the primal nature of existence in land where survival demands courage and cunning. the traäler draws a close, we're
with the haunting refrain: "Stay strong. Stay with me." " The screen fides to black, leaving us hungry for more of this epic tale set against With breathtaking visuals and evocative storytelling. this Red Dead Re- demption trailer promises an unforgettable journey through a world where danger lurks around every corner, and the line between hero and outlow blurs in the dust and shadows of the frontier.
backdrop the untamed West.
Figure 8: Dense caption generated by Sahavior-LLaVA for the video of the official trailer of the game Red Dead Redemption 2. The original video is posted at URL: https: / /www. youtube, www/wat tch pyxya
18

---

Published as a conference paper at ICLR 2025
The
packed stadium buzzing with excitement, the air thick with ticipation as Argentina and France face off in what promises to be an opic World Cup final. Lionel Messi steps up confidently the tension palpable as he eyes the goal. The commentator's 's voice echoes through the stadium, 's got the ground, he's 's got penalty! crowd holds its breath Messi strikes the ball cleanly, sending it into the back of the net. The stadium erupis with cheers as Argentina takes the lead, setting the stage for a thrilling encounter.
take crucial penalty early the match,
beats... And Messi!" The
As the game unfolds, both teams showcase their attacking prowess. Argentina's 's Angel Di emerges standout, capitalizing on swift counteratiack score spectacular goal. "Alvarez, here's sere's McAllister, Di Maria is the spare man, and how Di Maria! Glorious goal! Argentine gold!" Argentine fans go wild, waving their flags and chanting passionately. However, France lights back liereely, curning penully of their own which they duly convert.
b WAND
The drama intensifies in the second half as France's Kylian Mbappó dazzles with his speed and skill. scoring a breathtaking equalizer. "Mbappé, to Lamp, Mbappé! Oh wow!" match swings back and the dying moments of extra time, the score is deadlocked at 3-3 when Argenina's goalkeeper makes crucial save, denying France late winner. The match heads to nail biting penalty shootout. Wath nerves steel, Argentina's players step up and convert their penalties flawlesdly culminating in decisive save by the goalkeeper. "Arrcatiia, champions of the world"" The stadium crupts more, Messi lifts the World Cup tropy amidst sea of jubilant fans, marking a historic and unforgettable victory for Argentina
with both teams pushing for victory.
FROP
Figure 9: Dense caption generated by Bchavior-LLaVA for the video of Argentina vs France FIFA World Cup Qatar 2022 Highlights. The original video is posted at URL: https: httpp::/www..ott youtube com/watch?yv zhEWqfP6V..g ww.
The composition of the image presents compelling narrative of soldier immersed in war-tor landscape. Positioned amidst a hackdrop of explosive chaos and dense foliage, the soldier, adorned in traditional military gear, gazes directly at viewer with resolute demeanor. The soldier's 's face cancutlaged, blending seamlessly with the olive-green belmet-4 hallmark of battletield attire. Surrounding this central figure, tableau activity unfolds: tank looms in the background. its burrel skyward, suggesting recent action. Other soldiers, alert and vigilant, navigate the dense jungle terrain, underscoring the high stakes of the conflict. The foliage, lush yet forchoding, heightens the palpable tension seene.
Foregraaned by the silhonette of another soldier's 's helmered head, the viewer is drawn into the heart of the action. evoking seuse of shared experience amid the perils of warfare. The realism of the depiction accentuates the emotional weight of the moment, capturing the essence of buman resilience amidst the ravages of battle.
Overall, the image transcends role in the broader narrative of war-inbued with suspense, authenticity, and a profound exploration of the human condition within conflict.
representation, offering poignant reflection the individual's
Figure 10: Dense caption generated by Behavior- VA for a painting of soldier. The model captures many qualitative aspects that are usually missed in common captioning tasks.
19

---

Published as a conference paper at ICLR 2025
skca
reddit
reddit
Noon raiigg bahind tree
Thlppicuue would ruke preat now porter TiM
[reined)
Viea thit airp alburn cover
reddit
Sloodittoce diant
@eddiit
Bouty
Youw comcerrr
Lppota th roplle
pics
reddit
TooShiltyForlww cormennted on pest
reddit
Poited by maturn72
painted grizzly bear "gone fishin" today.
Lave your distinctive styte of bright colors and ewirls warks great
upvotes 6 reppy
preprandial_ joint romented post
reddit
this style called? St, murats with this exact style and LOVE m!
matten872 reptiidi
not sure what they cell it I'd to see their work cal prycheceiic tolk.
94K spvonss
Comment
upectes repptees
View commmnnts
Figure 11: Qualitative examples from r/pics in the BLIFT dataset, accompanied by their corresponding comments. The behavior from Reddit comments provides insights into various elements, including style, topic, emotional content, conceptual themes, object identification, aesthetic qualities, and descriptive details of the post.
20

---

Published as a conference paper at ICLR 2025
30
SIK PARK
HD
Figure 12: Qualitative example from YouTube along with its comments. The behavior from YouTube comments provides insights into various elements, including style, topic, emotional content, conceptual themes, object identification, aesthetic qualities, and descriptive details.
21

---

Published as a conference paper at ICLR 2025
Pepeerll
Fioe andia
Cenvres
Hal tikac must codla
oabacy aboutaat
Uhe the room
Reweder
Image
Figure 13: diagrammatic representation of the process followed process reddit data. Data collected from r/pics and r/videos and filtered ensure quality, excluding older, NSFW, duplicate, and irrelevant content. The final dataset comprises 400,000 images, 80,000 videos, and 1.8M+ comments, focusing on Reddit-hosted and YouTube videos. After filtering the data, we process the collected images and videos by generating captions and keywords for the images. For videos, segment them into scenes, generate captions and keywords for each scene, and produce ASR transcripts for the entire video. Finally, combine this information with post comments to create final prompts.
Tred Iervee
Wetther
Figure 14: understand topic distribution images BLIFT dataset, we extracted topics from image titles and captions using BERTopic. Further these topics were clustered and assigned a name by 4o---oiiii.. This figure shows that the BLIFT dataset covers a large and diverse types of image topics and themes.
22

---

Published as a conference paper at ICLR 2025
Conmonniatton
Nature
Gobee Rosdoonsio
deatoral & Themes
Rooee Enviromments
Peppie Empliion
Arg
saace
Comprettins dias Evente
Ehuneee suenort Exotrreten
koowlodae
Culturll Events
a wose cappee
Plrtform
Di Hoboles
IISNA for
Vouu 4co
Tou Irentt
Glste pisfirrm
phons
Figure 15: analyze the topic distribution of receiver feedback we extract topics from the comments all images using BERTopic. Further these topics were clustered and assigned a name by GPT-4o-mini. This figure shows that the BLIFT dataset covers complex and diverse types of responses including emotions, art styles, themes, coneepts and aesthetics.
Eara
Figure understand the topic distribution of the videos the BLIFT dataset, extracted the topics from video titles, description, seene captions and keywords using BERTopic. Further these topics were clustered and assigned a name by GPT-4o-mini. This figure shows that the BLIFT dataset covers large and diverse types of video topics and themes.
23

---

Published as a conference paper at ICLR 2025
OF
YOLR
Figure 17: To analyze the topic distribution of receiver feedback we extract topics from the comments of all Youtube and Reddit videos using BERTopic. Further these topics were clustered and assigned a name by GPT-4o-mini. This figure shows that the BLIFT dataset covers complex and diverse types of responses including emotions, art styles, themes, concepts and aesthetics.
24

---

Published as a conference paper at ICLR 2025
B QUANTITATIVE RESULTS
Training
Toplc
VideCharttLeen 21000
Generated Story GPT35 (ibamtcharrys 2003
Chasther el 2003)
(fiuutaccaryy eta.
22.22
Video4196
fihentschhnyyeet 2001)
Vileo4196
Fan-45-al Chasifier af
$7.38
Vilen4096
Chasilfer (Hinttacaaryy.ttcl
11.25
20.72
LCBM (Khande/wal
42.17
7.08
39.55
27.91
LLMA- VID video
10.11
3.42
5.25
12.32
29,61
HaMA-VID wi viico GPT33
42.22
11:05
64.00
$2,00
$2,76
42.33
Rotoone- L
22.55
43.56
Rehowice-L VA vilaa variulizaiion
46.34
11.7
64.13
53.33
$3.06
$2.03
AdLLaAA vilaa SPil-
51.16
11.33
68.00
53.11
43.26
51.45
Saaaiin--LLVAAa vallo GPP:33
60.09
12.84
79.94
36.12
67.30
79.18
Imprevement of Bchavioe-LLaVA oner LLaMA-VM
40.677
16.2%
24,569
12,625
77.75
87.05%
Fiactuned
VideoMAE (Tong 2022)
24.72
20.72
85.55
11:17
Hussain et (2017)
35.1
32.8
48,45
Intem Video (Wang al. 2022)
$7.47
36,08
$6.39
5.47
0.8
Vidbodt99 Generated Story Roberta Classiter (ihhttthaya et 2023)
71.3
33.02
64.20
64.57
42.96
39.09
LLaMA-VID VID vodeo + vorbolizziion
59.13
32.11
79.15
59.93
50.32
30.13
HLMA-VVD of video + GWP-33 Soory
63:11
35.01
64.15
55.01
57.11
45.75
1altorie--.111 97 mnl viltn
18171
225.1
69:41
25.21
99.11
51.35
Rehowize-I. Larriii----aaa LaVA of vilaa sppluliotionn
68:32
33.03
85.09
64.72
70.89
75.34
AJLLJJA wi valaa GPI--3.
66.3
36.24
84.00
58.31
68.15
78.15
Behovit-L_LaVA vilao GPI333
71.2
39.53
86.17
65.00
$0.44
81.67
Imprevement Bchavior Laaaaiio--Laaaa LLaMA-Va
12326 12.079
2.450
154219
40.35%
78.39
Table 5: Comparison of various models two video understanding benchmarks (Hussain et al., 2017; Kumar al., 2023) consisting of tasks related video advertisements understanding. The main goal of comparing on these benchmarks is to demonstrate Behavior-LLava's understanding of complex videos. Behavior- LLaVA improves on LLaMA- Vid on 5/5 tasks with an average improvement score of 43.18% in zero-shot and 27.64% in fine-tuned settings. Further, it outperforms the current state-of oftt--o-----rtt on 3/5 tasks in zero-shot and 5/5 in fine-tuned settings.
see that
MSVD-QA
MSRVTT-QA
ActivityNet-OU
Method
LLM
Acc
Score
Acc
Score
Acc
Score
FrozenBiLM (Yang et al., 2022)
DeBERTa- V2
32.2
16.8
24.7
Vide (Zhang et al., 2023a)
Vicuna-7B
51.6
2.5
29.6
1.8
12.4
1.1
LLaMA-Adapter (Zhang ct al., 2023b)
LLaMA-7B
54.9
3.1
43.8
2.7
34.2
2.7
VideoChat (l.i et al., 2023a)
Vicuna-7B
56.3
2.8
45.0
2.5
26.5
2.2
Video-ChatGPPT (Maaz et al., 2023)
Vicuna-7H
64.9
3.3
49.3
2.8
35.2
2.7
BT-Adapter (Liu et al., 2023b)
Vicuna-7B 7B
67.5
3.7
$7.0
3.2
45.7
3.2
LLaMA-VID
Vicuma-7B
69.7
3.7
S7.7
3.2
47.4
3.3
LLaMA-VID
Vicuna-133 13B
70.0
3.7
58.9
3.3
47.5
3.3
Ad-LLaVA
Vicuna-
70.0
3.7
59.0
3.3
47.4
3.3
Vicuna- 13B
70.1
3.7
59.2
3.4
47.5
3.3
Improvement of Vehavior-LLaVA over LLaMA-Vid
0.14%
0.5%
3%
Table Comparison various models on three conventional video question answering benchmarks consisting of question answers related to action understanding. The main goal of comparing on this benchmark is to show that Behavior-LLas does not perform worse on low-level understanding tasks like action recognition. We see that Behavior LLAVA marginally improves LLaMA Vid 2/3 benchmarks. Further, it performs equivalent the state-of-the-art in 3/3 henchmarks.
25

---

Published as a conference paper at ICLR 2025
Training
Models
Emotions
Random
Randoe
12.5
12.5
16,67
12.5
D-shot
GPT4-V
83.33
71.12
65.47
79.16
LLaMA-Vid
43.41
43.24
40.37
45.23
Ad-LLaVA
43.22
43.01
43.21
44.38
Behavier-LLan
57.97
64.21
49.71
50.38
Improvement
Behavior-L
Finctuned
MIDAN (Ku et al, 2022)
85.96
78:34
61.66
75.75
Stimuli-swaat (Yang at. 2021)
61.62
78.40
finetuned
A VA finetuned
85.13
71:16
62.66
79.88
Behavier-LLaV LLaVA finetuned
87.36
81.41
72.31
83.21
Improvement
LLMMA-VI
14.29%
Table 7: Comparison of various models on four image emotion understanding benchmarks (IAPSa-8 (Mikels al., 2005) Abstract (Machajdik & Hanbury, 2010), Emotionó (Peng al., 2015), Emoset (Yang al., 2023)). The main goal of comparing these benchmarks is to demonstrate Behavior-LLave understanding of complex tasks like image emotions. We see that Behavior- LLaVA improves LLaMA- average improvement score of 29, 14% in zero-shot and 8.95% in fine-tuned settings. Further, it outperforms the current state-of-the-art on 4/4 benchmarks in the fine-tuned settings.
benchmarks with an
Model
Correctness
Detail
Quality
Average
GPT4-V
8.4
8.5
8.4
8.43
LLaVA-I.6 (34B)
8.1
7.4
7.9
LLaMA-Vid
7.2
Ad-ILaVA (13B)
7,5
7.8
7.3
7,53
Behavior-LLaW (13B)
7.9
7.76
Improvement of Behavlor-LLaaae
LLaMA-Vid
-1.3%
0.37%
9.72%
4.8%
Table 8: Comparison of various models the image dense captioning task. The main goal of this task is demonstrate Behavior-LLaVa. image captioning ability. Despite not being explicitly trained on this task, Behavior- LL performs better than both Ad-LLaVA and LLaMA- -idd Vid on Detail and Quality aspects while losing marginally on correctness. On the aspects of detail and quality, it even outperforms the much larger model of I.L.VA--116 (34B).
Model
Scene
Object
Action
Event
Attribute
Concept
Overall
D-thot
LIIMA-XID
47.25
65.26
78.12
28.03
42.33
50.01
$1.83
Ad-LLaVA
65.35
77.45
31.45
45.33
50.70
$2.91
Behavior
77.95
Improvement
Behavior-LLVVA over LLaMA-Vid
10.12%
0.11%
0.22%
16.52%
7.89%
2.34%
4,46%
B-ahot wi story
Vides21%% -PT-33 genemite story Fla--5-od classifier
39.00
98.89
38.42
67.76
$6.99
75.12
Video4096 GPT-3.5 generated story GPT-3.5 classiferr
60.2
99.16
98.72
67.17
88.6
75.77
LLaMA-VID- Generated Stary
Behwvio--LaVA GPT3.5 geserated story
60.4
99.89
98. 23
40.97
67.23
88.33
75.84
Impeovement of LVV over LLMMA-Vil
0.16%
0.03%
0.788
4.17%
0.33
L147%
0.33%
Table 9: Comparison of various models on the Holistic Video Understanding benchmark (Diba al., 2020)
consisting of VQA tasks. We see that Behavior-LLat improves on LLaMA- -id on tasks with an average improvement of 5.88%. Further, it performs equivalent to the state-of-the-ae art in 6/7 tasks. State of the art is achieved by generating story and asking Behavior-LLaaA answer questions based on the generated story.
Task
MSRVTT-QA
CAER
Emoset
Comments
likes/views
Base
58.9
75.6
45.23
6.22
-0.1
Salicon [Region]
55.6
75.8
47.25
6.25
-0.07
Salicon [Object]
57.9
76.4
48.0
6.12
0.05
Like/esskes/Viees]]
58.2
76.2
47.12
6.15
0.38
BLIFT[Titles
58.4
78.1
48.12
5.09
0.13
[lomen[oomentts]
59.0
79.1
49.58
3.02
0.19
BLIFT
59.2
79.3
50.38
3.05
0.40
Table 10: Ablation on using comments and/or perception signals from Salicon
26

---

Published as a conference paper at ICLR 2025
Sampling Ratio
Epoch
Likes/Views R?
Comments Perplexity
Performance
Base-Model
-0.1
6.22
0.5
0.11
4.71
5,49
0.22
3.95
8.23
1.25
0.33
3.19
10.97
1:1
1.5
0.35
3.13
11.79
0.38
3.08
12.31
0.4
3.05
12.57
0.5
0.14
4.33
3,04
1:2
1.05
0.28
3.66
7,08
1.45
0.42
2.99
8.12
0.15
3.43
1.38
0.8
0.31
2,78
3.44
1:10
0.38
2,46
4.48
1.2
0.46
2.13
5.51
0.5
0.1
5.3
3.52
2:1
0.21
4.6
7.45
1.5
0.31
3.9
9.13
Table Ablation on different sampling ratios epochs training. Sampling ratio ratio of behaviour data to multimodal instruct data. Performance the average increase in 0-shot accuracy on 6 tasks with 250 samples each from the eval set. These tasks include image emotion recognition, video emotion recognition, persuasion strategy classification, MSRVTT, HVU and MSVD-QA
Model
Audio Summarization (3 Shot)
IMDb Sentiment
BLEU
ROUGE
METEOR
0-shot
I-shot
Behaviour-LLaVA
19.0
25.1
39.3
84.1
90.2
LLaMA-VID
15.1
18.3
30.7
80.3
87.9
VALOR (Chen et al,, 2023a)
6.6
10.0
23.9
Improvement of Mehavior-LLLaW Liid
25%
37%
28.01%
4.73%
2.61%
Table Evaluation on audio and text modalities. evaluate on audio summarization benchmark(Han et al., 2023) for audio and IMDB sentiment benchmark for (Mats et al., 2011)
27

---

Published as a conference paper at ICLR 2025
Task
0-Shot improvement over Llama-Vid
LVU
21.49%
Video Ad Understanding
43.18%
Video Emotion
51.85%
Image and Video Memorability
186.4%
Video QA
0.6%
Image Emotion
29.14%
Image Dense Captioning
4.95%
HVU
5,88%
Audio Summarization
30%
Sentiment Analysis
4.73%
Table 13: Average 0-Shot performance improvement Behaviour-LLaaA over Llama- Vid across all tasks including Image, Video, Audio, and Language modalities. The table shows that Behaviour-LLaa improves over its base model on all tasks.
Performnce (%) Llame-Vid
10
Behnvicuninstrul
1:1
-X-
1:2
1:10
6
noise fiterrng
Llama-Vid
2
0.0
0.5
1.0
1.5
2.0
2.5
Epoch
Figure 18: Percentage performance improvement over an untrained LLaMA Vid model, compared across various sampling ratios at different checkpoints. The 1:1 sampling ratio shows the best empirical performance. Performance is averaged over O>shot accuracy improvements on six tasks with 250 samples each from the evaluation These tasks include image emotion recognition, video emotion recognition, and persuasion strategy classification, MSRVTT, HVU and MSVD-QA. The figure also shows benefit of our data filtering process. While training on unfiltered BLIFT also improves the result over baseline performance of Llama- Vid but data filtering adds more improvement
of it.
C LISTINGS
Listing 2: GPT-4V Prompt to calculate correctness of image dense caption
You great critique for analyzing images and captions..
Assess the performance of dense image captioning model based the correctness of the captions generated.
assess correctness
provided caption relation
image. Consider
caption accurately
identifies and describes the main subjects objects depicted in the image. Assess whether the caption coneecty interprets relationships between elements within the image, such as actions. interactions or spatial arrangements. Focus precision and accurccy the information presented the caption. Provide of correctness, raging from (low comeciness ) to 10 (high correctness )
reflecting the level
Listing 3: GPT-4V Prompt calculate detail of image dense caption
great critique analyzing images captions.
performance image captioning model based
captions generated.
28

---

Published as a conference paper at ICLR 2025
Evaluate the level of detail captured in the provided caption. Consider how well the caption descrihes specific attributes
fentures or aspects whether the caption provides comprchensive information about the scenc depicted in the image, covering both prominent and subtle elements. Pay attention the depth and specificity of the details conveyed in the caption. Provide a score indicating richness of detail, ranging from (low detail 10 (high detail ).
image, including colors, shapes, textures sizes, any other relevant details Assess
Listing 4: GPT-4V Prompt calculate quality of a image dense caption
You are a great critique for analyzing images and captions.
Assess the performance of dense image captioning model based on the quality of the captions generated.
Assess whether the caption is concise descriptive providing meaningful and engaging information about the image. Evaluate the caption 's ability to evoke a clear menial image corresponding to the visual content. Additionalyy consider the reflecting the overall quality the caption
cuption is insightful imaginative its description. Provide ranging from (low quality) ) to 10 (high quality). ).
Listing 5: Prompt generate verbalization using Llava-1.6
For given image, For example:
USER:
line caption and maximum 20 desoriptive keywords, than that.
("caption" : "This sample eaption ", "keywords": "keyword. keyword keyword 33"
Answer in JSON format only. Do not include any other information answer.
<image >
ASSISTANT:
Listing 6: Perceptual Signal Instruction fine-tuning template for the image: http::/faant staticflickr..........tttt ctt./////////500000000000 (et.580015.ee055ddee 50.e035dd2d30.z jpg
<SYSTEM> You are an AT visual assistant. Answer all questions as seeing the media<<SSTTEM <USER> The objects
image no particular order are car, dog. friaboc Give the order of saliency of these objects. with the most salient object and end with the least salient object, each in sepurate line. Give the objects only and nothing else.
<image>
<ASSISTANT>
dog fristee
<ASSISTANT>
Listing 7: Perceptual Signal Instruction fine-tuning template for the image: Sttp::/farmee staticflickr cww.//1c1111111 011/5/116/555000 50 e035dd2d30 z. jpg
<SYSTEM>> are an
assistant Answer questions you are seeing media<< SYSTEM> USERRAssume the given image is broken into 3X3 grid the regions or tiles being named "upper-ledl" "upper-center". "upper-righi", "midkle- left "middle-cntter" "middle-right ", "bottom- left "bottom-cent bottem-right" Rank these regions or tiles based on their saliency. give me line separated ranking all regions decreasing
<ASSISTANT>>
middle-right right
bottom-center
bottomm-ighh right
upper-center
upper-right
upper- left
middle-Ieft left
botiom- left
Listing 8: Ad-LLaVA Instruction fine-tuning template for the video: httts:://ww. youtube com wa vc JKPQKjRF4yY
<SVSTEMeYou are an AI visual assistant. You are given detailed description of media, followed by the actual media. Answer all you seeing sedia,
<USER>
The video advert sersent is titted "Gatorade Make Your Own Footsteps wi Sunii Lee" for the brand Gatorade: The audio the ad says "CASR ...]", 4nalyze video deeply, write video, would top-5 popular consents on this video?
description video
following
parcentage vievers would Mhat would the replay graph values for each sceee be? S/vilic..//viddo s/video2
questions.
<videoe...
</USER0
29

---

Published as a conference paper at ICLR 2025
<ASSISTANT>
The scone-by-scene descriptions are:
Scene 1: The
line going around outside of the foreground colors the are black, rud green, gray, dark gray, and the background colors are brown, black, dark gray. The dominant tone of the scene is neutral. This scene is categorized by the togs: cosnetic, eyebrow, face, girl, ponytail, stand, string, woran.
looking into distance with
Scene The
yard, The foreground colers of the gray, olive, and the background colors are black, dark gray, gray, brown. dontnant tone the scane is neutral. This scene 15 categorized by the tags: athletic, balance, bean, car, girl, house exterior, hurdle, 295, legging, plark, rail,
balancing
skateboard
black, rud
dark
seesaN, wonan, yard. gray, dark blue, and background colors are cark brown, dark blue, purple, darkk pink, brown.
Scene
jurping
backyard. The foraground colors of
black, dark gray,
</ASSISTANT>
D DATASET DESCRIPTIONS
1. MSVD-QA and MSRVTT-QA: These datasets are based on Microsoft Research Video Description (Chen & Dolan, 2011) and MSR-VTT corpora (Xu et al., 2016b) and are extensively used in many video captioning and question-answering experiments. The MSVD-QA dataset has total number of 1,970 video clips and 50,505 question-answer pairs. The MSRVTT-QA dataset contains 10K video clips and 243k question-answer pairs.
2. ActivityNet-QA (Caba Heilbron et al., 2015) is benchmark primarily for human activity understanding containing 849 hours video, including 28,000 action instances.
3. VideoEmotion-8 (Asur& & Huberman, 2010) dataset comprises 1,101 user-generated videos sourced from YouTube and Flickr, each containing a minimum of 100 videos per emotional category, as per Plutchik Wheel's emotion model.
4. Ekman-6 (Xu et al., 2016a) dataset is compiled from social websites, with each of its 1,637 videos labeled with single emotion category based on Ekman's 's psychological research.
5. CAER (Lee et al., 2019) dataset, sourced from TV shows, consists of 13,201 clips with an average sequence length of 90, each manually labeled with six basic emotions, aligning with the Ekman-6 dataset.
6. IAPSa (Mikels et al., 2005) is a subset of IAPS, following the Mikels model with eight emotion categories such amusement, awe, contentment, excitement, anger, disgust, fear, and sadness. It consists of 395 affective images, marking the first visual emotion dataset with discrete categories.
7. Emotionó (Peng et al., 2015) features 1, 1,980 images sourced from Flickr, each labeled by 15 annotators according to the Ekman model, covering six emotion categories: happiness, anger, disgust, fear, sadness, and surprise.
8, EmoSet (Yang et al., 2023) encompasses a total of 3.3 million images, including 118,102 from social networks and artistic sources, evenly distributed across various emotion cate- gories. Based on the Mikels model, EmoSet categorized into eight emotion categories.
9. Abstract (Machajdik & Hanbury, 2010) exclusively consists of color and texture combina- tions without recognizable objects. Differing from the IAPS dataset where emotions often stem from identifiable objects, the abstract paintings dataset was peer-rated via web survey, with each image rated approximately 14 times. It comprises 228 images spanning eight categories similar to those in IAPS.
10. Memento I (Kewanm et al., 2020) is a short-term video memorability dataset comprising 10,000 video clips, with 900,000 human memory annotations recorded at various delay intervals. The video clips were, on average, 3s long.
11. VideoMem (Cohendet al., 2019) comprised of 10,000 soundless videos, each lasting 7 seconds, accompanied by memorability scores. Memorability was measured twice: first, shortly after viewing and again 24-72 hours later to capture both short-term and long-term memorability effects.
12. LaMem (Khosla et al., 2015) dataset is a short-term image memorability dataset comprising of 60000 images. The dataset contains scene-centric images, object-centric images and other types such as images of art, images evoking certain emotions, and other user-generated images.
13. SUN (Isola et al., 2011) dataset is a short-term image memorability dataset comprising 2222 images that were sourced from the SUN database.
14. MemCat (Goetschalckx & Wagemans, 2019) dataset is a short-term image memorability dataset comprising 10000 images. It consists of five broader memorability-releaam semantic
30

---

Published as a conference paper at ICLR 2025
categories (animal, sports, food, landscapes, vehicles), with 2K exemplars each, further divided into different subcategories (e.g., bear, pigeon, cat, etc. for animal). The images were sourced from existing image sets: ImageNet, COCO, Open Images Dataset, and SUN. 15. MediaEval (Kiziltepe et al., 2021) utilized publicty available links to short-form video clips, each averaging 6 seconds in duration, with both short-term and long-term memorability scores. Short-term memorability evaluations were conducted on videos viewed within the preceding few minutes, while long-term memorability assessments were based on videos viewed within the previous 24 to 72 hours.
16. LAMBDA at., 2024) is long-term memorability consisting 2205 video ads collected over 1749 participants covering 276 brands. The average video length is 33 seconds and the videos are highly complex, consisting of audio, logos, fast-moving scenes, emotions, etc.
E HYPERPARAMETERS
E.1 TRAINING HYPERPARAMETERS
1. Deepspeed Zero2 with Offload
2. Base Model: -imma-vidd-3b-full-2--------
3. Version: imgsp_vl
4. Vision Tower: LAVIS/eva g-ptt
5. Image Processor: wrocessor/ciippppcch142220
6. Multimodal Projector Type: mip2x-gelu
7. Multimodal Vision Select Layer: -2
8. Disable Use of Start/End Tokens for Images
9. Disable Use of Patch Tokens for Images
10. Image Aspect Ratio: Pad
11. Video Token: 2
12. BERT Type: prentniinppetrann.mm
13. Number of Queries: 32
14. Compression Type: Mean
15. Enable 6 Precision
16. Number of Training Epochs: 2.2
17. Per-Device Training Batch Size:
18. Per-Device Evaluation Batch Size: 4
Learning Rate: 2e-5
20. Weight Decay: 0
21. Warmup Ratio: 0.03
22. Learning Rate Scheduler: Cosine
23. Maximum Sequence Length: 2048
24. Enable Gradient Checkpointing
E.2
VERBALIZATION MODULE
1. Scene Splitting: pyscenedetect (Breakthrough, 2023)
2. ASR
(a) openai/whispper-lrge-33
(b) Batch Size: 200
(c) Chunk Length (s): 30
3. Caption and Keywords: Ilvva-v..6-vicuna-
LIMITATIONS
In this paper, try to show the hypothesis that training the behavior modality improves learning of the content modality. We train the models on comments and likes to show this. We test our models multiple benchmarks and obtain positive results. While try to while results do conclusively show that the hypothesis is true, yet, we can test on more benchmarks covering even tasks. Similarly, can show with multiple models, other than LLaMA-Vid
wide variety of tasks and
31

---

Published as a conference paper at ICLR 2025
G BROADER IMPACTS
Our paper talks about how behavioral training can positively impact content understanding of VLMs. We think this will be useful in various content understanding applications such question answering. captioning, etc.
H ETHICAL IMPLICATIONS
This paper demonstrates that training on behavioral modalities enhances the learning of content modalities. Models trained on user interactions such as comments and likes were tested on multiple benchmarks and yielded positive results. While these findings present exciting opportunities for advancing content understanding in AI systems, they also raise important ethical considerations that must be carefully addressed.
No personally identifiable information (PII) is used improve content understanding. Instead, aggregated behavioral data, including replays, likes, and comments, is utilized, ensuring user privacy. have implemented rigorous anonymization and aggregation techniques to protect individual user identities.
We explicitly acknowledge the inherent biases that may exist data sourced from social media platforms such as Reddit and YouTube. Despite our rigorous data filtering and cleaning processes, we recognize that the dataset may still be subject to demographic skews, self-selection bias, and algorithmic influences. These biases could potentially lead to uneven model performance across different user groups or reinforce existing societal biases. To mitigate these issues, we emphasize the importance of considering these broader implications when applying our model and interpreting its results.
3. We acknowledge the need for greater cultural diversity in dataset. To address this, plan to release our artifacts as open-source and encourage community contributions to incorporate multilingual and multicultural data. This could involve expanding the range of subreddits and YouTube channels included in our dataset, with the aim of capturing more diverse and representative spectrum of receiver behavior across different cultural contexts. By taking this approach, we hope to enhance the ethical considerations and societal impact of our work, providing a more holistic view of behavioral patterns in various cultural settings. However, we also recognize that this approach may introduce new challenges, such as ensuring the quality and reliability of community-contribue data.
ensure the safety and ethical integrity our dataset, we implemented a comprehensive con- tent filtering and monitoring strategy. During data preparation, employed advanced filtering mechanisms including -aaMM---Gard------ for content safety assessment, systematically removing posts, comments, and media flagged with NSFW tags from applied an extensive bad word list to screen potentially offensive content and retained only data explicitly marked safe for use. The filtering process involved multi-stage validation to identify and eliminate potentially harmful or biased content, focusing on maintaining the dataset's 's quality and ethical standards. Our generated responses underwent rigorous safety checks, utilizing automated moderation tools and community-devclope safety tags to minimize the risk of generating inappropri- ate or harmful content. By combining automated filtering, community-sourced moderation tags, and machine learning- based safety assessments, we developed robust approach to content monitoring that prioritizes responsible Al development and user protection.
platforms. We
32


Reviews:
[{"review_id": "s0M5xjlqqg", "reviewer": "Reviewer_VwGN", "summary": "The paper proposes that using user behavior data collected from Reddit and YouTube helps large vision and language models achieve better performance across a wide range of language and visual content understanding tasks. The gains are especially more salient on the higher-level tasks such as emotion recognition, persuasion strategy classification, and question answering than the lower-level tasks like action and object\nrecognition. Furthermore, the ablation study demonstrates the importance of behavior data and instruction fine-tuning on behavior data.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated, highlighting the importance of leveraging user behavior data for content-understanding tasks.\n2. It contributes a large-scale dataset BLIFT, consisting of 400k images and 330k videos, along with their receiver behavior, which is beneficial for research about vision language models.\n3. The proposed training method has gained significant performance on a variety of vision-language tasks, which shows the effectiveness of the method.", "weaknesses": "More details of the training process can be provided for reproducibility, e.g. the average length of the text input, and how to deal with extremely long ASR results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xrqEocAtSL", "reviewer": "Reviewer_6UPg", "summary": "Communication consists of not only the message but also the effects it has on the receiver. However, receiver behavior is commonly ignored when training vision language models. This paper shows that training VLMS on receiver behavior such as likes, comments and replay graphs improves their performance across 46 diverse tasks. Moreover, this data is commonly collected by default (through user interactions on Reddit and YouTube) and hence does not require additional annotation. They collect and release a large scale 730k-sample dataset containing this data.", "questions": "To the best of my knowledge, platforms such as YouTube and Reddit have stringent Terms of Use requiring how such behavioral data can be used and released as a dataset. How did the authors work together with the platform to ensure that collecting and releasing this data does not breach their terms of use?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1.\tThis proposed idea of using receiver feedback relating to images/videos is novel to the best of my knowledge (which should been taken with a pinch of salt since the reviewer mostly work on text-only alignment) and is certainly worthwhile exploring since the cost of using these receiver-behavior is essentially zero compared to how much it would cost for post-collection expert human annotations.\n2.\tThe data collection and preprocessing from Reddit and YouTube are detailedly described and carefully thought-out. I’m especially impressed by the extensive data cleaning done by the authors as well as the multitude of ways that the content behavior data can be converted to instruction tuning data.\n3.\tThe training setup designed by the authors seem reasonable, with well-designed ablations such as matching for additional data (Ad-LLaVA) \n4.\tThe evaluation is done rigorously with 46 different tasks and general improvements across many of these metrics, in relation to strong baselines.", "weaknesses": "1.\tThe introduction claims that “behavior data is considered noise and is ignored while training large language models” (Line 50). However, this statement ignores the substantial work done in harnessing receiver-generated behavior data especially on online forums such as Reddit (through upvotes and comments), which have been used by many works including [1] and [2]. Nonetheless, I have not heard of VLMs using organic/behavioral feedback data so I would suggest the authors to acknowledge that prior work have been done for text-only settings and only claim that this is done in this paper for VLMs, which is more appropriate to the remainder of the paper.\n2.\tThe organization for Section 3 Results and Discussion can be improved. Currently, it’s not easy to follow the main areas of improvement of the proposed methods. Because there are 46 evaluation metrics, it might be easier to follow if the results section starts with one table that shows the average gain in each task category e.g. VQA, which can be done by normalize all scores for a baseline model (e.g. LLaMa-Vid) to 100%, with the raw metrics in appendix. I would also recommend that the discussion is organized more clearly (i.e. each paragraph has one main point summarized as the paragraph heading) as it currently reads as simply describing the results and it’s hard for readers to follow along on the main insights arising from the results. \n\n[1] https://arxiv.org/abs/2009.06978\n[2] https://aclanthology.org/2022.wnu-1.3/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hSq1IrjH6L", "reviewer": "Reviewer_jLqC", "summary": "This paper demonstrates hat training VLMs to predict receiver behaviors, such as likes, comments, and replay graphs, which are available at scale, enhances the VLM’s performance across a broad range of downstream content understanding tasks. Next, this paper constructs a behavior-LLaVA instruction fine-tuning dataset, and obtain the fine-tuned Behavior-LLaVA and AdLLaVA. The experiments on the fine-tuned VLMs demonstrate the effectiveness of content and behaviour, and study the different effect of perception and action behavior.", "questions": "Any analysis on the harmfulness/trustworthiness/bias on the generated content from behavior LLaVa? The collected dataset could contain these harmful content.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The constructed dataset could be very helpful for VLM researches.\n2. The fine-tuned VLMs show improvement across various tasks.", "weaknesses": "1. The idea of the connection between VLM performances and fine-tuning on behavior data, upon which the paper is built, is questionable. The authors try to draw some analogy from relevant studies in human behavior, but fail to make a clear point with the work done in this paper (fine-tuning VLMs on behavior datasets). Also, the logic in this paper lacks clear or formal theoretical analysis on why fine-tuning VLMs on behavior datasets could improve performance, but rather just some abstract references to some concepts in the human behavior research. In this respect, the logic foundation of this paper is questionable and unconvincing.\n2. The study of the connection between LLM/VLM foundation models and human behaviors is not novel. In fact, many researches [1][2][3] have been performed from various different perspectives, such as social behavior and trust behaviors. If we are to view the study on the connection of behavior with VLMs as a contribution in this paper, there are already many researches delivering similar and more comprehensive conclusions on this topic. In this sense, the authors should clearly position this paper along the research line and clearly state the difference between other related works.\n3. The conclusion that learning behavior leading to learning content better is quite trivial and might not be very useful. In fact, adding more fine-tuning data of any types could lead to performance improvement on some relevant tasks. If the authors really want to make this point, they should curate the same dataset without explicitly telling VLMs what the behavior is and explore the performance of the VLM fine-tuned on this dataset.\n\nReferences:\n\n[1] Xie, Chengxing, et al. \"Can Large Language Model Agents Simulate Human Trust Behaviors?.\" arXiv preprint arXiv:2402.04559 (2024).\n\n[2] Li, Yuan, Yixuan Zhang, and Lichao Sun. \"Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents.\" arXiv preprint arXiv:2310.06500 (2023).\n\n[3] Chen, Weize, et al. \"Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors.\" The Twelfth International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XgKCiuAuyq", "reviewer": "Reviewer_Kqw1", "summary": "1. This paper proposes the use of behavioral data training to enhance the content understanding capabilities of multimodal models and has collected a cross-modal dataset called BLIFT.\n\n2. Building on LLaMA-Vid, this paper applies two different instruction fine-tuning strategies, Behavior-LLaVA and Ad-LLaVA, on the BLIFT dataset. The Behavior-LLaVA strategy leverages behavioral data, making it stronger than the Ad-LLaVA strategy, and demonstrates superior performance compared to baseline models across multiple multimodal downstream tasks.", "questions": "1. Did the authors consider the issue of distributional bias during data collection, and how might this impact the downstream tasks? The extent of data acquisition is also an important aspect of quality assessment. While collecting large-scale data, it's essential to consider the characteristics of the data being collected. Besides YouTube or Reddit, have other data sources been considered? In addition, please address the legality of your data acquisition and processing.\n\n2. Please provide more experimental details, including experimental design and setup. Additionally, only the source code for llama-vid is provided—have any model improvements beyond fine-tuning been considered? The supplementary materials only include the llama-vid code, with the only visible data being a small amount of Reddit data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an innovative perspective by examining large models' understanding of data through the lens of human behavioral performance.\n\n2. The authors have compiled a substantial multimodal dataset and fine-tuned the model from the perspective of behavioral performance, demonstrating impressive results in downstream tasks.", "weaknesses": "1. Lacks sufficient information on data collection, processing, and distribution. The description of the data collection and filtering process lacks logical coherence. The filtering appears to be largely empirical, and there is no thorough analysis of the feature distribution of the collected data.\n\n2. The paper fails to provide information about the specific experimental details used for instruction tuning (including in the appendix) and does not compare the changes in time efficiency resulting from tuning with additional commentary data versus without.\n\n3. The paper states that sensor data is inferior to behavioral data based on the insufficient quantity of sensor data. It could be argued that sensor data is challenging to collect; however, comparing the quality of sensor data requires more substantial evidence.\n\n4. In this paper, \"Ad-LLaVa\" sometimes appears as \"AdLLaVa.\" Please ensure consistency in spelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)