Teaching Human Behavior Improves Content Understanding Abilities Of LLMs

## Abstract

Abstract Communication is defined as “ Who says what to whom with what effect .” A message from a communicator generates downstream receiver effects, also known as behavior. Receiver behavior, being a downstream effect of the message, carries rich signals about it. Even after carrying signals about the message, the behavior signal is often ignored while training vision language models. We show that training VLMs on receiver behavior can actually help improve their content-understanding abilities. We demonstrate that training VLMs to predict receiver behaviors, such as likes, comments, and replay graphs, which are available at scale, enhances the VLM’s performance across a broad range of downstream content understanding tasks. We show this performance increase over 6 types of behavior, 46 different tasks covering image, video, text and audio over 26 benchmark datasets across both 0-shot and fine-tuning settings, outperforming many supervised baselines on diverse tasks ranging from emotion recognition to captioning by upto 150%. We note that since receiver behavior, such as likes, comments, and replay graphs, is collected by default on the internet and does not need any human annotations to be useful, the performance improvement we get after training on this data is essentially free-lunch. We also release BLIFT, our Behaviour-LLaVA IFT dataset comprising of 730k images and videos with their receiver behavior collected from multiple platforms on which we train our models to achieve this.

## 1 Introduction

Communication is defined by five factors: sender, message, channel, receiver, and behavior (Shannon & Weaver, ; Lasswell, ; ). Lasswell () encoded these five factors in the phrase, “Who says what to whom with what effect.” Human behavior occurs as a downstream artifact in the process of communication. Behavior is produced by the receiver as a response to the message sent by the sender. Being a downstream effect, behavior can help us infer important signals about the message itself. These signals, if properly harnessed, should be able to increase performance on the message understanding tasks popular in NLP and CV, like question answering, sentiment analysis, topic classification, etc. Despite this, behavior data is considered noise and is ignored while training large language models (Biderman et al., ; Penedo et al., ) and also large vision and language models (Liu et al., ; Zhu et al., ). In this paper, we explore this line of thought more.

Figure: Figure 2: Behavior-LLaVA is trained to answer behavioral questions like simulating user comments and likes on the video. The model, once trained, shows superior performance than LLaMA-Vid and other VLMs on content-related tasks like emotion recognition, action recognition, question answering, persuasion strategy classification, etc. The original video was showcased in SuperBowl-2024 and is posted on YouTube on the URL [https://www.youtube.com/watch?v=OU7BJc96lI4](https://www.youtube.com/watch?v=OU7BJc96lI4). The video is titled “Perfect 10: The Kia big game commercial featuring the 2024 Kia EV9” by Kia America.
Refer to caption: x3.png

Humans produce two kinds of behavioral signals upon observing a message (Bertenthal, ; Prinz, ): perceptual signals and actions as behavior. Perceptual signals, like seeing, touching, and hearing, help a receiver primarily sense the world around her, ultimately guiding her actions. Actions are how a receiver acts on the outside world.
The signals produced by the human receiver upon receiving a message carry information about the message itself (Fig. 1). For instance, if a person’s heartbeat rises upon watching a movie scene, it can help us infer that perhaps the scene was an exciting scene (Dzedzickis et al., ). Similarly, regressing while reading is indicative of important or confusing phrases (Bicknell & Levy, ). In these cases, perception behavior helps us derive inferences about content. In a similar vein, the actions a person performs after watching a movie, such as comments and likes, carry signals about the movie (Fig. 1, 2).

Expanding on these ideas, prior literature has shown that harnessing perceptual signals, like eye movements, saliency, keystrokes, mouse movements, and FMRI, by modeling them together with content understanding tasks can improve both NLP and CV tasks. For instance, integration with perception signals causes performance improvement in tasks like visual and natural language question answering (Patro & Namboodiri, ; Khurana et al., ; Sood et al., ), text and image sentiment analysis (Khurana et al., ; Barrett et al., ; Fan et al., ), natural language inference (Khurana et al., ), part-of-speech identification (Barrett et al., ; ), named entity recognition (Hollenstein et al., ; Hollenstein & Zhang, ), syntactic parsing (Plank, ), image captioning (Cornia et al., ), and visual object detection (Wang et al., ; Kruthiventi et al., ).

While the initial studies show that perceptual signals have much promise for improving downstream content understanding, they have a few significant issues due to which integrating human perception has not seen wide adoption in training LLMs. These perceptual signals can only be collected in lab settings requiring specialized lab equipment and are thus expensive to collect and thus are also limited in number. For example, the largest datasets containing the human processing signals are SALICON (Jiang et al., ) and Cheng et al. () for visual saliency (10k images each), CELER (Berzak et al., ) and Dundee corpus (Kennedy et al., ) containing eye movements over 28k sentences and 20 news articles respectively, and Dhakal et al. () containing keystroke patterns over 1.5k sentences. Clearly, these datasets, while making important contributions, do not scale to the level at which today’s large language models are trained (trillions of natural language and image tokens).

On the other hand, actions (the other type of behavioral signals produced by a human receiver) are collected at a large scale in the form of digital analytics. Examples of this kind of data are likes, views, shares, comments, and purchase histories on images, tweets, videos, webpages, and other kinds of media. Action data has a much broader representation than is possible in lab settings, is available on more diverse content, and is much cheaper to collect than using specialized lab equipment. At the same time, actions have not been much investigated in the literature for their potential to improve downstream content understanding.

Therefore, in this paper, we make initial efforts to collect and understand digital analytics at scale with the aim of integrating them with VLMs to improve their downstream content understanding capabilities. We introduce methods for filtering and cleaning behavioral data and then propose tasks for large language and vision models, leading to improvements in language and visual content understanding tasks. For this, we look to Reddit and YouTube as two major sources of visual content and human behavior in the form of viewer comments, likes, replay graphs, and upvotes. From Reddit, we collect 5 million images and videos along with their upvotes and top-upvoted comments from two major subreddits (r/pics and r/videos). Similarly, from YouTube, we collect 2.2 million videos from 30000 channels along with their likes, views, replay graphs, and top user comments. After extensive filtering and cleaning, we are left with 730k samples of videos and images across the two platforms which we use for the next steps.

After collecting user behavior over image and video content, we design tasks to teach large vision and language models (VLMs) to simulate user behavior. For this, we use an instruction fine-tuning format. Given a video or an image and the other metadata like time of post and channel, we ask the model to simulate user behavior of likes and comments. See Fig 2, Listing 1 for examples. We choose LLaMA-Vid (Li et al., ) as our base model to teach it the user behavior. We call the resultant model Behavior-LLaVA (Large Language and Vision Assistant) (Liu et al., ). We test Behavior-LLaVA on a diverse variety of tasks, evaluating its capabilities on image, video, text, and audio understanding tasks. We compare Behavior-LLaVA against its base model, LLaMA-Vid, and other supervised baselines. Further, to show the impact of behavior, we train another version of LLaMA-Vid, where we train it on the same set of videos and images as Behavior-LLaVA but do not include behavior information. We call this model Ad-LLaVA.

Figure: Figure 3: Behaviour-LLava achieves much higher zero-shot performance compared to Ad-LLaVA and the base model LLaMA-VID across a diverse suite of image, video, and audio benchmarks.
Refer to caption: x4.png

We make the following contributions with this work:
1) Behavior-LLaVA Instruction Fine-Tuning: We explore the idea of learning human behavior, resulting in better content understanding. We test this for action-level behavior data such as receiver comments, likes, and replay graphs. We collect a dataset called BLIFT, consisting of 400k images and 330k videos, along with their receiver behavior. Then, LLaMA-Vid is trained for the task of predicting receiver comments and upvotes given a media (a video or an image) (Listing 1). We show that using this simple task formulation over behavioral data collected in the wild, results in performance improvement over a hierarchy of tasks. We get improvements over the base LLaMA-Vid across 46 tasks over 26 benchmark datasets in both zero-shot and fine-tuned settings. We show this over low-level content understanding tasks like object and activity recognition and also over high-level tasks like topic and emotion detection.
Through this, we propose a scalable approach to increase the content understanding abilities of VLMs, requiring minimal cost and no architectural changes.

2) AdLLaVA: Disentangling the effect of content and behaviour: To disentangle the effect of training LLaMA-Vid on additional image and video data from the effect of training on behavior data, we train LLaMA-Vid on BLIFT’s videos and images without including behavior. We call this model Ad-LLaVA. We show that Ad-LLaVA shows equivalent performance as its base model LLaVA-Vid; however, Behavior-LLaVA performs better than both Ad-LLaVA and LLaMA-Vid, thus highlighting the importance of behavior data and instruction fine-tuning on behavior data.

3) Perception vs Action:We also show an ablation of Behavior-LLaVA across different kinds of behavior. We try out the perception behavior of saliency prediction over images and five types of action-level behavior over images and videos. We find that perception-level behavior does not result in significant performance improvements; however, action-level behavior shows improvements across all the tasks. We posit that one reason for this could be due to the scale for which action-level data is available (Table 11). While perception behavior is mostly collected in lab settings, action-level behavior data is diverse, can be collected in a scalable manner automatically and cheaply.

## 2 Methodology

In this section, we introduce our approach to train Behavior-LLaVA. Since no publicly available corpus consists of behavior together with image and video content, we first introduce our instruction fine-tuning dataset, “Behavior-LLaVA Instruction Fine-Tuning dataset” (BLIFT). Next, we introduce our methodology to train Behavior-LLaVA. Finally, we report the results of testing Behavior-LLaVA’s capabilities on a hierarchy of tasks. The tasks cover low-level media understanding tasks like object and activity detection, high-level media understanding tasks like emotion, topic, and persuasion strategy classification.

### 2.1 BLIFT Dataset

Given the abundance of media and behavioral data and its accessibility, our data collection relies on two primary sources: Reddit and YouTube. These platforms share similarities in terms of hosting media content (images and videos) and providing user engagement metrics in the form of Reddit upvotes and comments, and YouTube likes, views, comments, and replay graphs. Here, we delineate the process involved in constructing the instruction fine-tuning dataset, which we term as the Behavior-LLaVA Instruction Fine-Tuning (BLIFT) dataset.

#### 2.1.1 Data from Reddit

To collect a substantial corpus of diverse images and videos, we targeted at two specific subreddits, namely r/pics and r/videos. Established over 15 years ago, these subreddits had a user base exceeding 20 million during the data collection period, with an average of over 5,000 online users concurrently. Notably, due to stringent content moderation guidelines (Reddit, Inc., ; red, ; ; ) and the exclusive focus of these subreddits on media content, they offer a rich variety of content devoid of thematic biases. Our data collection spans until January 2022, during which the activity on these subreddits witnessed a notable decline following several policy adjustments and user protests (Hern, ; Economist, ).

To ensure data quality and relevance, we executed a series of filtering steps on the posts and comments from these subreddits. Initially, we excluded posts predating February 2015 from r/pics, coinciding with the implementation of a rule requiring images without digital/overlay text (red, ; ). This filtering step resulted in the exclusion of 3.1 million images and 2 million videos. Subsequently, considering the sustained popularity of both subreddits, with rankings within the top 20 since 2017 and consistent membership exceeding 20 million, we confined our dataset to posts from January 2018 onwards. This selection process yielded 1.4 million images and 1.1 million videos.

Further refinement of the dataset involved removing posts and comments marked as NSFW, BOT-generated, or [deleted], along with eliminating duplicate images and videos. This curation step reduced the dataset to 876,000 images and 983,000 videos. To address redundancy in comments, we excluded those comprising fewer than three words and employed TF-IDF-based deduplication with a similarity threshold of 0.6, determined through manual observations.

Following these steps, posts with fewer than two comments were filtered out, resulting in a dataset comprising 631,000 images and 397,000 videos. Additionally, videos exceeding a duration of 500 seconds were omitted, leaving 221,000 videos for analysis. Notably, images not directly hosted on Reddit were excluded due to scraping and copyright limitations. Similarly, for r/videos, only videos hosted on YouTube were considered. It is pertinent to mention that approximately 51% of YouTube videos collected during this period were either made private/unlisted or removed, resulting in 400,000 images and 80,000 videos, accompanied by 1.5 million and 312,000 comments, respectively. These comprehensive filtering steps ensured the construction of a diverse and relevant dataset for fine-tuning instruction-based models.

Figure: Listing 1: Behavior Instruction fine-tuning template for the video: [https://www.youtube.com/watch?v=BKPQkjRF4yY](https://www.youtube.com/watch?v=BKPQkjRF4yY)
Refer to caption: x5.png

#### 2.1.2 Data from YouTube

Our data collection from YouTube begins with querying Wikidata (Vrandečić & Krötzsch, ) for YouTube IDs to compile a list of channels. Wikidata, derived from Wikipedia, provides a curated selection of renowned channels, automatically filtering out noisy videos commonly found in datasets collected from diverse sources like user-generated videos. This initial step yielded a dataset of 2.2 million videos spanning the period from 2018 to 2023, sourced from approximately 6,000 channels collected from Wikidata.

To refine the dataset, manual filtering was employed to exclude certain categories deemed less relevant for our purposes. These categories included music and songs, gaming content, non-English videos, sports commentary, anime, memes, channels with disabled comments sections, and news-related content. Furthermore, and only videos with a substantial viewership, defined as greater than 10,000 views, were retained. We observed that these videos usually have less noisy comments and likes.

Subsequently, the top comments from each video, as ordered by YouTube (i.e., the most liked comments), were selected for inclusion in the dataset. To address redundancy in comments, a TF-IDF filter was applied with a threshold of 0.7, which proved effective in removing duplicate comments prevalent in YouTube data.

Comments were further filtered to include only those with a minimum of four words and a maximum of 100 words, ensuring a balance between relevance and conciseness. Additionally, to mitigate the presence of NSFW content, a vocabulary specific to NSFW terms () was employed to filter out inappropriate posts. On average, we finally get 3.1 comments per video, providing a substantial corpus of user-generated content for analysis. After applying these filtering steps, the dataset was reduced to 250,000 videos, ensuring a curated and relevant collection for subsequent analysis and model training.

### 2.2 Instruction Fine-Tuning LLaMA-Vid

After collecting Reddit and YouTube media and user behavior, we formulate instruction fine-tuning tasks for training LLaMA-Vid. In the training instruction, given the media content and automatic speech recognition if available, we ask the model to simulate the scene-by-scene description and user likes/views and top-5 comments. This instruction training template is given in Listing 1.
To generate the instruction data, first, frames are sampled using the 30-degree rule (SI et al., ; Arev et al., ; Friedman & Feldman, ), then the scene-by-scene description are obtained by concatenating automatically generated captions and tags from LLaVA-13B (Liu et al., ), colors and tone through Qin et al. (). This instruction format keeps the instructions similar to the instruction format for other VLMs like LLaVA (Liu et al., ), MiniGPT-4 (Zhu et al., ), BLIP (Li et al., ), LLaMA-Vid (Li et al., ), etc., while additionally teaching the model to learn behavior. We keep the instruction fine-tuning template similar for both YouTube and Reddit. The complete instruction is given in Listing 1.

We start with the trained LLaMA-Vid model. The LLaMA-Vid model uses two tokens to represent each frame in the video, which they call content and context tokens. While the context token encodes the overall image context based on user input, the content token encapsulates visual cues in each frame. For learning context tokens, the model uses attention queries that interact with previously generated image features in the designed attention module. To generate content tokens, the image features are average pooled.
This dual-token strategy significantly reduces the number of tokens needed to represent videos, thus enabling the model to scale to longer (hour-long) videos. To better support hour-long videos, LLaMA-Vid was trained on a 9k movie-level conversation instruction set containing plot reasoning and detail understanding questions.

Taking the base LLaMA-Vid model, we finetune it further on behavioral data. In our experiments we observe that the sampling ratio of BLIFT and IFT datasets is an important hyperparameter. We track 4 zero-shot metrics, likes/views, comments perplexity, and empirically find the best results with 1:1 ratio for 2.2 epochs (see Fig 11 and Table 12). For the best checkpoint, the perplexity on comments reduces from 6.22 to 3.05, and the R2
on likes/views goes from -5.1 to 0.45

We combine 730k instruction pairs from BLIFT with the original instruction tuning dataset consisting of 40K text conversations from ShareGPT, 625K single or multi-turn visual QA pairs, and 98K video QA pairs; all the modules except the Visual Encoder are kept frozen. We ablate on multiple sampling ratios from BLIFT. We train the LLaMA-Vid checkpoints with their original SFT mix along with BLIFT. We ablate different sampling ratios and found 1:1 to be empirically performing the best. We train the model for 2.2 epochs, keeping track of the 0-shot evaluation metrics and perplexity on comments in the eval set Fig 11 and Table 12 show the ablations on different sampling ratios and epochs of training.
For the best checkpoint, the perplexity on comments reduces from 6.22 to 3.05, and the $R^{2}$ on likes/views goes from -5.1 to 0.45.

AdLLaVA to show the impact of behavior data: To disentangle the effect of training on additional data samples from the effect of training on behavioral data, we train LLaMA-Vid on BLIFT with the video and image verbalization and do not include receiver behavior. Then, the overall instruction template consists of scene-by-scene automatically generated verbalization similar to Listing 1 without the likes and comment simulation. We call the LLaMA-Vid fine-tuned on this data, Ad-LLaVA. We compare Behavior-LLaVA with Ad-LLaVA and LLaMA-Vid along with other state-of-the-art literature benchmarks on various tasks (Tables 1-4).

Impact of filtering steps on performance: We use various filtering steps in our data pipeline, including NSFW filters, time filters, bot filters, and gaming and news video filters. To quantify the impact of our filtering steps on the final performance, we compare the performance from training on unfiltered data with filtered data (Figure 11).
The figure shows the benefit of our data filtering
process. While training on unfiltered BLIFT also improves the result over baseline performance of Llama-Vid
but data filtering adds more improvement on top of it

Ablation with perceptual behavior: As an ablation experiment, we also try teaching the Behavior-LLaVA perceptual signals. For this, we take the largest perception signal dataset in the literature - Salicon10k (Jiang et al., ). It consists of 10,000 MS COCO images (Chen et al., ) with free-viewing eye gaze data collected through a novel mouse-based interface. The dataset has been widely used in many studies. We formulate two tasks using this data, (1) Salicon [Object]: estimating the saliency over the objects in the image and (2) Salicon [Region]: estimating the saliency over a region, where the regions tiles formed by breaking the image into a 3x3 grid. For both tasks we try to model two objectives, ranking and predicting, we found ranking to be much more effective. The instruction are given in Listings 5 and 6.

Table: Table 1: Comparison of various models on the Long Video Understanding benchmark (Wu & Krahenbuhl, ) consisting of 9 VQA tasks. We see that Behavior-LLaVA improves on LLaMA-Vid on 9/9 tasks with an average improvement of 21.49%. Further, it outperforms the state-of-the-art in 5/9 tasks.

## 3 Results and Discussion

In the experimental results, we aim to showcase the diverse and emergent capabilities of our Behavior-LLaVA model through quantitative numbers on various tasks and qualitative examples. These abilities include generating detailed image and video descriptions, emotion and sentiment analysis, question answering, video understanding tasks like scene and action detection. Additionally, we present the ability of Behavior-LLaVA to transfer learn on other behaviors like memorability of a video - both short-term and long-term.

### 3.1 Evaluation

To test the effectiveness of Behavior-LLaVA, we conduct experiments involving 46 distinct tasks across 26 benchmark datasets. The diversity of tasks and datasets allows us to evaluate the performance and capabilities of Behavior-LLaVA thoroughly. Each of them is covered briefly next:

- 1.
Visual Question Answering (VQA): We evaluate the performance of visual question answering on the following benchmark datasets:
•
The Long-Video Understanding (LVU) benchmark by Wu & Krahenbuhl () comprises nine distinct tasks aimed at assessing long video comprehension, incorporating over 1000 hours of video content. These tasks encompass diverse aspects such as content understanding (including relationship, speaking style, scene/place), prediction of user engagement (YouTube like ratio, YouTube popularity), and movie metadata (director, genre, writer, movie release year).
•
The Holistic Video Understanding (HVU) dataset by Diba et al. () stands as the largest dataset for long video comprehension, comprising 572,000 samples. Encompassing a broad spectrum of semantic elements within videos, HVU tasks involve the classification of scenes, objects, actions, events, attributes, and concepts. Performance evaluation on HVU tasks is conducted using the mean average precision (mAP) metric on the validation set.
•
We also use MSVD-QA, MSRVTT-QA (Chen & Dolan, ; Xu et al., ), and ActivityNet-QA (Caba Heilbron et al., ) datasets. Their description is given in Appendix A.
- 2.
Video and Image Understanding Benchmarks: We use a wide variety of tasks to evaluate video and image understanding: topic, emotion, and persuasion strategy classification, action and reason retrieval and generation, and emotions. We briefly introduce the benchmarks:
•
The advertisements dataset by Hussain et al. () contains 3,477 video advertisements and the corresponding annotations for emotion and topic tags and action-reason statements for each video. There are a total of 38 topics and 30 unique emotion tags per video. Further, we have 5 action-reason statements for each video for the action-reason generation task.
•
Persuasion strategy dataset (Bhattacharyya et al., ) is a dataset consisting of 1002 video advertisements from popular brands and their persuasion strategy labels like social identity, anchoring and comparison, reciprocity, foot-in-the-door, etc.
Training
Model
Topic
Sentiment
Persuasion
Action
Reason
Clubbed
All labels
Random
Random
2.63
3.37
14.3
8.37
3.34
3.33
Zero-shot
VideoChat (Li et al., )
9.07
3.09
5.1
10.28
-
-
Video4096 - GPT-3.5 Generated Story + GPT-3.5 Classifier
51.6
11.68
79.69
35.02
66.27
59.59
LCBM (Khandelwal et al., )
42.17
7.08
58.83
32.83
39.55
27.91
LLaMA-VID w/ only video
10.11
3.42
5.75
12.32
29.61
24.11
LLaMA-VID w/ video + GPT-3.5 Story
42.72
11.05
64.02
32.07
37.76
42.33
Behavior-LLaVA w/ only video
22.65
11.13
60.04
13.39
42.66
33.33
Behavior-LLaVA w/ video + verbalization
46.34
11.7
64.13
33.33
52.06
52.03
Ad-LLaVA w/ video + GPT-3.5 story
51.16
11.33
68.03
33.11
43.26
51.45
Behavior-LLaVA w/ video + GPT-3.5 story
60.09
12.84
79.94
36.12
67.10
79.18
Improvement of Behavior-LLaVA over LLaMA-Vid
40.66%
16.2%
24.86%
12.62%
77.7%
87.05%
Finetuned
Video4096- Generated Story + Roberta Classifier
71.3
33.02
84.20
64.67
42.96
39.09
LLaMA-VID w/ video + verbalization
59.13
32.11
79.15
50.93
50.32
30.13
LLaMA-VID w/ video + GPT-3.5 Story
63.11
35.01
84.15
55.01
57.11
45.73
Behavior-LLaVA w/ only video
58.03
22.72
84.41
26.23
59.33
51.45
Behavior-LLaVA w/ video + verbalization
68.32
33.92
85.93
64.72
70.89
75.34
Ad-LLaVA w/ video + GPT-3.5 story
66.34
36.24
84.09
58.31
68.15
78.15
Behavior-LLaVA w/ video + GPT-3.5 story
71.2
39.55
86.17
65.03
80.44
81.67
Improvement of Behavior-LLaVA over LLaMA-Vid
12.82%
12.97%
2.4%
18.21%
40.85%
78.59%
Table 2: Comparison of various models on two video understanding benchmarks (Hussain et al., ; Kumar et al., ) consisting of 5 tasks related to video advertisements understanding. The main goal of comparing on these benchmarks is to demonstrate Behavior-LLaVA’s understanding of complex videos. We see that Behavior-LLaVA improves on LLaMA-Vid on 5/5 tasks with an average improvement score of 43.18% in zero-shot and 27.64% in fine-tuned settings. Further, it outperforms the current state-of-the-art on 3/5 tasks in zero-shot and 5/5 in fine-tuned settings. Full results are presented in the Table 6.
•
For emotion analysis, we use VideoEmotion-8 (Asur & Huberman, ), Ekman-6 (Xu et al., ), CAER (Lee et al., ), IAPSa (Mikels et al., ), Emotion6 (Peng et al., ), EmoSet (Yang et al., ), and Abstract (Machajdik & Hanbury, ) datasets. A brief description for each of them is given in Appendix A.
- 3.
Image Dense Captioning: Literature image captioning datasets such as MS-COCO (Chen et al., ) reduce the inherently rich information and fine-grained semantics to simplistic captions, with very brief statements focussing only on salient objects. Behavior data such as user comments help a model learn much more information such as object and material properties, world knowledge, emotion, character understanding, spatial relationships, aesthetics, etc. (see Fig. 1), enhancing the model’s captioning capability. Therefore, we design a captioning task to test this capability and compare it with respect to LLaMA-Vid and LLaVA-34B (a 2.5x larger model). Since we do not have ground truth for this task, following the LLM-as-a-judge paradigm, we use GPT-4V as the judge for all the models. GPT-4V is asked to evaluate the dense captions on three metrics: Correctness (Listing 2) evaluating the factuality and model hallucinations, Detail (Listing 3) evaluating the number and depth of details captured by the generated captions, and Quality (Listing 4) measuring the subjective quality of the concepts chosen to be highlighted by the captioning model and the arrangement, coherence, and the linking of various concepts.
- 4.
Image and Video Memorability Simulation: Behavior-LLaVA is trained on behavior along with the media. To check if training on behavior helps in solving other behavior tasks (Khandelwal et al., ), we test it over image and video memorability simulation. For this, we select seven benchmark datasets covering long-term and short-term memorability over images and videos: LaMem (Khosla et al., ), SUN (Isola et al., ), and MemCat (Goetschalckx & Wagemans, ) for images and
Memento10k(Newman et al., ), VideoMem (Cohendet et al., ), MediaEval (Kiziltepe et al., ), and LAMBDA (I et al., ) for videos. We briefly cover each of them in Appendix A.
- 5.
Modalities other than videos and images: Behavior-LLaVA, built on top of LLaMA-Vid and fine-tuned using BLIFT, is pretrained and fine-tuned on image and video datasets. To test if behavior data can improve the results on other modalities as well, we test Behavior-LLaVA’s performance on two tasks across audio and text modalities (Table 13). For audio, we evaluate on the audio summarization task (Han et al., ) and for text, we evaluate on the IMDB sentiment benchmark (Maas et al., ).
Training
Dataset
Video Emotion-8
CAER
Ekman-6
Random
Random
12.5
14.28
16.67
0-Shot
LLaMA-Vid
29.7
27.2
37.33
Behavior-LLaVA
41.35
51.0
49.33
Ad-LLaVA
29.8
27.3
37.66
Improvement of Behavior-LLaVA over LLaMA-Vid
39.22%
84.19%
32.14%
Finetuned
Zhao et al. ()
54.5
78.3
55.3
Zhang et al. ()
57.3
80.1
58.2
eMOTIONS (Wu et al., )
-
-
53.12
Arevalo et al. ()
53.7
77.3
54.2
Qiu et al. ()
53.3
-
57.3
Xu et al. ()
52.6
77.9
55.6
LLaMA-Vid
53.8
75.6
57.9
Ad-LLaVA
54.1
76.1
57.8
Behavior-LLaVA
56.9
79.3
58.4
Improvement of Behavior-LLaVA over LLaMA-Vid
5.76%
3.57%
0.86%
Table 3: Comparison of various models on three video emotion understanding benchmarks (Video Emotion8 (Jiang et al., ), CAER (Lee et al., ), Ekman-6 (Xu et al., )). The main goal of comparing on these benchmarks is to demonstrate Behavior-LLaVA’s understanding of complex tasks like video emotions of long-form videos. We see that Behavior-LLaVA improves on LLaMA-Vid on 3/3 benchmarks with an average improvement score of 51.85% in zero-shot and 3.39% in fine-tuned settings. Further, it outperforms the current state-of-the-art on 3/3 benchmarks in zero-shot and 1/3 in fine-tuned settings.

For Tables 1, 10, and 2, we follow the evaluation protocol of Video-4096 (Bhattacharyya et al., ), for Table 7 we follow the evaluation protocol of LLaVA and LLaMA-VID. For Tables 3, 8, and 4, for 0-shot evaluation results, we use the logits of the next token from the given task vocabulary. For Table 4, we use the evaluation protocol of SI et al. ().

Table: Table 4: Comparison of various models on seven video and image memorability benchmarks (Memento10k (Newman et al., ), VideoMem (Cohendet et al., ), LaMem (Khosla et al., ), SUN (Isola et al., ), MemCat (Goetschalckx & Wagemans, ), MediaEval (Kiziltepe et al., ), LAMBDA (I et al., )). The main goal of comparing on these benchmarks is to demonstrate Behavior-LLaVA’s understanding of complex and high-level tasks like memorability simulation. We see that Behavior-LLaVA improves on LLaMA-Vid on 7/7 benchmarks with an average improvement score of 186.4% in zero-shot and 39% in fine-tuned settings after seeing 25% train data. Further, it performs similarly to the current state-of-the-art on 7/7 benchmarks in the fine-tuned settings while still seeing only 25% data.

### 3.2 Discussion

Tables 1, 10, and 7 contain the results for the visual question answering tasks, Tables 2, 3, 8 contain the results for video and image understanding tasks, Tables 9 contains the results for dense-captioning, Table 4 contains the results for image and video memorability benchmarks, and Table 13 contains the results for the audio and text tasks. A common trend we observe across all the results is that Behavior-LLaVA performs better than the base model LLaMA-Vid and the finetuned model Ad-LLaVA on all tasks, especially in zero-shot settings. In fact, Ad-LLaVA performs very similar to LLaMA-Vid itself. This shows that BLIFT adds meaningful signals on an average rather than noise to the model. Interestingly, the performance gains remain even after fine-tuning on the task dataset (Tables 2, 3, 8).

The performance gains are relatively smaller for low-level tasks of action and object recognition (Tables 10, and 7), but much higher for the more high-level tasks of emotion understanding, sentiment analysis, persuasion strategy classification, and memorability simulation, longform-video understanding and other sub-tasks of Table 10. This indicates that receiver behavior has richer signals for higher-level tasks, infact fine-tuned Behaviour-LLaVA models outperform GPT4-V on image emotion recognition. The gains are observed across both image and video benchmarks. We also observe that classification using a story generated by GPT-3.5 (following Bhattacharyya et al. ()) results in better performance than only using the video (Tables 1, 10, 2).
To evaluate the generalizability of behavior data across modalities, we extend our evaluation to audio summarization and text sentiment analysis and observe improvements of 19.5% (Table 13).

Figures 6, 7, 10, and 5, 8, 9, show several randomly sampled qualitative examples for dense captions generated by Behavior-LLaVA over images and videos respectively. It can be noticed that despite not being explicitly trained for this task, the model performs quite well, picking up various artistic, cognitive, and object and material properties. From Table 9, while Behavior-LLaVA shows a decrease in correctness over LLaMA-Vid, it shows significant improvement in other aspects, including detail and quality. On these aspects, it even comes close to 2.5X larger models (LLaVA-1.6 (34B)).

Next, in Table 11 we compare the signals from behavioral data of perception and action. For this, we compare Behavior-LLaVA trained on BLIFT and Behaviour-LLaVA trained on Salicon salient regions and objects. Further, within BLIFT, we compare the performance from predicting singled out behaviours including likes/views, titles, comments. It can be noted that training on just Salicon results in a performance decrease for the lower-level task of action recognition (MSRVTT-QA) but improves on the higher-level task of Emotion recognition. However, the gains are smaller than those observed with training on BLIFT.

Table: Table 5: Improvement on downstream content understanding tasks by introducing more behaviour signals. Brackets [] denote the new behaviour that we include. Replay graphs (Khandelwal et al., ). Mem-Recalls (I et al., ) Evaluation done on Multi-shot video summarization (Han et al., ) and MomentDETR (Lei et al., )

## 4 Conclusion

In this paper, we explore the idea of learning behavior leading to learning content better. Humans produce behavior in response to content. Hence, logically, behavior should contain signals about content, which, if used as a training task, should help in learning content better. We follow this line of thought and show that training large vision and language models on user behavior data of comments and likes collected from Reddit and YouTube leads to performance improvements across a wide variety of tasks. The gains are higher on higher-level tasks such as emotion recognition, persuasion strategy classification, and question answering and smaller on lower-level tasks like action and object recognition. Further, the gains remain even after fine-tuning the VLMs on those benchmarks, thus demonstrating the importance of learning behavior in understanding content better.

## Appendix

Figure: Listing 2: GPT-4V Prompt to calculate correctness of a image dense caption

Figure: Listing 3: GPT-4V Prompt to calculate detail of a image dense caption

Figure: Listing 4: GPT-4V Prompt to calculate quality of a image dense caption

Figure: Listing 5: Perceptual Signal Instruction fine-tuning template for the image: [http://farm6.staticflickr.com/5106/5670500150_e035dd2d30_z.jpg](http://farm6.staticflickr.com/5106/5670500150_e035dd2d30_z.jpg)
[Uncaptioned image]: x6.png

Figure: Listing 6: Perceptual Signal Instruction fine-tuning template for the image: [http://farm6.staticflickr.com/5106/5670500150_e035dd2d30_z.jpg](http://farm6.staticflickr.com/5106/5670500150_e035dd2d30_z.jpg)
[Uncaptioned image]: extracted/5915812/images/listing-image-grid.jpg

Figure: Figure 5: Dense caption generated by Behavior-LLaVA for the video of a Volkswagen ad. The original video is posted at URL: [https://www.youtube.com/watch?v=kyuGXPNr-T0](https://www.youtube.com/watch?v=kyuGXPNr-T0). The red-colored text highlights the most important aspects of the video captured by Behavior-LLaVA, demonstrating an understanding of aesthetics, characters, world knowledge, emotion, and spatial relationships. More such examples are given in Figs. 6, 7, 10, and Figs. 8, 9 for images and videos respectively.
Refer to caption: x7.png

Figure: Figure 6: Dense caption generated by Behavior-LLaVA for an artistic drawing of a tree.
Refer to caption: x8.png

Figure: Figure 7: Dense caption generated by Behavior-LLaVA for a Nike ad. The red-colored text highlights the most important aspects of the video captured by Behavior-LLaVA, demonstrating an understanding of aesthetics, characters, world knowledge, emotion, and spatial relationships.
Refer to caption: extracted/5915812/images/nike-qualitative.jpeg

Figure: Figure 8: Dense caption generated by Behavior-LLaVA for the video of the official trailer of the game Red Dead Redemption 2. The original video is posted at URL: [https://www.youtube.com/watch?v=eaW0tYpxyp0](https://www.youtube.com/watch?v=eaW0tYpxyp0).
Refer to caption: x9.png

Figure: Figure 9: Dense caption generated by Behavior-LLaVA for the video of Argentina vs France FIFA World Cup Qatar 2022 Highlights. The original video is posted at URL: [https://www.youtube.com/watch?v=zhEWqfP6V_w](https://www.youtube.com/watch?v=zhEWqfP6V_w).
Refer to caption: x10.png

Figure: Figure 10: Dense caption generated by Behavior-LLaVA for a painting of a soldier. The model captures many qualitative aspects that are usually missed in common captioning tasks.
Refer to caption: x11.png

Table: Table 6: Comparison of various models on two video understanding benchmarks (Hussain et al., ; Kumar et al., ) consisting of 5 tasks related to video advertisements understanding. The main goal of comparing on these benchmarks is to demonstrate Behavior-LLaVA’s understanding of complex videos. We see that Behavior-LLaVA improves on LLaMA-Vid on 5/5 tasks with an average improvement score of 43.18% in zero-shot and 27.64% in fine-tuned settings. Further, it outperforms the current state-of-the-art on 3/5 tasks in zero-shot and 5/5 in fine-tuned settings.

Table: Table 7: Comparison of various models on three conventional video question answering benchmarks consisting of question answers related to action understanding. The main goal of comparing on this benchmark is to show that Behavior-LLaVA does not perform worse on low-level understanding tasks like action recognition. We see that Behavior-LLaVA marginally improves on LLaMA-Vid on 2/3 benchmarks. Further, it performs equivalent to the state-of-the-art in 3/3 benchmarks.

Table: Table 8: Comparison of various models on four image emotion understanding benchmarks (IAPSa-8 (Mikels et al., ) Abstract (Machajdik & Hanbury, ), Emotion6 (Peng et al., ), Emoset (Yang et al., )). The main goal of comparing on these benchmarks is to demonstrate Behavior-LLaVA’s understanding of complex tasks like image emotions. We see that Behavior-LLaVA improves on LLaMA-Vid on 4/4 benchmarks with an average improvement score of 29.14% in zero-shot and 8.95% in fine-tuned settings. Further, it outperforms the current state-of-the-art on 4/4 benchmarks in the fine-tuned settings.

Table: Table 9: Comparison of various models on the image dense captioning task. The main goal of this task is to demonstrate Behavior-LLaVA’s image captioning ability. Despite not being explicitly trained on this task, Behavior-LLaVA performs better than both Ad-LLaVA and LLaMA-Vid on Detail and Quality aspects while losing marginally on correctness. On the aspects of detail and quality, it even outperforms the much larger model of LLaVA-1.6 (34B).

Table: Table 10: Comparison of various models on the Holistic Video Understanding benchmark (Diba et al., ) consisting of 7 VQA tasks. We see that Behavior-LLaVA improves on LLaMA-Vid on 6/7 tasks with an average improvement of 5.88%. Further, it performs equivalent to the state-of-the-art in 6/7 tasks. State of the art is achieved by generating a story and asking Behavior-LLaVA to answer questions based on the generated story.

Table: Table 11: Ablation on using comments and/or perception signals from Salicon

Table: Table 12: Ablation on different sampling ratios and epochs of training. Sampling ratio is the ratio of behaviour data to multimodal instruct data. Performance is the average increase in 0-shot accuracy on 6 tasks with 250 samples each from the eval set. These tasks include image emotion recognition, video emotion recognition, persuasion strategy classification, MSRVTT, HVU and MSVD-QA

Table: Table 13: Evaluation on audio and text modalities. We evaluate on the audio summarization benchmark(Han et al., ) for audio and IMDB sentiment benchmark for text.(Maas et al., )

Figure: Figure 11: Percentage performance improvement over an untrained LLaMA-Vid model, compared across various sampling ratios at different checkpoints. The 1:1 sampling ratio shows the best empirical performance. Performance is averaged over 0-shot accuracy improvements on six tasks with 250 samples each from the evaluation set. These tasks include image emotion recognition, video emotion recognition, and persuasion strategy classification, MSRVTT, HVU and MSVD-QA. The figure also shows the benefit of our data filtering process. While training on unfiltered BLIFT also improves the result over baseline performance of Llama-Vid but data filtering adds more improvement on top of it.
Refer to caption: extracted/5915812/images/performance_vs_epoch.png

## Appendix A Dataset Descriptions

- 1.
MSVD-QA and MSRVTT-QA: These datasets are based on Microsoft Research Video Description (Chen & Dolan, ) and MSR-VTT corpora (Xu et al., ) and are extensively used in many video captioning and question-answering experiments. The MSVD-QA dataset has a total number of 1,970 video clips and 50,505 question-answer pairs. The MSRVTT-QA dataset contains 10K video clips and 243k question-answer pairs.
- 2.
ActivityNet-QA (Caba Heilbron et al., ) is a benchmark primarily for human activity understanding containing 849 hours of video, including 28,000 action instances.
- 3.
VideoEmotion-8 (Asur & Huberman, ) dataset comprises 1,101 user-generated videos sourced from YouTube and Flickr, each containing a minimum of 100 videos per emotional category, as per Plutchik Wheel’s emotion model.
- 4.
Ekman-6 (Xu et al., ) dataset is compiled from social websites, with each of its 1,637 videos labeled with a single emotion category based on Ekman’s psychological research.
- 5.
CAER (Lee et al., ) dataset, sourced from TV shows, consists of 13,201 clips with an average sequence length of 90, each manually labeled with six basic emotions, aligning with the Ekman-6 dataset.
- 6.
IAPSa (Mikels et al., ) is a subset of IAPS, following the Mikels model with eight emotion categories such as amusement, awe, contentment, excitement, anger, disgust, fear, and sadness. It consists of 395 affective images, marking the first visual emotion dataset with discrete categories.
- 7.
Emotion6 (Peng et al., ) features 1,980 images sourced from Flickr, each labeled by 15 annotators according to the Ekman model, covering six emotion categories: happiness, anger, disgust, fear, sadness, and surprise.
- 8.
EmoSet (Yang et al., ) encompasses a total of 3.3 million images, including 118,102 from social networks and artistic sources, evenly distributed across various emotion categories. Based on the Mikels model, EmoSet is categorized into eight emotion categories.
- 9.
Abstract (Machajdik & Hanbury, ) exclusively consists of color and texture combinations without recognizable objects. Differing from the IAPS dataset where emotions often stem from identifiable objects, the abstract paintings dataset was peer-rated via a web survey, with each image rated approximately 14 times. It comprises 228 images spanning eight categories similar to those in IAPS.
- 10.
Memento10k(Newman et al., ) is a short-term video memorability dataset comprising 10,000 video clips, with 900,000 human memory annotations recorded at various delay intervals. The video clips were, on average, 3s long.
- 11.
VideoMem (Cohendet et al., ) is comprised of 10,000 soundless videos, each lasting 7 seconds, accompanied by memorability scores. Memorability was measured twice: first, shortly after viewing and again 24-72 hours later to capture both short-term and long-term memorability effects.
- 12.
LaMem (Khosla et al., ) dataset is a short-term image memorability dataset comprising of 60000 images. The dataset contains scene-centric images, object-centric images and other types such as images
of art, images evoking certain emotions, and other user-generated images.
- 13.
SUN (Isola et al., ) dataset is a short-term image memorability dataset comprising 2222 images that were sourced from the SUN database.
- 14.
MemCat (Goetschalckx & Wagemans, ) dataset is a short-term image memorability dataset comprising 10000 images. It consists of five broader memorability-relevant semantic categories (animal, sports, food, landscapes, vehicles), with 2K exemplars each, further divided into different subcategories (e.g., bear, pigeon, cat, etc. for animal). The images were sourced from existing image sets: ImageNet, COCO, Open Images Dataset, and SUN.
- 15.
MediaEval (Kiziltepe et al., ) utilized publicly available links to short-form video clips, each averaging 6 seconds in duration, with both short-term and long-term memorability scores. Short-term memorability evaluations were conducted on videos viewed within the preceding few minutes, while long-term memorability assessments were based on videos viewed within the previous 24 to 72 hours.
- 16.
LAMBDA (I et al., ) is a long-term memorability consisting of 2205 video ads collected over 1749 participants covering 276 brands. The average video length is 33 seconds and the videos are highly complex, consisting of audio, logos, fast-moving scenes, emotions, etc.

## Appendix B Limitations

In this paper, we try to show the hypothesis that training on the behavior modality improves learning of the content modality. We train the models on comments and likes to show this. We test our models on multiple benchmarks and obtain positive results. While we try to cover a wide variety of tasks and while results do conclusively show that the hypothesis is true, yet, we can test on more benchmarks covering even more tasks. Similarly, we can show it with multiple models, other than LLaMA-Vid.

## Appendix C Broader Impacts

Our paper talks about how behavioral training can positively impact content understanding of VLMs. We think this will be useful in various content understanding applications such as question answering, captioning, etc.

## Appendix D Ethical Implications

This paper demonstrates that training on behavioral modalities enhances the learning of content modalities. Models trained on user interactions such as comments and likes were tested on multiple benchmarks and yielded positive results. While these findings present exciting opportunities for advancing content understanding in AI systems, they also raise important ethical considerations that must be carefully addressed.

1. No personally identifiable information (PII) is used to improve content understanding. Instead, aggregated behavioral data, including replays, likes, and comments, is utilized, ensuring user privacy. We have implemented rigorous anonymization and aggregation techniques to protect individual user identities.

2. We explicitly acknowledge the inherent biases that may exist in data sourced from social media platforms such as Reddit and YouTube. Despite our rigorous data filtering and cleaning processes, we recognize that the dataset may still be subject to demographic skews, self-selection bias, and algorithmic influences. These biases could potentially lead to uneven model performance across different user groups or reinforce existing societal biases.
To mitigate these issues, we emphasize the importance of considering these broader implications when applying our model and interpreting its results.

3. We acknowledge the need for greater cultural diversity in our dataset. To address this, we plan to release our artifacts as open-source and encourage community contributions to incorporate multilingual and multicultural data. This could involve expanding the range of subreddits and YouTube channels included in our dataset, with the aim of capturing a more diverse and representative spectrum of receiver behavior across different cultural contexts.
By taking this approach, we hope to enhance the ethical considerations and societal impact of our work, providing a more holistic view of behavioral patterns in various cultural settings. However, we also recognize that this approach may introduce new challenges, such as ensuring the quality and reliability of community-contributed data.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: NA
soundness: 3
strengths: 1. The paper is well-motivated, highlighting the importance of leveraging user behavior data for content-understanding tasks.
2. It contributes a large-scale dataset BLIFT, consisting of 400k images and 330k videos, along with their receiver behavior, which is beneficial for research about vision language models.
3. The proposed training method has gained significant performance on a variety of vision-language tasks, which shows the effectiveness of the method.
summary: The paper proposes that using user behavior data collected from Reddit and YouTube helps large vision and language models achieve better performance across a wide range of language and visual content understanding tasks. The gains are especially more salient on the higher-level tasks such as emotion recognition, persuasion strategy classification, and question answering than the lower-level tasks like action and object
recognition. Furthermore, the ablation study demonstrates the importance of behavior data and instruction fine-tuning on behavior data.
weaknesses: More details of the training process can be provided for reproducibility, e.g. the average length of the text input, and how to deal with extremely long ASR results.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 2
questions: To the best of my knowledge, platforms such as YouTube and Reddit have stringent Terms of Use requiring how such behavioral data can be used and released as a dataset. How did the authors work together with the platform to ensure that collecting and releasing this data does not breach their terms of use?
soundness: 4
strengths: 1.	This proposed idea of using receiver feedback relating to images/videos is novel to the best of my knowledge (which should been taken with a pinch of salt since the reviewer mostly work on text-only alignment) and is certainly worthwhile exploring since the cost of using these receiver-behavior is essentially zero compared to how much it would cost for post-collection expert human annotations.
2.	The data collection and preprocessing from Reddit and YouTube are detailedly described and carefully thought-out. I’m especially impressed by the extensive data cleaning done by the authors as well as the multitude of ways that the content behavior data can be converted to instruction tuning data.
3.	The training setup designed by the authors seem reasonable, with well-designed ablations such as matching for additional data (Ad-LLaVA) 
4.	The evaluation is done rigorously with 46 different tasks and general improvements across many of these metrics, in relation to strong baselines.
summary: Communication consists of not only the message but also the effects it has on the receiver. However, receiver behavior is commonly ignored when training vision language models. This paper shows that training VLMS on receiver behavior such as likes, comments and replay graphs improves their performance across 46 diverse tasks. Moreover, this data is commonly collected by default (through user interactions on Reddit and YouTube) and hence does not require additional annotation. They collect and release a large scale 730k-sample dataset containing this data.
weaknesses: 1.	The introduction claims that “behavior data is considered noise and is ignored while training large language models” (Line 50). However, this statement ignores the substantial work done in harnessing receiver-generated behavior data especially on online forums such as Reddit (through upvotes and comments), which have been used by many works including [1] and [2]. Nonetheless, I have not heard of VLMs using organic/behavioral feedback data so I would suggest the authors to acknowledge that prior work have been done for text-only settings and only claim that this is done in this paper for VLMs, which is more appropriate to the remainder of the paper.
2.	The organization for Section 3 Results and Discussion can be improved. Currently, it’s not easy to follow the main areas of improvement of the proposed methods. Because there are 46 evaluation metrics, it might be easier to follow if the results section starts with one table that shows the average gain in each task category e.g. VQA, which can be done by normalize all scores for a baseline model (e.g. LLaMa-Vid) to 100%, with the raw metrics in appendix. I would also recommend that the discussion is organized more clearly (i.e. each paragraph has one main point summarized as the paragraph heading) as it currently reads as simply describing the results and it’s hard for readers to follow along on the main insights arising from the results. 

[1] https://arxiv.org/abs/2009.06978
[2] https://aclanthology.org/2022.wnu-1.3/

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Any analysis on the harmfulness/trustworthiness/bias on the generated content from behavior LLaVa? The collected dataset could contain these harmful content.
soundness: 3
strengths: 1. The constructed dataset could be very helpful for VLM researches.
2. The fine-tuned VLMs show improvement across various tasks.
summary: This paper demonstrates hat training VLMs to predict receiver behaviors, such as likes, comments, and replay graphs, which are available at scale, enhances the VLM’s performance across a broad range of downstream content understanding tasks. Next, this paper constructs a behavior-LLaVA instruction fine-tuning dataset, and obtain the fine-tuned Behavior-LLaVA and AdLLaVA. The experiments on the fine-tuned VLMs demonstrate the effectiveness of content and behaviour, and study the different effect of perception and action behavior.
weaknesses: 1. The idea of the connection between VLM performances and fine-tuning on behavior data, upon which the paper is built, is questionable. The authors try to draw some analogy from relevant studies in human behavior, but fail to make a clear point with the work done in this paper (fine-tuning VLMs on behavior datasets). Also, the logic in this paper lacks clear or formal theoretical analysis on why fine-tuning VLMs on behavior datasets could improve performance, but rather just some abstract references to some concepts in the human behavior research. In this respect, the logic foundation of this paper is questionable and unconvincing.
2. The study of the connection between LLM/VLM foundation models and human behaviors is not novel. In fact, many researches [1][2][3] have been performed from various different perspectives, such as social behavior and trust behaviors. If we are to view the study on the connection of behavior with VLMs as a contribution in this paper, there are already many researches delivering similar and more comprehensive conclusions on this topic. In this sense, the authors should clearly position this paper along the research line and clearly state the difference between other related works.
3. The conclusion that learning behavior leading to learning content better is quite trivial and might not be very useful. In fact, adding more fine-tuning data of any types could lead to performance improvement on some relevant tasks. If the authors really want to make this point, they should curate the same dataset without explicitly telling VLMs what the behavior is and explore the performance of the VLM fine-tuned on this dataset.

References:

[1] Xie, Chengxing, et al. "Can Large Language Model Agents Simulate Human Trust Behaviors?." arXiv preprint arXiv:2402.04559 (2024).

[2] Li, Yuan, Yixuan Zhang, and Lichao Sun. "Metaagents: Simulating interactions of human behaviors for llm-based task-oriented coordination via collaborative generative agents." arXiv preprint arXiv:2310.06500 (2023).

[3] Chen, Weize, et al. "Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors." The Twelfth International Conference on Learning Representations. 2023.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Did the authors consider the issue of distributional bias during data collection, and how might this impact the downstream tasks? The extent of data acquisition is also an important aspect of quality assessment. While collecting large-scale data, it's essential to consider the characteristics of the data being collected. Besides YouTube or Reddit, have other data sources been considered? In addition, please address the legality of your data acquisition and processing.

2. Please provide more experimental details, including experimental design and setup. Additionally, only the source code for llama-vid is provided—have any model improvements beyond fine-tuning been considered? The supplementary materials only include the llama-vid code, with the only visible data being a small amount of Reddit data.
soundness: 2
strengths: 1. The paper presents an innovative perspective by examining large models' understanding of data through the lens of human behavioral performance.

2. The authors have compiled a substantial multimodal dataset and fine-tuned the model from the perspective of behavioral performance, demonstrating impressive results in downstream tasks.
summary: 1. This paper proposes the use of behavioral data training to enhance the content understanding capabilities of multimodal models and has collected a cross-modal dataset called BLIFT.

2. Building on LLaMA-Vid, this paper applies two different instruction fine-tuning strategies, Behavior-LLaVA and Ad-LLaVA, on the BLIFT dataset. The Behavior-LLaVA strategy leverages behavioral data, making it stronger than the Ad-LLaVA strategy, and demonstrates superior performance compared to baseline models across multiple multimodal downstream tasks.
weaknesses: 1. Lacks sufficient information on data collection, processing, and distribution. The description of the data collection and filtering process lacks logical coherence. The filtering appears to be largely empirical, and there is no thorough analysis of the feature distribution of the collected data.

2. The paper fails to provide information about the specific experimental details used for instruction tuning (including in the appendix) and does not compare the changes in time efficiency resulting from tuning with additional commentary data versus without.

3. The paper states that sensor data is inferior to behavioral data based on the insufficient quantity of sensor data. It could be argued that sensor data is challenging to collect; however, comparing the quality of sensor data requires more substantial evidence.

4. In this paper, "Ad-LLaVa" sometimes appears as "AdLLaVa." Please ensure consistency in spelling.

Meta-review: This paper proposes leveraging large-scale user behavior data to enhance VLM performance on a broad spectrum of downstream tasks. The novel BLIFT dataset and the instruction-tuned Behavior-LLaVA and Ad-LLaVA models show promising gains, particularly on complex tasks like emotion recognition and persuasion strategy classification. Reviewers appreciate the paper's strong motivation, extensive data cleaning, and thorough evaluations across 46 tasks. Though some note that theoretical foundations and comparisons to related work in social and behavioral modeling could be more explicit, the consensus is that this approach is a meaningful contribution. The authors offer a new resource and a perspective that may inspire future work on integrating behavioral cues into VLM training. Given these strengths and the author's willingness to refine and release resources to the community, I recommend acceptance.

Decision: Accept (Poster)