Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior
Authors: Ashmit Khandelwal
Aditya Agrawal
Aanisha Bhattacharyya
Yaman Kumar
Somesh Singh
Uttaran Bhattacharya
Ishita Dasgupta
Stefano Petrangeli
Rajiv Ratn Shah
Changyou Chen
Balaji Krishnamurthy
Venue: ICLR 2024 spotlight (2024)
Keywords: large language models
behavior simulation
large content and behavior models
behavior understanding
behavior in the wild
computational marketing
computational behavior science
Primary area: generative models

Abstract: Shannon and Weaver's seminal information theory divides communication into three levels: technical, semantic, and effectiveness. While the technical level deals with the accurate reconstruction of transmitted symbols, the semantic and effectiveness levels deal with the inferred meaning and its effect on the receiver. Large Language Models (LLMs), with their wide generalizability, make some progress towards the second level. However, LLMs and other communication models are not conventionally designed for predicting and optimizing communication for desired receiver behaviors and intents. As a result, the effectiveness level remains largely untouched by modern communication systems. In this paper, we introduce the receivers' "behavior tokens," such as shares, likes, clicks, purchases, and retweets, in the LLM's training corpora to optimize content for the receivers and predict their behaviors. Other than showing similar performance to LLMs on content understanding tasks, our trained models show generalization capabilities on the behavior dimension for behavior simulation, content simulation, behavior understanding, and behavior domain adaptation. We show results on all these capabilities using a wide range of tasks on three corpora. We call these models Large Content and Behavior Models (LCBMs). Further, to spur more research on LCBMs, we release our new Content Behavior Corpus (CBC), a repository containing communicator, message, and corresponding receiver behavior (https://behavior-in-the-wild.github.io/LCBM).

Published as a conference paper at ICLR 2024
LARGE CONTENT AND BEHAVIOR MODELS TO UN- DERSTAND, SIMULATE, AND OPTIMIZE CONTENT AND BEHAVIOR
Yaman K Singla" +
IIID
Ashmit Khandelwal
Aditya Agrawal*
Aanisha Bhattacharyya'
A
Somesh Singh
Uttaran Bhattacharya
Ishita Dasgupta
Stefano Petrangeli
IID
Rajiv Ratn Shah
Changyou Chen
Balaji Krishnamurthy
M
Adobe,
BITS, Pilani.
III
DeT-Delii,
State University of New York at Buffalo
ABSTRACT
Shannon and Weaver's seminal information theory divides communication into three levels: technical, semantic, and effectiveness. While the technical level deals with the accurate reconstruction of transmitted symbols, the semantic and effec- tiveness levels deal with the inferred meaning and its effect the receiver. Large Language Models (LLMs), with their wide generalizability, make some progress towards the second level. However, LLMs and other communication models are not conventionally designed for predicting and optimizing communication for de- sired receiver behaviors and intents. As result, the effectiveness level remains largely untouched by modern communication systems. In this paper, we intro- duce the receivers' "behavior tokens," such shares, likes, clicks, purchases, and retweets, in the LLM's training corpora to optimize content for the receivers and predict their behaviors. Other than showing similar performance LLMs on content understanding tasks, our trained models show generalization capabilities on the behavior dimension for behavior simulation, content simulation, behav- ior understanding, and behavior domain adaptation. We show results all these capabilities using wide range of tasks on three corpora. We call these models Large Content and Behavior Models (LCBMs). Further, to spur more research on LCBMs, we release our new Content Behavior Corpus (CBC), a repository containing communicator, message, and corresponding receiver behavior
INTRODUCTION
Vetta oue Corket
Image
Behaavoor
and uhic ios?
Varaee
Simulation
Ai
Simulation
Content
Fmy berget engreeet new panents how can mosilly my get thruugh
Content-Behavicn (LCBM)
Large
refter moche
Whits
Content
WWatt ia the octor of the wall behind man??
Eriyy res to and
Understanding
nomator say
gets
abora
Understanding
Behavior
100k reveed
dick on 87
why 503 people
The will chesen burper gatx shown dutin morning time. People ueuulily
prnter henttier optimee dusing marning
Figure I: Encoding and predicting content (images, videos, and text) and behavior the language space. Large Content Behavior Models (LCBMs), once trained, can enable a host of different applications, including behav- ior simulation, content understanding, content-behavior optimization, and content-behavio understanding.
* EEqual Contribution. Contact ykumar @c admbe.com for questions and suggestions.
Work done while at Adobe. hittps://behavee the-wild wi www...giihhii...... thub. 10/LCBM

---

Published as a conference paper at ICLR 2024
Soorea
0913
de daceomy
Eintton
den Jalee
Strelog
sasing
poonsded
usod
RS
(a)
(b)
(c)
(d)
Figure 2: Comparison GPT-3.5, GPT-4, Vicuna-13B. and LCBM-13B on: (a) Behavior Simulation accuracy on two types of behaviors: replay value prediction and likes/views prediction. The task is, given the video content and channel information, to predict replay values corresponding to each scene and the ratio of likes to views. (h) Content simulation and hehavior understanding tasks. The task for content simulation is, given the channel information and scene-level behavior, to predict the scene content. Given information platform and the video content, the task behavior understanding is to predict and explain the sentiments of the viewers and the commenters. Six evaluators scored the models' explanations between 0-5 to get the predicted sentiment and explanation scores by comparing the ratings and reasons with the user comments. The annotatoos did not know which model gave the reasoning. (c) Content understanding tasks. We evaluate four tasks: emotion, topic, and persuasion strategy prediction, and
the video
understanding. (d) Behavior Simulation on in-house Email Marketing Data (R² score) and Twitter likes (accuracy), and Content Simulation on Twitter tweet prediction (BLEU/ROUGE scores), It can be noted that on the behavior simulation, content simulation, and behavior understanding tasks, LCBM performs better than 3-shot GPT-3.5 and 10-shot GPT-4 (covering a larger area. On the content understanding tasks, while LCBM outperforms similar-sized Vieuna models, GPT-3.5 performs better. However, we also note that GPT-3.5 and GPT-4 are at least times larger than LCBM- 13B. Further, we show the behavior domain adaptation results in Table 6
g
Shannon & Weaver (1949). in their seminal paper on communication, includes all of the proce- dures by which one mind may affect another. This includes all forms of expression, such as words, gestures, speech, pictures, and musical sounds. They mentioned that the broad problem of commu- nication can be studied at three levels: technical, semantic, and effectiveness.
Level A: Technical. How accurately can the symbols of communication be transmitted?
Level B: Semantic. How precisely do the transmitted symbols convey the desired meaning?
Level C: Effectiveness. How well does the received meaning induce the desired conduct in the receiver?
These three levels build on top of each other. Thus, solving the problem at Level C. necessarily requires solving the corresponding problems at Levels A and B.
Since the publication of this seminal paper, the tremendous growth in the field of telecommunica- tions, particularly the advent of the Internet and mobile devices, has led to affordable, wide-seale
solutions for Level A. With the recent advances in large language models (LLMs) such as BERT (Devlin et al.] 2018), GPT-3 and 4 (Brown et al. 20200 OpenAl| 2023). T5 (Raffel et al. 2020). and many more, we have witnessed significant improvement in the performance of various Natural Language Processing (NLP) tasks. I.MSs in zero- or few-shot settings can easily handle tasks such as question answering, summarization, translation, and many more. This has helped us progress towards solving Level B to a large extent. However, the Level C problem of effectiveness remains largely unsolved. Effectiveness refers to designing messages that can fulfill the communicators' underlying objectives, such as explaining complex concepts to the receivers and informing the re- ceivers' choices (e.g., when making purchase decisions).
How do we solve the effectiveness problem while retaining the other two levels? To solve the effectiveness problem, we can take inspiration from how the semantic problem is being solved. Raa- fel et al. (2020). in their seminal work on T5, mention that the basic idea underlying large language models is to treat every text processing problem as a "text-lo-text" problem, i.e., taking the text as input and producing new text output. This framework allows for direct application of the same model, objective, training procedure, and decoding process to every task we consider. Further, this allows us to pre-train model on a data-rich task like the nort-word prediction, which can then be transferred to downstream tasks. Notably, thanks to the Internet, the next-word prediction task has huge amounts of available data. Consider the Common Crawl project (hupss//commoncrawl..ogg
2

---

Published as a conference paper at ICLR 2024
conteciin norr ground playerty crass
Missagge
doene zoplayei
soutsbe
Communccoror
Regsiver Getacmaberer Goneral Soutuke
Nike b/neeelbrrst 15sr
Channel
Nodionno
Figure 3: Five factors of communication: Communicator, Message, Channel, Receiver, and Effect,
one common source of data included in most language models. It produces more than 20TB of text per month sampled from random web pages across the internet.
T5 and other language models like GPT-3, Pythia (Biderman at. 2023 and Llama Touvron et al., 2023) can solve a wide variety of tasks, including the ones for which they were not explicitly trained. For instance, language models trained pabilities across a wide variety of tasks like question-answeringg summarization, natural language inference, and translation (Brown et al., 2020 Recently, a series of papers have shown that this
the next word prediction task showed generalization ca-
generalized "world understanding" captured in LLMs can be leveraged to enable them to "see" (Liu
al. 2023 Li et al. 2023b aj Zhu et al. 2023 Ge et al., 2023 Zhang al. 2023 Bhattacharyya
et al. 2023). This is significant capability enhancement since model trained in language only
settings can be made to reason about images and videos. These papers follow the same transfer learning approach advocated by T5, where they convert visual information to language space to leverage the "text-to-text" framework. They show that is possible to teach a large language model, the new modality of vision, without needing to pre-train the model from scratch. Rather, using only a few million tokens, it is possible to scale LLMs' abilities to vision as well. Following this chain of thought, it could be possible to solve the effectiveness problem by posing it "text-to-text" problem. This is the paradigm we explore in this work.
How can pose the effectiveness problem as a text-to-text problem? The problem of effect is to know what the receiver does after receiving the message (Shannon & Weaver 1949). In general, for piece of content, other than the content itself, often have information about who consumes the content and what his action is on consuming the content. The latter is the effect described in Shannon's 's three levels of communication. For instance, an email, as a message from the communi- cator to the receiver, clicits certain actions from the receiver like link-clicks, replies, and read-time. While LLMs are trained on trillions of tokens of content, the training does not include the receiver effect. For instance, Enron Email (Klimt & Yang 2004) is popular corpus that is included in the training of LLMs like Pythia (Biderman et a (023). contains 600K email content sourced from the Enron corporation, which LLMs use to learn how to write emails. However, it does not contain data about the receivers' activities, such as whether they opened the email, how long they kept open (read-time), and what their reply was. Similarly, while major text corpora include a large num- ber of public blogs and user forums to train LLMs like CommonCrawl. they are stripped of receiver behavior on forum messages, such the number of likes, shares, and replies, before including them in LLM training (for instance, see (Biderman et al.) 2022 Penedo et al. 2023)). To pose the effec- tiveness problem as a text-to-text problem, we can include these behavior tokens in the text along with content tokens and train the LLM to model both of those in the same space. This might help the LLM simulate the receiver effect, optimize for it, and reason about it.
In this paper, we show initial experiments to integrate behavior as a new modality to increase the scope of multimodal LLMs from only content both content and behavior. We call this new type of model a Large Content Behavior Model (LCBM). This class of models shows promise in en- abling the I.LMs to not only reason about content but also reason about and predict human behavior over that content. Further, LCBMs have the potential for behavior domain adaptation where models trained on one type of behavior can generalize on another behavior type (Fig, 2). Behavior simula- tion can enable many real-world applications, such content recommendation, customer journey optimization, and A/B testing. To build LCBM, we introduce behavior instruction tuning ($2.4) an attempt to extend the instruction tuming paradigm to behavior space, bringing all five communi- cation factors (communicator, message, channel, receiver, and effect) into the same space (Fig. 3 Similar to Brown al. (2020): Raffel al, (2020); Liu et al. (2023): Gie al. (202)), we do not
3

---

Published as a conference paper at ICLR 2024
design best-in-class predictors for any of the downstream tasks. Rather, show model which shows generalization capabilities across a wide variety of content- and behavior-related tasks. To summarize, our paper makes the following two contributions:
Large Content Behavior Model (LCBM). We develop large multimodal model that shows capabilities of behavior simulation (given content), content simulation (given behavior), content understanding, and behavior understanding (Fig. Following the text-to-text framework, connect the Vicuna LLM (Tourron et al., 2023 Chiang et al.) 2023) with an open-set visual encoder of EVA-CLIP (Sun et al.) 2023) and instruction fine-tune it end-to-end on behavior in- struction data. EVA-CLIP and QFormer (Li et al., (023a) help the model to understand visual content in the language space, making it Vision Language Model (VLM). During behavior in- struction tuning. we teach the model to predict behavior given content and content given behavior using various instruction tasks ($2.4) This helps us teach behavior modality to the VLM while grounding it in the natural language space. We use three datasets to show the performance of LCBM: a dataset consisting of YouTube videos as the content and the corresponding retention graph, likes, the number of views, and comment sentiment as receiver behavior; a dataset consist- ing of Twitter posts (text, images, and videos) and corresponding human behavior (like counts) extracted from 168 million tweets across 10135 enterprise Twitter accounts from 2007 to 2023 Khurana et al. (203bb); and an internal dataset of In-house Marketing Email (content) and the tlick-through rate corresponding to each segment they were sent (behavior). We observe that teaching the LCBM behavior and content simulation improves its capabilities on them (expected), but the model also shows signs of domain-adaptation in behavior modality (few-shot capability, unexpected) (Tables 6 and improvements in behavior understanding (Figs, 5 3 (zero-shot capability, wnexpected) Brown al. 2020). See Fig. [ a radar plot of all the capabilities and comparisons of performances across LCBM and of-te-of-the-ar LLMs: GPT-3.5 and GPT-4
Dataset and Test Benchmark. To spur research on the topic of large content and behavior models, we release our generated behavior instruction fine-tuning data from over 40,000 public- domain YouTube videos and 168 million Twitter posts. The data contains: 1) YouTube video links, automatically extracted key scenes, scene verbalizations, replay graph data, video views, likes, comments, channel name, and subscriber count at the time of collection, and 2) Twitter ex- tracted account names, tweet text, associated media (image and video) verbalizations (including image captions, keywords, colors, and tones), tweet timestamps, and like counts Khurana et al. (2023b). We also release benchmark to test performance on the joint content behavior space (2.3)) introducing two types of tasks in this space: predictive and descriptive. In the predictive benchmark, we test the model's ability to predict behavior given the content and predict content given the behavior. In the descriptive benchmark, we validate its explanation of human behav- ior by comparing it with ground-truth annotations we obtain from human annotators that try to explain human behavior. See Figs. 6 for few examples.
2
SETUP
In this section, we introduce our approach to model content and behavior together as text-to-text problem. Since most publicly available corpora strip off receiver behavior from content, we first introduce our dataset, "The Content Behavior Corpus (CBC)", dataset consisting of content and the corresponding receiver behavior. Next, we introduce our methodology to convert the content and behavior into text and our approach to model it using LLM. Then, we cover the tasks through which we test various capabilitiess of LCBM (Fig. [ content-understandiig behavior understand- ing, content simulation, behavior simulation, and behavior domain adaptation.
2.1 THE CONTENT BEHAVIOR CORPUS (CBC)
The availability of large-scale unlabeled text data for unsupervised learning has fueled much of the progress of LLMs. In this paper, we are interested in modeling content and the corresponding receiver behavior in the same space. While available datasets contain trillions of content tokens (text, images, audio, and videos), they unfortunately do not contain the receiver effect. To address this, we utilize YouTube and Twitter, two large publicly available sources of content-behavee data,
We obtain In-house Marketing Emails dataset by collaborating with the In-house team.
4

---

Published as a conference paper at ICLR 2024
GMHRA
Enteddigg
Visual
Encoder
QFormer
é Large
pedter rates
Content Behavior Model
will likve 2.35
Visual
Content
Veraalbe Resalier A-lonn
Content
Behavior
Figure 4: Encoding and predicting content (images, videos, and text) and hehavior in the language space. Strat- egy hehavior instruction fine-tune (BFT) LLMS to create ICBMMs. We capture visual concepts through the visual encoder (EVA-CLIP) and world knowledge is through an L.M (Llama). To leverage the rich knowl- edge of LLMs. we use GMHRA QFormer to convert visual tokens of VIT to language tokens that Llama can understand. Further, we find that verbalizing the visual stimulus helps Llama to gather information more explicitly than what is provided by +Fr+eFrrm We fine-tune the combined model end-to-end to predict 1) behavior given content and 2) content given behavior. Snowflake and fire symbols denote the frozen and unfrozen parts of the architecture.
consisting of (a) account name, account description, and number of subscribers and followers (com- municator data). (b) rich content in the form of videos, images, creator prrviidpr captions, titles, and descriptions (message), (c) behavior in the form of likes, views, comments, and replay graph (receiver effect). This covers all the five factors communication (Fig. B). with the channel being fixed (as YouTube Twitter) and receivers being average channel followers and viewers of the communicator. Since content data multimodal in the form of a combination images, videos, and text, and behavior data is in the form of numbers, to model it using text-to-text paradigm, verbalize both of them following the methodology we detail next.
Verbalization: For the video V, YouTube provides us with 100 average viewer retention values for i E 00., corresponding to the entire video. The sampling rate of 100 is constant and inde- pendent of video length (I)). Replay value r; corresponds video frames between the timestamps (T/100 x i, 1100 x (i 1)). which denotes how often these frames were replayed compared to the most replayed frames. The metric has value between 0 and I that identifies the video's 's relative re- tention performance at given point in the video. To accommodate longer video lengths, we merge replay values until T/100 100 x (i (i+)) j) - T 100 > second with j E {i + 1, 1003. We choose the replay value for this merged group of scenes as maz(ri, .... Tz). Using this logic, we get replay values R2 for [0...r]. where m = [100/([ ([100(())))) Next, we sample two frames randomly corresponding to each i E [0..m]. We caption the frames using BLIP (Li al. 2023a). We also obtain the automatic speech recognition for the speech for the video between the timestamps corre- sponding replay value R& using Whisper (Radford al. 2023). The ASR and BLIP captions content for scenes, and replay values are the behavior corresponding to them. We include the scene content and behavior in the video verbalization (Listing) with the sampling for both scene content and behavior as described above.
We also include video content by encoding video frames through EVA-CLIP (Sun al., 2023) (explained in 22.2). Other than video embeddings, we include the video title and description as part of the video content. Corresponding to the overall video content, we verbalize overall video behavior metrics like video views and the ratio of likes and views. Finally, we append it with communicator information on the video channel and the subscriber count. The Listing | presents the overall verbalization for video and frame level content and behavior. The verbalization for Twitter posts is similar and is given in Listing 7
Listing Verbalization pattern for inputting content and behavior in the same space
Input: <vvidco>> [Vidoo Tokens] <VVidcoo>>
The video has the following sccnes:
Scene (ASR: Welcome quick tutorial OCR: Adobe Premiere Pro, Captions: desktop interface Replays: 60),
Scene (ASR: on using Premiere Pro to edit, Captions: A computer interface with an image of a white horse. Ohjects Horse,
Grass, Fence., Replays: 53).
was posted on Adobe's YoueTbb channel with channel has 100k subscribers. This video viewed by 346 thousand people and liked (as pereentage of likes /views) by 2.3% people.
"Using Premiere Pro a on Aug Adohe's YouTuhe
5

---

Published as a conference paper at ICLR 2024
Model
&Parans
Training
Past
Future
Randon Window Size
All Masked
RASE
Accuracy
Aveurncy
Accuracy
Aecuracy
LOBM
3-DTT
8.12
55,10
15.05
42,42
3.55
61.41
9.91
55.10
LCBM
15B
BET
11.53
52.06
12.02
53.06
83.13
64.83
9.22
60.26
31.34
17.16
LCBM
7-RFT
16.17
35.61
15.14
44.11
9.02
59.22
10:47
$3.84
LCRM
11-FTT
18:25
30.95
15.05
41.44
10.01
55.15
10:49
52.1
GPT4
GPT4
1000
10-hhot KCL
35.05
34.45
20.55
19.34
19.51
18.07
36,08
59.31
22.99
17.42
26.99
18.10
21
27.25
28.65
17:27
37.60
38,52
14,26
13.73
GPT33
1758
34.10
24.71
27.14
24.11
26.30
18.74
13.47
30.10
1070
54.0
10.00
1000
1000
Table I: Behavior Simulation. Mean RMSE and accuracy scores for scene-by-scene predictions of video replay values. Replay values are the normalized replay scores each scene as provided by YouTube. The normalized scores are considered to 2 decimal places and multiplied by hundred to convert the score to an integer score in the range 0-100. RMSE is calculated for each video in the test set and the mean is calculated for this score and reported. The model is said to classify correctly if the absolute error between the predicted and ground truth value is less than or equal to 5. The scores are calculated four regimes: past, future, random, and all-masked. In the past (future) regimes, first (last) 5-20% scenes are masked; in the random setting. 5-20% scenes are masked randomly, and in all masked setting, everything is masked. LCBM was bchavior-finc-tume (BFT) with 3,5,7,11 context window masking strategy, while GPT was compared with an in-context learning (ICL) setting. We note that behavior fine-tuned LCBM, while being least 10x smaller than other models, performs the best. Best models are denoted in green and runner-ups in blue..
2.2 MODEL
To understand both visual and textual contents, we follow a similar approach as was taken by recent models like BLIP, Llava, VideoLlama, and others (Liu al. 2023 Ge et al.) [023 Li et al. (203a Zhu et al. 2023). we use visual encoders to encode visual knowledge and an LLM to encode text and world knowledge. Fig. 4 shows our architecture to encode visual content into the language space. We include video content by encoding video frames through EVA-CLIP (Sun et al., 2023) and Global Multi-Head Relation Aggregator (GMHRA) from Uniformer eLi at. 2021). GMHRA helps aggregate the information better across the time dimension. The combination of ViT and GMHRA gives us a good representation of the visual content. Next, to effectively leverage the LLM's rich language representations, we use Q-Former from BLIP-2 (Li et al., (023a) with an extra linear layer and additional query tokens to convert from visual tokens to language tokens. Further, similar to Bhhttaccarrya al. (2023), we find that while encoding visual tokens is powerful, converting visual content to text adds to the downstream performance. Therefore, we include the BLIP caption for each scene along with the scene replay graph.
We use the Llama-base Vicuna- 13B LLM (Touvron et al. 2023 Chiang et al. 2023) as our base
L.M.. Similar to prior works (Liu et al. 2023 Ge al. 2023 Li et al.) 2023a Zhu et al. 2023)
we follow a two-stage training paradigm where in the first stage, we utilize the Web (Bain et al., 2021). COCO caption (Chen al., 2015). Visual Genome (Krishna al., 2017), CC3M (Sharma et al. 2018), and CC12M (Changpinyo et al. 2021) datasets to align the visual encoder embeddings with LLM. In the second stage, we train the model with behavior instructions prepared by following the approach described in $2.4 In summary, LCBM takes concatenated inputs of visual tokens, scene ASR, caption, scene behavior of replays, channel information, and video title and behavior metrics of views and ratio of likes to views. Based on the instruction, we test LCBM's abilities on various tasks we cover the next paragraphs.
2.3 CONTENT BEHAVIOR TEST BENCHMARK
We test the capabilities of large content-behavion models on predictive and descriptive abilities on content and behavior, as illustrated in Fig: [ We design the following five tasks to test these pabilities: behavior simulation, content simulation, content understanding, behavior understanding, and behavior domain adaptation. We cover each of these tasks next.

---

Published as a conference paper at ICLR 2024
1. Behavior Simulation. We test simulation capability four behaviors across two datasets: YouTube replay values, the ratio of YouTube likes to views. Twitter likes, and the number of views of the YouTube video. The common task amongst all of them is to predict the behavior given the content and content attributes like captions, scene-by-scene descriptions for videos, and sender characteristics like account and subscriber count and date of posting. The behavior to be predicted is masked and asked as a question to the LLM. Listings [6] and lists the verbalization pattern for this task. For replay value prediction, we test the masked behavior in three settings: Masked Past (all replay values of the first 5-20% scenes are masked), Masked Future (all replay values of last 5-20% scenes are masked), and Random Masks (random masking of replay values for 5-20% scenes).
Content Simulation. Here, the task is to predict content given receiver behavior (Listing 5 For YouTube, given the video content in terms of scene-by-scene descriptions with the content of one group of five consecutive scenes content being masked, behavior values of all scenes, and channel information, the task is to choose the masked scene speech from a list of 25 options, chosen randomly from the entire test set. For YouTube, we chose to model this task as a dis- criminative task instead of generative one since videos are generally long, and there could be multiple possible contents for a given behavior, whereas the ground truth is available only for one specific characterization of the content for a given behavior. For Twitter, we model this task as content generation. The Listing 8 presents the format for this task.
Behavior Understanding. The goal of this task is to check if the model can reason about ob- served or unobserved receiver behavior. For this task, we could ask the model to explain any behaviors given the content. However, only the YouTube receiver comments have ground truth available with the video. Without ground truth, we found that other behaviors, such as replay values, likes, and views, are difficult to explain by non-experts. Therefore, we ask the model to simulate the sentiment of the receivers' comments and describe its reasoning. To evaluate, we asked six annotators to annotate the reasons provided by the model on a scale of 0-5, with O implying the LLMs provided no sentiment reasoning and 5 implying perfect reasoning. The annotators were free to rate the LLMs as they seemed fit. The annotators were asked to review the video content and the comments to help them evaluate the reasons. We average the ratings of three annotators to get an average rating for every video. Similarly, to review the sentiment correctness, we asked the annotators to judge the predicted sentiment rating with respect to user comments,
Content Understanding, To check if a model trained on both content and behavior tokens does not forget its original content understanding capabilities. we test the content understanding tasks on YouTube videos, following [Bhattachrryya et al. (0023). They use the following tasks for video-understandinn:: topic, emotion, persuasion, and action-reason classification. For topic, emotion, and action-reason classification tasks, they use the advertisements dataset by Hussain eet al. (2017), which contains 3,477 video advertisements and the corresponding annotations for emotion and topic tags and action-reason statements for each video. There are a total of 38 topics and 30 unique emotion tags per video. Further, we have 5 action-reason statements for each video for the action-reason generation task. For our experiment, we use the subset of 1,785 public videos. Following [Bhattacharyya et al.] (2023), for the topic and emotion classification task, we evaluate our pipeline using top-1 accuracy as the evaluation metric. Further, we evaluate emotion classification on clubbed emotion labels as well. For action and reason prediction, we evaluate our accuracy on the action and reason retrieval tasks where 29 random options along with 1 ground truth are provided to the model to find which one is the ground truth. In the per- suasion strategy classification, we use the 1002 persuasion strategy videos and corresponding labels released by |Bhattacharyya et al. (2023). Given the video, the model has to predict which persuasion strategy the video conveys. Persuasion strategy classification could be important task for evaluating LCBM since the concept of persuasion in psychology views human commu- nication as the means to change the receiver's 's beliefs and actions (i.e., persuade) (Kumar et al.; 2023). and understanding the different strategies present in communication may help understand human behavior better. We evaluate the top-1 accuracy of the model on this task.
5. Behavior Domain Adaptation. In the past work, we have observed strong generalization ca- pabilities from I.MMs (OpenAl 2023] Ouyang al., 2022 Raffel et al. (020). While training on next token prediction. LLMS show generalization across tasks, including question answer- ing, natural language inference, and sentiment analysis. Given this, the natural question is, does LCBM, too, show this kind of generalization, where a model trained on one kind of behavior, can show performance on another behavior? To understand this, we test the model on a different
7

---

Published as a conference paper at ICLR 2024
dataset and task than what it was originally trained for. We do this over three datasets, LVU (w & Krahenbuhl 2021). In-house Email Marketing and generalization between Twitter and YouTube likes.
LVU Benchmark. [W & Krahenbuhl (2021) released benchmark for long video under- standing with over 1000 hours of video. the benchmark, they have two behavior related tasks: ratio of likes to likes+dislikes and view prediction. YouTube has discontinued the dislike count, therefore, our corpus does not contain the dislike count. We use the LVU test benchmark to check if model trained on other available behaviors (views, likes, and replay graphs) is able to predict the like ratio.
In-house Email Marketing. In this task, ask the model predict the click-through rate for a given target segment of an email, given the email content, subject, and verbalized descriptions of the images the email. We use the emails sent by In-house marketing team to its subscribers. The emails were sent from April 1, 2022 to June 12, 2023 and covered many of the premiere products The emails were sent to many customer segments (as defined by the marketing team) across 225 countries (Table 9 Listing 3 lists the verbalization format to verbalize emails to input to the LCBM.
2.4 BEHAAIIOR INSTRUCTION FINE-TUNING (BFT)
To teach LLM the behavior modality over multimodal content, convert both the visual tokens and behavior modality in the text format and instruction fine-tune the LLM end to end. This follows
two-stage approach: first, teach the LLM the visual modality ($ (2.2). and next, we teach the LLM the behavior modality. We call the latter "Behavior Instruction Fine- Fine-Tumine Tuning (BFT)" inspired by instruction fine-tuning (IFT) and its variants like visual instruction tuning (Liu et al.] 2023).
We prepare the content-behavior instruction datasets as explained next.
Teaching behavior in the forward direction (predict behavior given content): this instruction tuning task, we teach the model to predict behavior given the message sent by the communica- tor. Essentially, this teaches the model to predict behavior the forward direction (as Fig. 3 Concretely, we include the following information as part of verbalization image and video embed- ding converted to the text space (using EvaCLiP (Sun et al., 2023)), ), scene-by-scene verbalization covering automatic speech recognition, scene captions, video/post caption and description, receiver behavior covering replay rates, views, and likes, and communicator information covering account name and follower count. The verbalisation pattern for this task is the same given in the Listing
Teaching behavior in the reverse direction (predict content given behavior): This task teaches the model to learn about behavior in the reverse direction (Fig. (3). Here, the model learns to simulate content given behavior. The instruction for this task is given in Listing
Using the prepared content and behavior instruction datasets consisting of pairs of content and be- havior tokens, we treat the content tokens (Xc) as input and behavior tokens (X,,2; 24 E x)) as output of the language model. We then perform instruction-tuning of the LLM on the prediction tokens, using its original auto-regressive training objective. Specifically, for a sequence of length L, we compute the probability of generating target answers (X.) by:
L p(X n X = II po(a; Xc, N.
For the behavior instruction tuning, we keep the visual encoder weights frozen and continue to update the pre-trained weights of the LLM in LCBM.
RESULTS AND DISCUSSION
Here, we discuss the results for the five tasks we discuss in Section 2.3 namely, behavior simulation, content simulation, behavior understanding, content understanding. and behavior domain adapta- tion. We compare the behavior fine-tuned model discussed in $2.4 with state-of-the art content-only models like GPT-3.5, GPT-4, and Vicuna-13B. This allows us to compare how much including be- havior tokens in the training of an LLM helps in improving the LLM's understanding of behavior and joint content and behavior spaces while retaining its understanding of the content space.
8

---

Published as a conference paper at ICLR 2024
Model
#Params
Accuracy
Model
#Tarams
Sentiment Accuracy
Rewsoning Score
Vicuna
13B
19.30%
Vicuma
13B
65.66%
2.23
LCBM
13B
48.68%
LCBM
13B
72.73%
4.00
GPT-3.5
175B
34.98%
GPT-335
175B
61.62%
1.67
Random
4%
Table 2: Content Simulation. In this task, the models have to choose the speech segment from list of 25 options given the video description, non- masked scenes. and replay behavior. We see that despite being similar to masked language mod- eling (which is content-only task), LCBB per- forms better than both Vicuna and GPT-3.5. Best models are denoted in green and runner-ups in
Table 3: Behavior Understanding. In this task, the mod- cls have to simulate the sentiment of comments that a vidco would get by looking at only the vidco. Further, they also have to explain the reason for such sentiment.
The responses were annotated by humans on a scale of O-
5 for the reason, with 0 being no response provided and 5 being the response matches exactly with the (ground truth) comments received on the video. Best models are denoted in green and runner-ups in blue
blue
The results for the five tasks are presented in Tables 3 and We note few general trends. LCBM, while being 10x smaller than GPT-3.5 and 4, performs better than them on all behavior-related tasks. Further, we see that there is no significant difference between 10-shot and 2-shot GPT-4 or between GPT-3.5 and GPT-4, indicating that unlike other tasks, it is harder to achieve good performance through in in-context-learning on the behavior modality. It can be observed that often GPT-3.5 and 4 achieve performance comparable to (or worse than) random baselines. Interestingly, the performance of GPTs on the content simulation task is also substantially behind LCBM. The way we formulate the content simulation task (Listing 5). it can be seen that a substantial performance could be achieved by strong content knowledge, and behavior brings in little variance. We still see a substantial performance gap between the two models. All of this indicates that large models like GPT-3.5 and 4 are not trained on behavior tokens.
For the content understanding tasks (Table predictably GPT-3.5, being the largest model, achieves the best results. However, we see that BFT helps the LLM to learn most content understanding tasks better than the base LLM. LCBM gets better results than both Vicuna and VideoChat. This indicates that behavior modality might carry additional information about the content, which might help an LLM understand content better (Khurana et al., 2023a; Klerke et al., 2016 Plank (016). Next, we see that LCBM also shows signs of domain adaptation in the behavior modality. We see that on five tasks: comment sentiment prediction, comment sentiment reasoning (Table 33, email behavior simulation (Table/6), and Twitter behavior (Table and content simulation (Table(8). We note that the LCBM is trained on only email behavior simulation samples, it underperforms the model trained on both YouTube data and a few samples to make the model learn email format. Similarly, LCBM trained on both Twitter and YouTube performs better than the one just trained on Twitter, showing performance improvement by domain adaptation. Finally, Figs. 6 show a few samples where we query LCBM explain replay and comment behavior and compare it with human explanations. We see that LCBM while verbose, can explain behavior well.
4 CONCLUSION
In this paper, we make initial strides towards solving the effectiveness problem proposed by Shan- non in his seminal paper on communication. The effectiivnees problem deals with predicting and optimizing communication to get the desired receiver behavior. This can be seen as consisting of string of capabilities: behavior simulation, content simulation, and behavior domain adaptation. We show that while large language models have great generalization capabilities, are unable to perform well on the effectiveness problem. We posit that the reason for this could be a lack of "behavior tokens" in their training corpora. Next, train LLMs behavior tokens to show that other than content understanding tasks, the trained models are now able to have good performance across all the achavior-relatee tasks well. We also introduce new Content Behavior Corpus (CBC) to spur research on these large content and behavior models (LCBMs).
Note that we connot compare this model GPT-3 due the privati noture data.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Rajiv Ratn Shah is partly supported by the Infosys Center for AI, the Center of Design and New Media, and the Center of Excellence in Healthcare IIIT Delhi.
Changyou Chen is partially supported by NSF AI Institute-2229873, NSF RI-2223292, an Amazon research award, and an Adobe gift fund. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences, or the U.S. Department of Education.
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Steven C Bankes. Agent-based modeling: A revolution? Proceedings of the National Academy of Sciences, 99 (suppL3):7199-7200) 2002.
Aanisha Bhattacharyya, Yaman K Singla, Balaji Krishnamurthy, Rajiv Ratn Shah, and Changyou Chen. A video is worth 4096 tokens: Verbalize videos to understand them in zero shot. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 9822 9839, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 101188661122223 10.18653/1/0023.mmmllpp emnlp- main.608. URL//nttps://accanntoooog hrrp:/tthhhoooooor/22333333 emm.:.mmmppmmmmmmmmmmmmmmmpmmmmm emnlp-main main.
Stella Biderman, Kicran Bicheno, and Lco Gao. Datashect for the pile. arXiv preprint arXiv:2201.0711 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.
PMLR, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- ter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image- text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Ad- vances in neural information processing systems, 34:15084-15097, 2021.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint N04::1504.0032. 2015.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Erie P. Xing. Vicuna: An open-souree chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:://mmsss.orr/blll
3023-03-38-vicuma.
Jacob Devlin, Ming Bing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint /rXiv:/8/0004800. 2018.
Keyan Ding, Ronggang Wang. and Shiqi Wang. Social media popularity prediction: A multiple feature fu- sion approach with deep neural networks. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2682-2686, 2019.
Yuying Ge, Yixiao Ge, Ziyun 7eng, Xintao Wang, and Ying Shan. Planting seed of vision in large language model. arXiv preprint 2307 2023.
10

---

Published as a conference paper at ICLR 2024
Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong. and Adriana Kovashka. Automatic understanding of image and video advertisements, 2017.
Wang-Cheng Kang, Jianmo Ni, Nikhil Mchta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. Do llss understand user preferences? evaluating llms on user rating prediction. arXiv preprint arXiv:2305.06474, 2023.
Aditya Khosla, Atish Das Sarma, and Raffay Hamid. What makes an image popular? In Proceedings of the 23rd international conference on World wide web, pp. 867-8767 876, 2014.
Varun Khurana, Yaman Kumar, Nora Hollenstein, Rajesh Kumar, and Balaji Krishnamurthy. Synthesizing human gaze feedback for improved NI. performance. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1895-1908, Dubrovnik, Croa- tia. May 2023a. Association for Computational Linguisties. doi: 10.18655/w//0223.ccm main. URL https://aclanthoooogyy or or 2023. eacc-mmin....... 139
Varun Khurana, Yaman K Singla, Jayakumar Subramanian, Rajiv Ratn Shah, Changyou Chen, Zhiqiang Xu, and Balaji Krishnamurthy. Behavior optimized image generation. arXiv preprint arXiv:2311. 109995 2023b.
Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. Improving sentence compression by learning to predict gaze. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 1528-1522 1533. San Diego, Califormia, June 2016. URL https://aclannthology
Association for Computational Linguistics. doi: 10. 10.18653/ 1179.
org/N16-1179
Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification research. In European conference on machine learming, pp. 217-226. Springer, 2004.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. (06-715, 2017.
Yaman Kumar, Rajat Jha, Arunim Gupta, Milan Aggarwal, Aditya Garg, Tushar Malyan, Ayush Bhardwaj. Rajiv Ratn Shah, Balaji Krishnamurthy, and Changyou Chen. Persuasion strategies in advertisements. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 57-66, 2023.
Harold D Lasswell. The structure and function of communication in society. The communication of ideas, 37 (1):136-139, 1948.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a.
Kunchang Li, Yali Wang. Gao Peng. Guanglu Song. Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial-temporal representation learning. In International Conference on Learning Representations, 2021.
KunChang Li, Yinan He. Yi Wang. Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, arXiv preprint arXiv:2305.06355, 2023b.
Haotian Liu. Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023.
Masoud Mazloom, Robert Rietveld, Stevan Rudinac, Marcel Worring, and Willemijn Van Dolen. Multimodal popularity prediction brand-related social media posts. Proceedings of the 24th ACM international conference on Multimedia, pp. 197-201, 2016.
Denis McQuail and Sven Windahl. Communication models for the study of mass communications. Routledge, 2015.
Gerald R Miller. On defining communication: Another stab. Journal of Communication, 1966.
OpenAl. Gpt-4 technical report, 2023.
Charles Egerton Osgood, George J Suci, and Percy II Tannenbaum. The measurement of meaning. Number 47. University of Illinois press, 1957.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances Neural Information Processing Systems, 35:27730-27744, 2022. 2022.

---

Published as a conference paper at ICLR 2024
Joon Sung Park, Joseph C O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bern- stein. Generative agents: Interactive simulacra human behavior. arXiv preprint arXiv:2304.088 03444, 2023.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/116 2023.
Barbara Plank. Keystroke dynamics as signal for shallow syntactic parsing. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 609 619, 2016.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Hsuan Yang, Huisheng Wang, Serge Belongic, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6964 6974, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and 1lya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp- 28492-28518 18. PMLR, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21 21(1):5485-5551, 2020.
Elena Romero, Manuel Chica, Sergio Damas, and William Rand. Two decades of agent-basch modeling in marketing: a bibliometric analysis. Progress in Artificial Intelligence, pp. 1-17, 2023.
Claude Shannon and Warren Weaver. The mathematical theory of communication. The mathematical theory of communication. University Illinois Press, Champaign, II., US, 1949.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-2565, 2018.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Vidcobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7464-7473, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2.303 2...9999......, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Francisco Villarroel Ordenes, Dhruv Grewal, Stephan Ludwig, Ko De Ruyter, Dominik Mahr, and Martin Wetzels. Cutting through content clutter: How speech and image acts drive consumer sharing of social media brand messages. Journal of Consumer Research, 45(5):988-1012, 2019.
Kc Wang, Mohit Bansal, and Jan-Michael Frahm. Retweet Tweet popularity prediction via dynamic multimodal regression. 2018 IEEE winter conference on applications of computer vision (WACV), pp- 1842-1851. IEEE, 2018.
Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1884-1894, 2021.
Fanyi Xiao, Kaustav Kundu, Joseph Tighe, and Davide Modolo. Hicrarchical self-supervised representation learning for movie understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9727-9736, 2022.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning. arXiv preprint MrXiv:2301.13800 2023.
Hang Zhang. Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0265 02858. 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint xxXx:2304.1059) 2023.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 VERBALIZATION LISTINGS
Listing 2: Verbalization pattern of videos for the behavior understanding task:
Input: <<vidco>> <Nidoo>>
The video has the tollowing soenes:
Scene (ASR: Welcome yuick tutorial OCR: Adobe Premiiee Pro, Cuptions: desktop interface Replays: 60).
Scene 2: [ASR: on using Premiere Pm to edit, Captions: A computer interface, with an image of white horse. Objects Horse, Grass, Fence., Replays: 53},
was posted on Adobe's YouTube channel with the title "Using Premiere Pro like a Pro' on Aug 15 2022. Adobe's YouTube channel has 100k subseribers. This video viewed by 346 thousand people and liked (as pereentage of likes (views) by 2.3% people. Why is the scene 23 one of the most replayed scenes?
Ourput: The scene shows the transformation of the image after the changes.
Listing 3: Verbalization pattern of emails for the behavior domain adapation task. The email content and CTR is for demonstration purposes only.
Input: Email with Subject: Lock it down before you send out.
Header: Nobody messes with your PDFs.
Body text: Add passward protection secure eneryption. and restricted editing to your PDFs with Adohe Acmoat Pm DC. Share only what you want and nothing more. button that says 'Get started image of laptop, with window open on it.
Image wat: " Protect using password".
Foreground coloes: grey. blue. Background colors: lavender, white. Imuge Emmttions: security. serious. Imuge keywords: laptop-
protect, passward, lock. Aesthetic value: low. Chutter level: medium. The email is created by Creative Professional the product Adobe Acrebat Pro.
sent users in the United States, in commercial market. Specifiially. is sent to Power users with the inteat of Active Use.
The email was sent 109 times between 25 August, 2022 and 26 August, 2022, and had a click through rate of [MASK]%K
Output: 0.037%.
Listing 4: Verbalization pattern teach behavior in the reverse direction (predicting content given behavior):
Input:
<video> -. <<vido>>> Thee video following scenes: Scene [ASR: [MASK], Replays: 60%) Scone (ASR: with Premiere, Captions: Woman looking at screen, Replays: 34%).
Scene 5: [ASR: has never been. Captions: Colour Pallete, Roplays: 47%
Scene {ASK: been easier. Captions: Colour Paliete. Replays: 54%),
It was posted on Adobe's YouTube channel with the titl "Using Premiere Pro like a Pro' on Aug 13 2022. It is viewed 203k times
1.2%. Adobe's YouTube chameel has 100% subserihers. Predict masked ASR value the masked scenes.
Output: Soone I:(ASR: Welcome to a quick nutorial
.)
Listing 5: Verbalization pattern of videos for the content simulation task:
Input: <viden <Nideo's The video has following scenes: Scene (ASR: [MASK], Replays 60%s Scene (ASR: with Premiere, Captions: Woman looking at screen, Replays: 34%).
Scene (ASR: has been. Captions: Colour Pallete Replays: 47%).
Scene 6: (ASR: been easier. Captions: Colour Pallete, Replays: 54%),
It was posted on Adnhe's YouTube channel with the title "Using Premiere Pm like a Pro' on Aug 15 2022. It is viewed 203k times and liked 1.2%. Adobe's YouTube channel has 100k subscribers. Predict the masked ASR value for scene Choose from the given options.
Option-I: Welcome a quick Iutorial
Option-:: Samsung Galaxy A20 smartphone,
Option-25: regulations. haven't had.
Listing 6: Verbalization pattern of videos for the behavior simulation task:
Input: <idco'> < The vidco has the following sccues:
Scene 1: [ASR: Welcome to a quick tutorial OCR: Adobe Premiere Pro, Captions: A desktop interface Replays: [MASK]]-
Scene [ASR: using Premiere
to edit, Captions: A computer interface,
an image of white horse. Objects Horse,
Grass, Fence., Replays: [MASK] ),
was posted on Adobe's YouTube channel with the title "Using Premiere Pro like a Pro' on Aug 15 2022. Adohe's YouTuhe channel has 100k subseribers... Can you tell the replay values for scenes 2 How many times will this video he
viewed and liked as a percentage of likes /viows?
Output: Scene I: (Replay: 60% ). Scene 2: (Replay: 85%), Views: 2.1 Million, Likes-per-Vient 2,5%
I Listing 7: Verbalization pattern of Twitter posts for the behavior simulation task:
Input: Given a tweet of plizer posted by the accoun PfizerMed on 2023-01-12, Tweet: Annoncing a new ASGCT-Plier grant to support independent medical education initiatives on genetic medicines. details, click Request for Proposals. <
13

---

Published as a conference paper at ICLR 2024
hyperlink?>. Applly by Jamuary 30, 2022 # rarediseuse #ASGCT KCieneTherrapy <hypprlink?. Verbaliiaation media content: caption'"": " ' close-up of DNA double helix, showcasing its structure and blue color\"" "Kkywords' "t \ ("DNA, double helix, stracture, blue, close-up, molecular biology, geneties, biology, scientific illustration \"). Predict whether will recieve high or low likes P".
Output: This tweet has low likes.
Listing 8: Verbalization pattern of Twitter posts for the content simulation task:
Input: Generate tweet given the media verbulization and the likes got. Tweet for plizer to be posted by account PfizerMed on 2023-01-12. Verbalisation of media cuntent: ; \ ""caption]" \"A close-up of a DNA double belix, showcasing structure and blue color \o, "Keywawa'' ( (DDAA, double helix, structure, blue, close uup molecular biology. genctics, biology. scientific illustration \"). This tweet has low likes !"
Ourput: "Tweet: Announcing new ASGCT-Pfizet grant to support indopendent medical education initiatives on genetic medicines For details click Request for Proposals. <hypertink>. Apply by January 30, 2022 rarediscase #ASGCT #Gene Therapy <hyperlink>">
A.2 OTHER RELATED WORK
Communication is the situation in which a source transmits message to a receiver with conscious intent to affect the latter's behaviors (Osgood et al. 1957 Miller 1966). Thus, in the most general terms, communication implies a sender, a channel, mes- sage, receiver, relationship between sender and receiver, an effect, context in which communi- cation occurs and range of things to which 'messages' refer (McQuail & Windahl| 2015 Lasswell 1948]. As per this, all of the content produced by humanity is essentially communication from sender to a receiver over some channel and with some effect. Despite much research on commu- nication in social sciences since the 1900s, there has been little adoption of it in machine learning modeling. A prime artefact of this is that the biggest models in machine learning (LLMs) are trained only on content (messages) and ignore other factors in communication (the intended receiver, chan- nel, and behavior) even when they are available.
Models of Human Communication:
While there has been much research in ML to model human behavior, it has been disconnected from language and, sometimes, real-world data. For instance, Agent-based modeling (ABMs), a popular paradigm in Reinforcement Learning, has been employed to model behavior (Bankes| 2002; Romero et al.! 2023 Park al.) (023). Nevertheless, ABMs tend to view humans as rational economic agents who communicate primarily through their actions, ne- glecting the significance of content in communication. In ABMs. agents strive maximize their rewards, whereas communication does not always aim to optimize specific, well-defined reward signals. Moreover, the scarcity of large repositories containing extensive records of human actions poses a challenge when training ABMs to learn human behavior. Consequently, existing large mod- els trained on human behavior, such as the ABMs and decision transformer and its variants, often rely on simulated data, such as game environments, rather than real human behavior (Chen et al.) 202T]. This reliance on artificially generated data introduces biases inherent to the creators of the training data, making it difficult to capture authentic human behavior. However, recent advance- ments have demonstrated the potential of large models trained on real-world tokens encompussing various modalities, like images, videos, audio, and text, as the basis for diverse tasks (Ge et al., 2023 Li et al.) 2023a). Notably, L.LMs, as exemplars of foundation models, have exhibited imprrs- sive performance across range of tasks, including those they were not explicitly trained for, such as emotion recognition, named entity recognition, and complex tasks like table understanding (Ye) al.
Prior Efforts To Model Behavior:
2023 Bhhattcharyyaa et al., 2023).
Further, there has also been much work in modeling behavior using conventional modeling tech- niques, such as regression, bagging and boosting (Mazloom et al.) 2016 Villarroel Ordenes et al., 2019) neural networks (Ding et al. 2019 Wang et 2018 Khosla et al. (2044). and transtormers (Wu Krahenbuhl] [021] Xiao et al., (022). While these models can certainly model behavior, LLMSs show generalization powers which extend to capabilities much beyond just behavior simu- lation. For instance, once trained on behavior tokens, other than behavior simulation, LLMs can now generate behavior optimized content (Table 2). explain behavior (Table 3) and domain-adapt to other behaviors (Table none of which are shown by other models. The other concurrent works which model behavior using LLMs (Kang al. 2023) model just behavior (for example, by CTR prediction) by attaching classification or regression heads to LLMs and thereby lose out on the text-to-text paradigm where I.LMs show their best performance and generalization capabilities. In
14

---

Published as a conference paper at ICLR 2024
addition, similar to LLM paradigm, this method loses out other capabilities like generating behavior optimized content and explaining behavior.
A.3
REMAINING TABLES AND FIGURES
Model
#Params
Training
Training
RMSE
R²
Accuracy
LCBM
BFT
Replay values 3-masked
1.31
0.87
15.89
LCBM
13B
BFT
Replay values m-maskdd
1.48
0.82
19.93
LCBM
BFT
Roplay values 7-masked
1.71
0.78
15.20
LCBM
BFT
Replay values I1-masked
1.55
0.82
13.94
GPT-
100F
ICL
10-shot
GPT4
ICL
2-shot
3.58
.0.03
5.39
[CT.
1758
GPT-3.5
ICL
64.88
-375.83
1,27
Random
4.67
3.94
Table 4: Behavior Simulation. RMSE, R², and accuracy scores for like/view ratio prediction task. To calculate accuracy, the model is said to classify correctly the absolute error between the predicted and ground truth likes/views less than equal to 10%. BFT denotes behavior fine-tuning, and ICL stands for in-context learning. Replay values k-masked means model which trained by masking & consecutive values of the replay graph while doing BFT. We note that LCBBM while being at least smaller than other models, performs the hest. The best results over four runs are reported for all models. Best models are denoted in green and runner-ups in blue
Training
Model
#Params
Topic
Emotion
Persuasion
Action
Reason
All labels
Clubbed
2.63
3.34
Zero-shot
GPT-3.5
175B
51.6
11.68
79.69
35.02
66.27
59.59
Vicuna
13B
11.75
10.5
68.13
26.59
20.72
21.00
VideoChat
[1 al. 2023b
5.1
LCBM
13R
42.17
7.08
58.83
32.83
39.55
27.91
Table Content Understanding. Comparison several models, including behavior instruction tuned models before and after BFT. We compare the models across topic, emotion, and persuasion strategy detection tasks as the framework given by /Bhattachrryaa et al. (2023). We see that our model outperforms similarly sized models (Vicuna, VideoChat) in most tasks. Best models are denoted in green and runner-ups in blue
'The exact of GPT-4 is unknown.
*Brand Separated means that the train and test set don't have overlap in terms of brands. Time Separated that the test set starts after the last tweet in the train BFT denotes behavior fine-tuning, and ICL stands for in-context learning. The best results over four runs are reported for all models. Best models denoted green and runner-ups in hlue
15

---

Published as a conference paper at ICLR 2024
In-house Email Marketing
LCBM Type
Fine-tuned
Trained
Tested
RMSE
Ra
on
Unique Emails
Unique Segments
Email-Segment
YouTube?
Pairs
Domain- Adapted
Yes
100
Different Segment (emails could be same)
14.47
0.64
In-
No
600
560k
350k
25.28
0.55
Domain
Domain-
Yes
100
10
1k
Different Segments & Different Emails
27.28
0.54
Adapted
In-
No
600
560k
350k
29.28
0.5
Domain
Benchmark
Training
Model
Testing
R101 slowsaastt
Trained
Wi Krahenbahl
2021
Test set
0.386
Trained
VidcoBEKI
Test set
0.32
2019)
Trained
Oaa et m
414
Test set
0,353
Trained
X150 ct al
Tost set
0.444
Trained
Object Transtomers
Wi Krahenbuh] 2021
Test set
0.23
Zero-shot
LCBM (Ours)
Test set
0.14
Zero-shot
GPT.3.5
Tost set
0.03
Zero-shot
Vicuna
Complete dataset
0.44
Zero-shot
LCBM (Ours)
Complete dataset
0.30
Zero-shot
GPT-3.5
Complete dataset
0.02
Table 6: Behavior Domain Adaptation. We test the generalization capability of LCBM on two lasks: (1) Be- havior simulation In ho-so Email Marketing Data, (2) Behavior simulation the LVU benchmark. For (I), we train two versions LCBM with the In-house Email Marketing data: one was trained on YouTube videos and further BFT on a few email samples (domain-adapted) and the other was BFT on a larger set of emails, but not including YouTuhe data (in-domain We report the RMSE and R² scores for this task. For (2), we compare LCBM with other maat--oftthe-ar results and GPT-3. In (1), we note that the better than the in-domain I.CBM in both settings. We posit that YouTube data helps LCBBM understand how company's viewers like hear from it, giving LCBM an edge over a model trained small amount of same data (600 unique emails). In (2), LCBM performs better than the existing state-of-the-art Surprisingly, GPT-3.5 does better than LCBM this task. From both (1) and (2), gather that model trained on certain YouTube behaviors performs better other behaviors, thus showing promisc of domain- addattttoott in the be-
LCBM performs
havior modality. Best models are denoted in green and runner-ups in hlue
16

---

Published as a conference paper at ICLR 2024
Training
Time Separated 58.84
Brand Separated
Model
#Params
Training
type
GPT-3.5
I75B
ICL
Few-shot
64.19
LCBM
13B
BFT
Twitter
74.3
97.69
Twitter and
LCBM
13B
BFT
76.87
92.19
YouTube data
Table 7: Behavior Simulation and Behavior Domain Adaptation Two-way classification accuracies for
like prediction on Twitter. Given content, channel, and time, predict behavior (High, Low). We note that LCBM trained on Twitter and YouTube performs better than the one trained only on Twitter, showing signs of performance improvement by domain adaptation.
Model
Training
Test
BLEU-1
BLEU-2
BLEU-3
BLEU-4
ROUGE-
Brand Separated
53.95
42.36
31.84
24.28
15.24
GPT-3.5
ICL
Time Separated
$7.69
45.11
33.67
25.52
15.27
BFT on Twitter
Brand Separated
62.29
46.59
33.98
25.64
14.
LCBM
Time Separated
54.4
41.43
32.48
17.38
BFT on Twitter
Brand Separated
64.28
48.1
35.17
26.63
14.83
LCBM
Youtube
Time Separated
70.23
54.54
41.52
32.54
17.45
Table 8: Content Simulation and Behavior Domain Adaptation Given behavior, channel, time, tweet media caption as prompt, predict content (tweet text). We note that LCBM trained on Twitter and YouTube performs better than the one trained only on Twitter, showing signs of performance improvement by domain adaptation.
Video lanne from the 2örd-foor window to ascape the police
Question: Would the avarage sentiment of the comments of the audience be postive, neutral, negative? Explain why.
Posted by: South
Momning
Link youtu beF77DW2IT
LCBM The average sendiment the comments of the audience watching the vidoo would regative. This bocause the video shows a man our of high-die buiding Iklly feel sympaihy and sadress towards the man's dituation.
Comments from Video: loot shown LoMbicc55))
the polios. The ausience
fool bad the right mind mite looal then hang window.
The ausience lulld likely have a mix of positive and negative ment sowards the video. Viewers may feel sympathesi towards the person trying escape and praise thair bravery.
Vicuna:
he
cried,
altuation justfy
GPT-3.5: 5; The video depicts dangerous and polentially Iio-troalarig silutiio,, which may cause some viewors foel Uneasy or unconfortacle. However, others may find the events escitng or thriling to wetch.
Commentar falt sorry young but you wrong.
Video Title: The new Ponsche 718 Spydor, Perfectly irational.
Question Civen the vidao, would the average audiencs sendment be positive, neutra, or negalive? Explain why.
Posted by: Possche Link: youtu. S0555
LCBM:
Posltive average sentiment the audinnos The video featurso driving along scenic coastal med, which would evoke senso of owe and admiratin in the vewers, Additoorlly the Porsche 710
Comments from Video: (net oen LLMaLLB00)
video is an iconto sports car that widely recogrize lzzd respacted automative community, contribuiig
posilive sontimnt.
angine, aspiration, b6, ned, manual, lightwelght, open top
Commenter:
average
comments
beautriu scanic righ Addilinnlly, onioyment may contribute position
Commenter: Al the makings of apic sports car
Commenter: This perfect. I'
sentimennt may
GPT-3.5 The contiment the audience watching the video posttive because the video showcases new sports which evsiting and desirable car anthusieedd
Commenter: add and wishes love
Figure 5: A few examples showing LCBM's ability to understand and explain human behavior of audience sentiment. We also compare it against other models like Vicuna and GPT-3.5.
17

---

Published as a conference paper at ICLR 2024
Gatrcaa babi
Titte: (Rig Po
production
Video Tite: Andy Rulz Anthony Joshua |The Rermanch Confirmed
station that anywhere went.
Posted Armuilimecks
Posted by: 11A-NN stosing Link, youtu.bul-f PintggsAAA
Link wuuuube/500ne SFU.be//SOdduuuFFFUU
Question: Why a the sccem from 1:00 1:11 one the moet replayed acenes?
Why
most reployed scenes?
instrument
Human: Becouse it shows people's
partly.
favorile bower wirning the fight..
LCBM: The Nighightt the which pote customers looking versellle and bent music solution.
Keys
LCBM: The scone shows the moment when Antionyy Joalus knocked by Andy Ruiz. J..
Video Titler butt password manageer your Goople
Google Lina:
Software Engineer
Posted by: Google
Posted
Link: youtu MMMgPOnn0M
be NyG2oeAA
Quastion Why is aaaree 0.20 one the most replayed scenes??
Question:
replayed
Human: introcuces the bullt password managor afte diecureing Pee rebeted presoncrcde.
Human: Bиcause of feremation spacilic the Google uorriptoon
The namator mentions that users don't have werry about remembering passwents thei Google account has built in passward manages This scere is in amportant because highlghhts conventent and secure feature of Google's serdices.
LCBM:
LCBM: The scene showe Line talking about overheeing people
the nox office and coes not provide snuch information aboutt work at Google.
Figure 6: A few examples showing LCBM's ability to understand and explain human behavior of scene playability. We compare against numan-provided explanations of the same.
Date Range
April 1, 2022 to June 12, 2023 225
Number of Countries
Target Products
Top Products used by millions of users
Customers Segmented the basis
expertise, frequency of and others
Type
Table 9: Details of the In-house Marketing Email dataset used to evaluate behavior generalization capabilities of the LCBM
18

---

Published as a conference paper at ICLR 2024
Logo
Logo
Nobody messes with your PDFs.
Going, going, almost gone.
paosoaus profection
product
Share
Our Black Friday deal on the
product
pian ends
Save
Usin Pawawced
TTAT
Logo
OON
Figure 7: The In-house marketing emails used in the Email dataset look similar to the ones shown here.
19


Reviews:
[{"review_id": "RvAP0k3mMg", "reviewer": "Reviewer_73Nf", "summary": "This paper aims to explore the content-behavior understanding abilities of LLMs in modern communication systems and proposes large content behavior models(LCBMs) to this end. Motivated by the recent vision-language model, LCBMs formulate the YouTube data (such as video frames, channel name, shares and likes) as the multimodal input sequence, and finetune a LLM in an autoregressive manner. Also, the authors release their generated data and a benchmark to spur the research purpose. Extensive results on five tasks show better performance of the proposed model than other baselines, including GPT-3.5 and GPT-4.", "questions": "1) Please provide more detailed information on how the visual encoder handles video input.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The task that employs the LLM to perform the content-behavior understanding sounds interesting. This provides new application directions to study the decision process of the LLMs. \n\n2) The released content-behavior datasets and benchmarks contribute to a better community, which facilitates the follow-up research.\n\n3)  Extensive experiments are conducted to test the performance of the proposed model, which shows the generalization capabilities on content-behavior understanding.", "weaknesses": "1) One of the main concerns is the technique novelty. From my understanding, the proposed model is mostly based on InstructBLIP[1] and only the GMHRA is newly introduced to aggregate the time dimension in video frames. \n\n2) More experiment details need to be described. Is the baseline models are finetuned using the behavior instruction datasets ? If not, then it may be unfair to compare the proposed model with these baseline models.\n\n\n\n[1] Dai et.al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noJRi76AHQ", "reviewer": "Reviewer_jTBK", "summary": "The paper presents a novel and interesting question: predicting and optimizing communication to get the desired receiver behavior for intent alignment. The paper uses behavior tokens in the training corpora to guide models to finish behavior simulation, content simulation, and behavior domain adaptation tasks. Based on existing multi-modal LLMs, the proposed methods achieve promising results on several benchmarks such as LVU and Email Marketing Data.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Using behavior tokens to guide the model to finish various tasks is novel and important to me. In fact, \"human:, assistant\" question-answer pairs used in modern SFT models is also a kind of behavior token. The paper extends the scope to a broader concept. The experimental results are convincing. I believe that this paper has the potential to provide valuable insights to the field of intention alignment in large-scale language models, extending beyond the scope of video click-through rate as described in the paper.", "weaknesses": "Similar to Instruction Following Finetuning (SFT), it demonstrates that a small amount of data can activate the model's capabilities, resulting in more human-like responses. It would be interesting to study the impact of varying data quantities on the results in the context of this work, particularly in terms of CBC (Contextual Bandit Control). Additionally, exploring the influence of data quantity on the results when behavior tokens are present would also be insightful. Investigating these aspects would provide a further understanding of the relationship between data volume and the performance outcomes in this study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HvMgyFZvJR", "reviewer": "Reviewer_A7CN", "summary": "This paper studies the problem of behavior simulation, content simulation, and behavior domain adaptation in LLMs. The authors train LLMs on behavior tokens to show that other than content understanding tasks, the trained models are now able to have good performance across all the behavior-related tasks as well.", "questions": "See weaknesses above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Formulating the problem as behavior/content prediction and understanding is interesting;\n\n2. The authors systematically studied the behavior/content related problems in online website domains, and train an LLM to solve these problems;\n\n3. The experiments demonstrates promising results for simulating and understanding tasks.", "weaknesses": "1. The authors claimed that the paper studies the problem of predicting and optimizing communication for desired receiver behaviors and intents, i.e., the effectiveness level of communication, but what the authors actually did in this paper is simply exploring the Youtube and email datasets. As far as I understand, the key to improve the performance of behavior/content prediction and understanding is to improve the general reasoning ability of LLMs, rather than finetuning LLMs on specific datasets. From this perspective, the contribution of this paper is very limited.\n\n2. In terms of presentation, I think the authors make things too complicated by discussing the effectiveness and behavior issues, which make the readers hard to quickly get the main idea and contribution of this paper.\n\n3. The behavior simulation task sounds like a data mining problem, e.g. predicting whether a user will watch a video. LLMs are not quite suitable for such tasks because they are too slow and they cannot make use of massive user-item interaction data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7D1hIpglr", "reviewer": "Reviewer_aTeR", "summary": "In this paper, the authors attempts to tackle the challenge of communication effectiveness by LLMs. The authors posed the effectiveness problem as a text-to-text tokens by including \"receiver behavior\" as a new modality and introduce \"behavior tokens\" to LLMs. The authors made several contributions towards tackling this problem (1) the authors created a large corpus of data (CBC) containing Youtube video segments as well as viewer retention/replay values and number of viewers and likes, as well as communicator information (channel name and subscriber counts), and (2) a comprehensive test benchmark for content behavior, including behavior prediction, content prediction, behavior understanding, content understanding, and behavior domain adaptation to non-CBC domains (such as email marketing), and (3) The authors introduced a new method called Behavior Fine-tuning, and used it to fine-tune a 13B parameter model (LCBM), and the resulting model outperformed much larger models such as GPT-3.5 and GPT-4 on the content behavior test benchmark.", "questions": "(1) Is BFT training using CBC data? Is the test benchmark also using CBC data? If both are using CBC data, is there a train/test split within CBC so a different split is used for training and testing?\n\n(2) In section 3, the paper mentioned that the behavior tokens are not used for GPT-3.5 and GPT-4. What about the visual tokens about the videos? Did you find a way to input them to GPT, or did you passed in the visual content of the video through some different means?", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper made significant contribution to an important problem: teaching LLMs to understand and reason about communication effectiveness. The authors supported the significance of the problem from Shanon and Weaver's Information Theory, and they carefully posed this problem into a text-to-text problem so researchers can now explore ways to tackle this problem via LLMs. \n\n2. The paper introduced a large dataset of behavior-content data, as well as a test benchmark that comprehensively evaluates an LLM's ability to reason about content behavior and communication effects (including some human-annotated ground truths). These contributions will be very valuable for future researchers working on communication effectiveness or other related problems.\n\n3. The paper introduced a novel method to fine-tune LLMs for behavior understanding, and their resulting 13B parameter model outperformed GPT-4/GPT-3.5, often by large margins, on most tasks in content behavior test benchmark, which is really impressive.\n\n4. The figures in the paper are really informative and can really help readers better understand the problem, LCBM, and tasks in the test set.", "weaknesses": "Overall this is a really nice paper. A very minor weakness of this paper may be that the font sizes of the figures and tables are really small and sometimes inconsistent between different tables and figures. I understand that this is likely due to space limit in the main paper, so perhaps moving some parts of the detailed discussions in the introduction (such as the backgrounds in \"How do we solve the effectiveness problem while retaining the other two levels?\") to the appendix would alleviate the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)