Kati8/ICLR_2023_2025 huggingface dataset:
authors: Ashmit Khandelwal
authors: Aditya Agrawal
authors: Aanisha Bhattacharyya
authors: Yaman Kumar
authors: Somesh Singh
authors: Uttaran Bhattacharya
authors: Ishita Dasgupta
authors: Stefano Petrangeli
authors: Rajiv Ratn Shah
authors: Changyou Chen
authors: Balaji Krishnamurthy
reviews: 
Comments: In this paper, the authors attempts to tackle the challenge of communication effectiveness by LLMs. The authors posed the effectiveness problem as a text-to-text tokens by including "receiver behavior" as a new modality and introduce "behavior tokens" to LLMs. The authors made several contributions towards tackling this problem (1) the authors created a large corpus of data (CBC) containing Youtube video segments as well as viewer retention/replay values and number of viewers and likes, as well as communicator information (channel name and subscriber counts), and (2) a comprehensive test benchmark for content behavior, including behavior prediction, content prediction, behavior understanding, content understanding, and behavior domain adaptation to non-CBC domains (such as email marketing), and (3) The authors introduced a new method called Behavior Fine-tuning, and used it to fine-tune a 13B parameter model (LCBM), and the resulting model outperformed much larger models such as GPT-3.5 and GPT-4 on the content behavior test benchmark.
Strengths: 1. This paper made significant contribution to an important problem: teaching LLMs to understand and reason about communication effectiveness. The authors supported the significance of the problem from Shanon and Weaver's Information Theory, and they carefully posed this problem into a text-to-text problem so researchers can now explore ways to tackle this problem via LLMs. 

2. The paper introduced a large dataset of behavior-content data, as well as a test benchmark that comprehensively evaluates an LLM's ability to reason about content behavior and communication effects (including some human-annotated ground truths). These contributions will be very valuable for future researchers working on communication effectiveness or other related problems.

3. The paper introduced a novel method to fine-tune LLMs for behavior understanding, and their resulting 13B parameter model outperformed GPT-4/GPT-3.5, often by large margins, on most tasks in content behavior test benchmark, which is really impressive.

4. The figures in the paper are really informative and can really help readers better understand the problem, LCBM, and tasks in the test set.
Weaknesses: Overall this is a really nice paper. A very minor weakness of this paper may be that the font sizes of the figures and tables are really small and sometimes inconsistent between different tables and figures. I understand that this is likely due to space limit in the main paper, so perhaps moving some parts of the detailed discussions in the introduction (such as the backgrounds in "How do we solve the effectiveness problem while retaining the other two levels?") to the appendix would alleviate the problem.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper studies the problem of behavior simulation, content simulation, and behavior domain adaptation in LLMs. The authors train LLMs on behavior tokens to show that other than content understanding tasks, the trained models are now able to have good performance across all the behavior-related tasks as well.
Strengths: 1. Formulating the problem as behavior/content prediction and understanding is interesting;

2. The authors systematically studied the behavior/content related problems in online website domains, and train an LLM to solve these problems;

3. The experiments demonstrates promising results for simulating and understanding tasks.
Weaknesses: 1. The authors claimed that the paper studies the problem of predicting and optimizing communication for desired receiver behaviors and intents, i.e., the effectiveness level of communication, but what the authors actually did in this paper is simply exploring the Youtube and email datasets. As far as I understand, the key to improve the performance of behavior/content prediction and understanding is to improve the general reasoning ability of LLMs, rather than finetuning LLMs on specific datasets. From this perspective, the contribution of this paper is very limited.

2. In terms of presentation, I think the authors make things too complicated by discussing the effectiveness and behavior issues, which make the readers hard to quickly get the main idea and contribution of this paper.

3. The behavior simulation task sounds like a data mining problem, e.g. predicting whether a user will watch a video. LLMs are not quite suitable for such tasks because they are too slow and they cannot make use of massive user-item interaction data.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: The paper presents a novel and interesting question: predicting and optimizing communication to get the desired receiver behavior for intent alignment. The paper uses behavior tokens in the training corpora to guide models to finish behavior simulation, content simulation, and behavior domain adaptation tasks. Based on existing multi-modal LLMs, the proposed methods achieve promising results on several benchmarks such as LVU and Email Marketing Data.
Strengths: Using behavior tokens to guide the model to finish various tasks is novel and important to me. In fact, "human:, assistant" question-answer pairs used in modern SFT models is also a kind of behavior token. The paper extends the scope to a broader concept. The experimental results are convincing. I believe that this paper has the potential to provide valuable insights to the field of intention alignment in large-scale language models, extending beyond the scope of video click-through rate as described in the paper.
Weaknesses: Similar to Instruction Following Finetuning (SFT), it demonstrates that a small amount of data can activate the model's capabilities, resulting in more human-like responses. It would be interesting to study the impact of varying data quantities on the results in the context of this work, particularly in terms of CBC (Contextual Bandit Control). Additionally, exploring the influence of data quantity on the results when behavior tokens are present would also be insightful. Investigating these aspects would provide a further understanding of the relationship between data volume and the performance outcomes in this study.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: This paper aims to explore the content-behavior understanding abilities of LLMs in modern communication systems and proposes large content behavior models(LCBMs) to this end. Motivated by the recent vision-language model, LCBMs formulate the YouTube data (such as video frames, channel name, shares and likes) as the multimodal input sequence, and finetune a LLM in an autoregressive manner. Also, the authors release their generated data and a benchmark to spur the research purpose. Extensive results on five tasks show better performance of the proposed model than other baselines, including GPT-3.5 and GPT-4.
Strengths: 1) The task that employs the LLM to perform the content-behavior understanding sounds interesting. This provides new application directions to study the decision process of the LLMs. 

2) The released content-behavior datasets and benchmarks contribute to a better community, which facilitates the follow-up research.

3)  Extensive experiments are conducted to test the performance of the proposed model, which shows the generalization capabilities on content-behavior understanding.
Weaknesses: 1) One of the main concerns is the technique novelty. From my understanding, the proposed model is mostly based on InstructBLIP[1] and only the GMHRA is newly introduced to aggregate the time dimension in video frames. 

2) More experiment details need to be described. Is the baseline models are finetuned using the behavior instruction datasets ? If not, then it may be unfair to compare the proposed model with these baseline models.



[1] Dai et.al. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good