GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:53:47

Abstract: Recent advances in Multimodal Large Language Models (MLLMs) have constituted a significant leap forward in the field, particularly in the processing of videos, which encompasses inherent challenges such as spatiotemporal relationships. However, existing MLLMs are predominantly focused on the comprehension of video inputs, with limited capabilities in generating video content. In this paper, we present GPT4Video, a unified framework that seamlessly and lightly integrates with LLMs, visual feature extractors, and stable diffusion generative models for cohesive video understanding and generation. Moreover, we propose a text-only finetuning approach to equip models for instruction-following and safeguarding in multimodal conversations without requiring costly annotated video-based instructions. Additionally, we construct multi-turn and caption-interleaved datasets for finetuning and benchmarking MLLMs, which serve as solid resources for advancing this field. Through quantitative and qualitative assessments, GPT4Video demonstrates the following advantages: 1) The framework incorporates video generation ability without adding extra training parameters, ensuring seamless compatibility with various video generators. 2) The model achieves superior performances across a variety of benchmarks. For instance, it outperforms Valley by 11.8% on video question answering, and surpasses NExt-GPT by 2.3% on text-to-video generation. 3) As safety pioneers in open-source MLLMs, we developed finetuning and evaluation datasets, securing an F1 score exceeding 80% in blocking harmful content during understanding and generating videos. In general, GPT4Video shows potential to function as a real-life assistant, marked by its effectiveness, adaptability, and safety. We will open-source our code, data, and models.

Main paper: 
 introduction: 
 Large language models (LLMs) such as LLaMA [43] , ChatGLM [13] , Vicuna [11] (in Figure 1 (a)), have undergone significant advancements, paving the way for general artificial intelligence (AGI) and showcasing remarkable zero-shot capabilities across diverse linguistic tasks. Based on this foundation, multimodal large language models (MLLMs) like MiniGPT-4 [63] , Video-LLaMA [55] , Macaw-LLM [30] (in Figure 1 (b) ) have been successively introduced. These models are capable not only of processing textual information but also of handling data across various modalities such as images, audio, and videos, demonstrating exceptional performance in multimodal tasks [54] . Nevertheless, current MLLMs primarily focus on processing multimodal inputs, yet they fall short in generating multimodal information on the output side. For a refined AGI system [17] , it is crucial not only to understand multimodal inputs but also to efficiently generate them, mirroring human interaction in real-world settings.Nonetheless, endowing LLMs with multimodal generation capabilities presents a significant challenge: what content aligns with user requirements, and when to generate it appropriately. Several recent studies have made preliminary forays in this direction [23, 48, 62] . GILL [23] initially investigated leveraging LLMs for image generation, introducing "generative vokens" (i.e. the special token "<IMG>") into the text vocabulary of LLMs. NExt-GPT [48] further extends the concept of "generative vokens" to a variety of modalities (e.g. image, audio, and video). Though the approaches have demonstrated certain efficacy, they have several inherent limitations:(1) The representational capacity of "generative vokens" is limited. Due to the constraints of their training mechanism, they fail to fully harness the outstanding potential of LLMs in text generation and emergence ability.(2) Adding new vocabulary to the LLM's lexicon may disrupt the LLM's original capabilities. (3) These methods lack the flexibility to adapt different generation models. For instance, upgrading the text-to-image/video model necessitates retraining the LLM.Note that, quantitative analyses on the representative method [48] in Section 5 provide support for the above claims.To address the limitations, we present GPT4Video, a unified framework for augmenting LLMs with both video understanding and generation capabilities (in Figure 1 (Ours)). Inspired by previous success [53] , we leverage the robust capabilities of pretrained LLM and visual feature extractor for video understanding. Considering the the inherent complexity of video over images, we introduce a video abstractor with a dual attention mechanism [14, 61] to facilitate precise alignment between video and text. For video generation, instead of incorporating "generative vokens", we train LLMs to produce the video descriptions with tags at the appropriate moment. These tags and descriptions respectively act as trigger ("when") and input prompts ("what") for a pre-trained text-to-video model.Previous studies commonly created and then finetuned MLLMs using multimodal instruction datasets for understanding abilities (e.g. text-video interleaved data) [8, 10, 30] . In contrast, we investigate a text-only finetuning method for video generation, replacing video with its caption in instruction datasets during the second-stage training. 1 Accordingly, we construct datasets and apply the proposed finetuning for two scenarios: equipping models for instruction-following (45K instances) and safeguarding (4K instances). Our GPT4Video has the following appealing advantages:• Seamless Integration: The framwork requires no additional modules or training parameters, thus can be seamlessly integrated into a variety of text-to-video models (e.g. Zero-Scope [29] and VideoCrafter [9] ). • Instruction-Followed Video Generation: Textual data has more powerful abstraction and expression capabilities in summarizing content. The fine-tuning with our text-based instructions is expected to efficiently harness LLMs' generalization abilities in generating content-rich video prompts that align with instructions. • Safeguarding: Through a simple safety alignment, our model substantially rejects both the processing of harmful video inputs and the generation of harmful video outputs. This makes it one of the first models to incorporate safeguarding features among open-source MLLMs. 2We conduct experiments on a variety of multimodal benchmarks, including open-ended question-answer [7, 50] , video captioning [50] , text-to-video generation [50] and safeguarding (ours). Results show that our model significantly and consistently outperforms the advanced models, NExt-GPT [48] , demonstrating the effectiveness and universality of our framework and approach. Our main contributions are:• We propose a unified framework that enhances LLMs with video understanding and generation capabilities through seamless and lightweight integration with pre-trained Abstractor and T2V models, demonstrating remarkable extensibility. • We explore a simple and effective method, text-only finetuning, for developing instruction-followed models, showing substantial improvements in performance. As safety pioneers, we further employ this approach for safety alignment, offering an alternative to einforcement learning from human feedback (RLHF). • We build and release valuable datasets to facilitate future work on multimodal LLMs, including (1) a instruction dataset for video generation that covers a wide range of conversational scenarios; (2) a safety benchmark for evaluating and enhancing safeguarding capabilities in video-interleaved conversations. 
related work: 
 Multimodal Language Models Numerous studies have previously developed multimodal language models that can handle visual inputs and text outputs, or vice versa, such as [6, 52] . With these advancements of LLMs, some researches have focused on learning a joint embedding space for multiple modalities, as demonstrated in [16, 36] . Others have combined pre-trained single-modality models to showcase impressive zero-shot capabilities [1, 24] . More recently, there has been a growing interest in enabling multi-modal LLMs to follow instructions, as shown in [12, 53, 63] . To facilitate research in this area, [51] introduced MultiInstruct, the first multi-modal instruction tuning benchmark dataset covering a wide range of tasks and categories. Additionally, [27] explored multi-modal instructiontuning using machine-generated data, while [30] fine-tuned all model parameters to allow the textual LLM to process four modalities.Large Language Models Large language models (LLMs) commonly refer to as Transformer-based language models with billions of parameters [44] and have revolutionized the research paradigm in natural language processing community [13, 43] . Furthermore, recent works have demonstrated that supervised fine-tuning, also known as instruction-tuning, can effectively improve the zero-shot performance of these LLMs [11, 42] . Zhao et al. [60] present a comprehensive survey on the research of LLMs.Text-to-Image/Video Generation Text-to-image/video generation refers to the task of producing realistic images or videos based on natural language descriptions. One of the earliest approaches to this task was the use of conditional GANs [37] . Since then, various techniques have been developed to improve the quality of the generated images [33] . Compared to text-to-image generation, textto-video generation is relative new and still remains challenging. Previous approaches have utilized techniques such as VAEs with recurrent attention [32] and expanding GANs from image to video generation [26] . Diffusion models have also been used to generate videos in recent works [2, 18, 40, 47] .Responsible AI As the AI systems become increasingly powerful, developing responsible AI have drawn significant scientific attention recently [22] . Various works have pointed out the safety risks of LLMs, such as toxicity [39] , and hallucination [57] . The safety of LLMs is commonly measured by specialized benchmarks, such as These prompts are then used to create videos from the Text-to-Video Model Gallery. Icons of a snowflake and a flame visually distinguish between the non-trainable and trainable parameters within the system. The two bullet points highlight GPT4Video's dual capabilities in understanding and generating video content.RealToxicityPrompts on toxicity [15] . More recently, [58] present SafetyBench, a large-scale diverse set of multiple choice questions across several aspects of safety concerns. 
our framework: 
 We introduce the GPT4Video, a unified framework designed to endow LLMs with advanced video understanding and generation proficiencies. As shown in Figure 2 , the architecture is composed of three integral components: 1) video understanding module that employs a video feature extractor and a video abstractor to encode and align video information with the LLM's word embedding space; 2) LLM, utilizing the structure of LLaMA and employing parameterefficient fine-tuning (PEFT) methods, specifically LoRA [19] , while keeping the original pre-trained parameters intact; and 3) video generation part that conditions the LLM to generate prompts for a model from Text-to-Video Model Gallery through a meticulously constructed instruction dataset.Visual Encoder Given a video denoted as v, we uniformly sample 𝑇 frames, which are represented asv = [v 1 , v 2 , ..., v 𝑇 ].For each individual frame v 𝑖 (where 𝑖 ranges from 1 to 𝑇 ), we utilize a pre-trained CLIP visual encoder to extract its visual features, denoted as f 𝑖 𝑣 = ViT(v 𝑖 ). Here, ViT represents the Vision Transformer used in CLIP for feature extraction. After extracting the visual features f 𝑖 𝑣 from each frame, we compile the video featuresF 𝑣 = [f 1 𝑣 , f 2 𝑣 , ..., f 𝑇 𝑣 ].More specifically, GPT4Video adopts the CLIP ViT-L/14 model [36] for the visual encoding task. This model comprises 24 layers, with each layer having a hidden dimension of 1024 and processing patches of size 14. Within this module, images are uniformly resized to a standard dimension of 224 x 224 pixels and then divided into patches using a stride of 14 pixels. These patches are treated as input tokens for the transformer block, enabling its self-attention mechanisms to effectively generate detailed image embeddings. Consequently, the output of ViT contains 256 spatial patch features and 1 global feature (identified as the "[CLS] token"), resulting in the image features f 𝑖 𝑣 ∈ R 257×1024 and video features F 𝑣 ∈ R (𝑇 * 257) ×1024 . Video Abstractor The input sequence for video features, denoted as F 𝑣 ∈ R (𝑇 * 257) ×1024 , becomes considerably large, particularly with an increased number of sampled frames, 𝑇 . This results in a substantial computational challenge when processing through a LLM. To mitigate this, we have implemented a video abstractor. This tool efficiently condenses the visual information into a few learnable tokens, thereby generating high-level visual representations and significantly reducing the computational demands. Many image-based MLLMs [24, 53] achieve this by adopting a Qformer-like structure, which incorporates learnable query tokens to distill image features. Considering the complexity of video over image, we further introduce a learnable enhance tokens to augment video feature extraction by forming a dual attention mechanism. We have validated the effectiveness of this structure in our experiments detailed in Table 5 . We mathematically define query tokens as Q 𝑠 ∈ R 𝑁 𝑠 ×𝐷 and enhance tokens as Q 𝑡 ∈ R 𝑁 𝑡 ×𝐷 , where 𝑁 𝑠 and 𝑁 𝑡 denote the counts of query and enhance tokens, respectively, and 𝐷 signifies the dimension of the token embeddings.The Q 𝑠 and Q 𝑡 respectively compute cross attention with the video features 𝐹 𝑣 . Our cross attention module [44] consists of six encoder layers, wherein the pivotal component of each layer is the multi-head self-attention mechanism (MSA) [44] .We treat Q 𝑠 and Q 𝑡 as queries, respectively, and the result of concatenating them with the video feature F 𝑣 serves as the key and value to compute cross attention. This process yields video features F 𝑠 ∈ R 𝑁 𝑠 ×𝐷 and F 𝑡 ∈ R 𝑁 𝑡 ×𝐷 , respectively. We set 𝑁 𝑠 and 𝑁 𝑡 to be the same and then sum the query and enhance features to obtain the final video feature F𝑣 ∈ R 𝑁 𝑠 ×𝐷 . The mathematical expression is as follows:EQUATIONEQUATIONEQUATIONwhere ";" indicates concatenation, LN(•) denotes layer normalization [3] . Finally, we employ a linear mapping layer to project the lowdimensional video feature F𝑣 into the high-dimensional word embedding space of the LLM, mathematically denoted as F 𝑣𝑖𝑑𝑒𝑜 = W F𝑣 + b, where W and b are the learnable weight matrix and bias vector, respectively, of the linear layer.Existing model [48] of integrating "generative vokens" only accommodates one specific text-to-video model per training session. This limitation arises because different text-to-video models may utilize various text encoders. In contrast, GPT4Video replaces "generative vokens" by generating T2V models' textual prompts. As demonstrated in the Text-to-Video Gallery, our approach is compatible with the full spectrum of text-to-video models, including Make-A-Video [40] , Tune-A-Video [47] , Text2Video-Zero [20] , VideoCrafter [9] , ZeroScope [29] . More importantly, should more advanced models be integrated into the Text-to-Video gallery in the future, GPT4Video will seamlessly adapt without necessitating any modifications. In this work, we employ ZeroScope [29] 3 as our default video generation model. We also report VideoCrafter1 [9] in Table 6 .LLMs have demonstrated remarkable capability in understanding and following human instructions. Thus, we leverage pretrained LLMs as our cognitive module. It is worth noting that the cognitive module also serves as the textual modality encoder in our approach. We adopt Llama-7B model with LoRA [19] as the LLM component of GPT4Video.3 https://huggingface.co/cerspense/zeroscope_v2_576w 
our training method and dataset: 
 GPT4Video employs a two-stage training strategy. In the first phase, we focused on enabling GPT4Video to comprehend video content. Inspired by Video-ChatGPT's success with LLaVA's pretrained parameters, we used mPLUG-Qwl's parameters for GPT4Video's initial alignment. This laid a foundation in image comprehension. To tailor for video understanding, our video abstractor was further trained on VideoChat 11K, chosen for its 7K high-quality video-text alignments. Results show that built on the model's initial image understanding, a modest amount of high-quality data is sufficient to endow GPT4Video with robust video comprehension capabilities. In the second phase, we employed text-only finetuning to equip the model with capabilities for video generation and safety guarding.The training pipeline for GPT4Video and examples of the training data for stage two are illustrated in Figure 3 .GPT4Video employs a two-stage training strategy. In the first phase, we freeze the parameters of LLM and focus on training the video extractor to effectively align video features with the LLM's feature space, to enhance the model's comprehension of video content. The second stage involves pure text-based instruction tuning, where we employ the LoRA method for efficient fine-tuning of the LLM, focusing solely on training the newly added parameters in the LLM. The input prompts for LLM are as follows:###Human:<video>video_embed</video> ###Human:video_instruction ###AI:where "video_embed" represents the video features F 𝑣𝑖𝑑𝑒𝑜 , and it was only employed in the first stage. "video_instruction" refers to the instruction data constructed for videos.During the first training phase, we utilized the VideoChat-11k dataset [25] , which includes 7,000 detailed descriptions and 4,000 multi-turn conversations. In the second phase, we employed our self-constructed GPT4Video-50k dataset along with a safety-aligned dataset, which will be elaborated upon in Sec. 3.2 and Sec. 4.3.We perform instruction-tuning of the LLM only on the text tokens, using its original auto-regressive training objective. Specifically, for a video instruction X 𝑡 of length 𝐿, conditioned on visual information F 𝑣𝑖𝑑𝑒𝑜 , our loss function, captured as the negative log likelihood, is formulated:EQUATION)where 𝜃 is the trainable parameters, X 𝑡,<𝑖 is the text tokens before the current prediction token 𝑥 𝑖 .Our method initially trains the LLM to learn how to generate textual prompts based on contextual information, which in turn drives a pre-trained text-to-video model to produce videos. To effectively train the LLM to generate these prompts, we meticulously developed an instruction-following dataset named GPT4Video-50k. The fundamental idea behind constructing this dataset is to utilize video descriptions in place of actual video content. Subsequently, we leverage the most advanced language model currently available, GPT-4, to create extensive dialogue data centered around these video descriptions.More specifically, first, we randomly extracted 10,000 data examples from the Webvid10M [4] dataset and obtained their video descriptions. We used the string "<video> Video Caption </video>" as a placeholder for the actual video in the prompt to GPT-4, where "Video Caption" served as the placeholder for the video descriptions, and the "<video>" and "</video>" tags marked the boundaries of the video description. In the prompt, we require GPT-4 to construct three dialogue between two individuals (not a person and an assistant like most other LLMs) centered around the provided Video Caption (representing a real video). This approach makes our model's responses more reflective of real human emotions, rather than just those of an assistant. The complete example of dialogues and the detailed prompt, please refer to our supplementary materials.Furthering our efforts, to endow the model with the capabilities for multi-turn dialogues and cross-video conversations, we developed a multi-round, interleaved instruction dataset centered around multiple video contents. Unlike the single-video centric dialogue data mentioned earlier, in this setup, we tasked GPT-4 with constructing rich conversational scenarios around the contents of multiple videos. Similarly, we utilized the format "<videoX> Video Caption </videoX>" to denote the actual videos, where "X" represents the video index, "Video Caption" is the description of the content of video X, and "<videoX>" and "</videoX>" serve as the delimiters for the description's beginning and end. Specifically, we randomly selected 5000 samples from the WebVid10M [4] dataset and obtained their video descriptions. Diverging from the singlevideo centric data construction approach, our multi-video centric method requires at least two or more video descriptions as input prompts for GPT-4. To ensure that the dialogue data constructed by GPT-4 closely mirrors real scenarios and maintains logical coherence, we posit that these video descriptions should be semantically related. To acquire pairs of semantically related video descriptions, we employed a text retrieval-based approach. The process involved three steps: firstly, using sentence transformers, we extracted feature embeddings for the video descriptions of the selected 5000 samples and the remaining descriptions in the WebVid10M dataset; secondly, we identified the top 10 similar video descriptions for these 5000 samples by calculating the cosine similarity between their feature embeddings; and finally, we randomly selected one or two from these top 10 descriptions to form pairs of video descriptions. For examples of the multi-video centric dialogues and the detailed prompts used for GPT-4, please refer to the supplementary materials accompanying this paper.For MLLMs, ensuring the generation of both useful and safe content, especially visuals, is imperative for their broad adoption. Take the handling of sexually explicit content as an example: Current text-to-image models employ an ancillary NSFW (Not Safe For Work) detection system. This system, upon recognizing potentially inappropriate content, replaces the intended output with a simple black image. While this approach effectively blocks unsafe content, it can lead to unnecessary consumption of time and computational resources. Moreover, when used in LLM-based models, this approach may lead to a disconnection between generated text and visual content. To address this issue, we have developed a safetyaligned instruction dataset that trains the model to politely refuse to respond to inappropriate user requests, thereby eliminating the need to initiate the generation model.Specifically, we utilized the real-toxicity-prompts dataset [15] , which comprises 100k texts along with their corresponding Toxicity scores. This dataset includes various categories for detection such as sexually_explicit, identity_attack, flirtation, threat, insult, and se-vere_toxicity. Focusing on the sexually_explicit and severe_toxicity categories, we extracted 1,500 texts from each, selecting those with toxicity scores exceeding 0.9. We then tasked GPT-4 to construct In addition to ensuring the safety of generated content, we have also implemented safety measures at the input stage. To this end, we introduced an additional detection model specifically designed to assess the safety of user input. In cases where inappropriate content is input by users, we employ a method similar to the one described above, training the model to tactfully decline responding to such content (see data example in the righy panel of Figure 3 ). In the experimental section 5.3, we conducted an in-depth and detailed evaluation of our model's safety features. For examples of more Safeguarding dataset and the detailed prompts used for GPT-4, please refer to the supplementary materials accompanying this paper.To investigate the safety of MLLMs in video understanding and generation, we construct a multimodel benchmark for not safe for work (NSFW) content. The benchmark examines the percentage of harmful content that is rejected by the MLLMs when presented with various video inputs and queried with harmful and common questions. Referring to a text-only NSFW benchmark [21] , we collect a diverse collection of video clips and queries with varying levels of harmfulness and safety. In specific, our benchmark includes two classification tasks focused on video understanding and generation, respectively. Each task includes 60 queries with an equal distribution of harmful and safe content. In the video generation task, in addition to the three-level harmful contents [21] , the queries include natural scenes, human and animal activities, and real and comic episodes to ensure a diverse range of topics. Additionally, we design various video and question pairs to prompt MLLMs to provide inappropriate responses in the video understanding task. In this way, our benchmark can provide a comprehensive evaluation of the safety capabilities of MLLMs in video understanding and generation. 
experimental settings: 
 Datasets and Evaluation Metrics. To ensure a fair comparison, we followed work [28, 31] and assessed the model's understanding abilities through the Zero-shot Video Question Answering task. We conducted a comprehensive quantitative assessment using two widely-accepted open-ended question-answer datasets: MSRVTT-QA [50] , MSVD-QA [7] . These evaluations were conducted in a zero-shot setting, utilizing GPT-assisted assessments to gauge the model's performance. This process aims to quantify the accuracy of the model's predictions, assigning scores from 1 to 5. We further conducted video captioning task on MSRVTT [50] dataset, using BLEU-4 [34] and METEOR [5] as evaluation metrics.On the other hand, in terms of evaluating video generation capabilities, we followed NExt-GPT [48] and conducted Text-to-Video Generation tasks on the MSRVTT [50] dataset, using Fréchet inception distance (FID) [35] and CLIPSIM [46] (average CLIP similarity between video frames and text) as evaluation metrics. Implementation Details. We leveraged the pre-trained parameters of mPLUG-Owl [53] to enhance our model's capability in video understanding. We uniformly sample four frames from the video and set the number of temporal and spatial tokens to 64. In the first phase, we set the learning rate to 1e-4 and trained GPT4Video for two epochs. In the second phase, we adjusted the learning rate to 2e-5 and continued fine-tuning for an additional three epochs. All the experiments are conducted on 8×A100 40G GPUs.Zero-shot Video Question Answering. To assess the comprehension capabilities of GPT4Video, following VideoChat [25] , we conducted comparative experiments on two most widely-used video question answering benchmarks. The experimental results of zeroshot inference are presented in Table 1 . Specifically, we compare GPT4Video with six State-of-the-art models, including Frozen-BiLM [52] , VideoChat [25] , LLaMA Adapter [56] , Video LLaMA [55] , Video-ChatGPT [31] and Valley [28] . As can be observed, our proposed model GPT4Video reaches SOTA on all four benchmarks. Zero-shot Video Captioning. We further compared the model's performance on the video captioning task using the MSR-VTT dataset. We evaluated two traditional transformer-based methods, ORG-TRL [59] and GIT [45] , along with three LLM-based approaches, including mPLUG-2 [49] , CoDi [41] , and NExT-GPT [48] . As shown in Table 2 , our model achieved better performance on both the BLEU-4 and METEOR metrics. Zero-shot Text-to-Video Generation. We compare GPT4Video on the MSR-VTT [50] dataset for text-to-video generation task in a zero-shot manner quantitatively in Table 3 . Following prior works [2, 48] , we assess the visual quality using the FID metric and evaluate semantic consistency through the average CLIP similarity between video frames and the corresponding text. We benchmarked GPT4Video against four text-to-video specialized models: CogVideo [18] , MakeVideo [40] , Latent-VDM [38] , and Latent-Shift [2] , as well as two LLM-based methods, including CoDi [41] and NExt-GPT [48] . The results, presented in Table 3 , demonstrate that GPT4Video outperforms the aforementioned models across all evaluated metrics. It is noteworthy that while NExt-GPT integrates ZeroScope as the video decoder in its video generation process, In Table 4 , we test the safety performance of recent MLLMs in terms of popular classification metrics on our constructed safety benchmark. The understanding (VU) results show that most MLLMs had a low precision score, below 0.15, when it came to rejecting harmful queries (refusal was treated as a positive in our experiment). Differently, our method demonstrates a strong ability to reject harmful queries, while maintaining a high recall rate for general queries. Upon examining the failure cases, we find that current MLLMs can only refuse to respond to harmful queries with textual hints, such as 'this is a harmful video?' It suggests that these MLLMs' ability to detect harmful content is mainly derived from LLMs. In the generation tasks with text-only queries, our method achieves a high Precision of 90%, compared to NExT-GPT with only 37%. Our method takes a more conservative approach to generating content, which may result in the refusal to generate some general queries related to harmful content, such as sexual dancing. Despite this, our method still achieves a slightly lower but comparable recall Recall compared to NExT-GPT.Qualitative Analysis. In this section, we present some cases to demonstrate the superior video understanding and generation capabilities of GPT4Video, as shown in Figure 4 , 5. More cases are provided in supplementary materials. Ablation Study. In our ablation study presented in Table 5 , we evaluated various configurations of the video abstractor. Experiment #1 omitted temporal tokens; #2 and #3 used Q-former with 64 and 128 query tokens, respectively; and #4 applied spatial and temporal pooling, a method used in Video-ChatGPT [31] and Valley [28] . Results on the V2T task clearly demonstrate that our GPT4Video's video abstractor achieves superior performance, validating its efficacy.Comparison of T2V models. We initially selected ZeroScope model for evaluation due to its use in Next-GPT, ensuring a fair comparison. We have now incorporated evaluations using a more advanced VideoCrafter1 model. The results, as detailed in Table 6 , demonstrate superior performance of our GPT4Video in text-tovideo tasks. This enhancement underscores our method's capacity for seamless T2V model upgrades without necessitating modifications to the underlying LLM, unlike Next-GPT which requires complete retraining.  
conclusions: 
 We present GPT4Video, a novel framework that significantly enhances Large Language Models with advanced video understanding and generative functions. Our approach leverages the descriptive power of LLMs to create detailed prompts for generative models, maintaining model simplicity and flexibility. The framework's effectiveness is underscored by its superior performance on multimodal benchmarks and its innovative approach to addressing content safety issues. The release of the specialized multi-modal instruction dataset promises to catalyze future research in the field. Limitation: GPT4Video currently specializes in video modality, with plans to expand to more modalities like image and audio in future updates.Additionally, while we have initiated steps to address content safety, our consideration of safety aspects is not yet exhaustive. In future work, we will strive to further refine and enhance this aspect.  
 
 Appendix: The assumption is that the model has acquired vision-language knowledge at the first-stage training[27,63].We primarily focus on explicit content, as pornography represents the most significant risk in the realm of video generation. 


Decision: The paper introduces GPT4Video, a video language model framework for video understanding and generation. The approach innovatively utilizes a text only finetuning strategy to achieve the dual aims of understanding and generation. In a departure from previous works on video and language alignment, it introduces mechanisms to compress the video content for suitable and efficient alignment. Another valuable contribution is the introduction of a safety-aligned dataset. The proposed approach outperforms prevailing models in terms of standard metrics for video understanding and generation.

The reviewers were mostly positive about the work. There were some concerns about the flexibility of the data engine in accommodating video models and minor concerns about grammar/typos. But overall, the proposed framework and datasets should serve as a good contribution for the multimedia and specifically, video understanding/generation community. **I recommend acceptance of this work.** As promised, the authors should open-source the materials related to this work.

Reviews: review: paper_summary: paper_summary: This paper introduces GPT4Video, a framework that enhances Large Language Models (LLMs) with advanced video understanding and generative functions while also addressing content safety issues. GPT4Video employs a two-stage training strategy, with the first phase focusing on enabling the model to comprehend video content and the second phase focusing on text-only finetuning for video generation and safety guarding. The framework achieves state-of-the-art performance in zero-shot video question answering, video captioning, and text-to-video generation tasks on widely-accepted benchmarks. GPT4Video is a unified framework that integrates LLMs, visual feature extractors, and generative models for video understanding and generation. The results demonstrate the effectiveness of GPT4Video in bridging the gap between language and video understanding, showcasing its potential in various multimodal applications.
strength_weakness: strength_weakness: The paper demonstrates several strengths in terms of :
1. novelty:  It introduces GPT4Video, a framework that enhances Large Language Models (LLMs) with advanced video understanding and generative functions with state-of-the-art performance in video question answering, video captioning, text-to-video generation tasks and safeguarding against harmful content or generation. 
2. methodology: The paper presents a two-stage training strategy for GPT4Video, focusing on video comprehension and text-only finetuning for video generation and safety guarding. The integration of LLMs, visual feature extractors, and generative models showcases a well-designed theoretical approach. Finally, the validation for the efficacy of their approach is backed by the ablation study in Table 5.
3. evaluation:  The paper provides comprehensive evaluations on various multimodal tasks. The comparisons with state-of-the-art models and the use of multiple evaluation metrics strengthen the evaluation process. 
4. Clarity: The paper is very well-written and organized, making it easy to understand the proposed framework, experimental setup, and results.
5. Applications: The potential application of GPT4Video is significant in the world of multimodal understanding and in bridging the gap between language and video understanding
limitations: limitations: I think the paper is very well-written and detailed backed with substantial experiments. However, I would like to know if authors have performed any latency or resource utilization - comparison experiments on the tasks.
paper_summary: paper_summary: This paper introduces a text-only fine-tuning approach named GPT4Video that aims to enhance the video understanding and generation capabilities of LLMs. The authors propose using ViT to build a video abstractor and fine-tune Llama2 with video feature embeddings using LoRA. Experimental results demonstrate that GPT4Video outperforms existing models such as Video LLaMA, Video-Chat, and NExT-GPT in both video understanding and generation tasks.
strength_weakness: strength_weakness: 1) This work is open-source, and the authors also provide their datasets to benefit the community. 
2) The paper includes responsible AI considerations and proposes a safeguard dataset to address these concerns. 
3) The paper is easy to follow, and the figures are well-drawn.
limitations: limitations: 1) As one of the major contributions of this work, the necessity of the cross-attention mechanism needs more emphasis. Corresponding ablation studies can effectively demonstrate its efficiency. Currently, it is unclear why GPT4Video requires cross-attention and how the enhanced tokens and query tokens are generated (specifically, how Q, K, and V are obtained in Figure 2).

2) The authors claim in line 424 that they train the video extractor in the first phase. This is confusing—what does "video extractor" mean in this context? Is it the same as the Abstractor or the ViT? Please proofread the content to ensure coherence.

3) There are some typos in this manuscript, for example:
Line 182: "einforcement learning from human" should be "reinforcement learning from human"
Line 411: "we used mPLUG-Qwl’s parameters" should be "we used mPLUG-Owl’s parameters"
paper_summary: paper_summary: 1. GPT4Video is a framework that integrates video understanding and generation, with a focus on video understanding. It utilizes the parameters of mPLUG-Owl for fine-tuning, resulting in improved performance of video LLMs.

2. By conceptually integrating understanding and generation at the text level, existing text-to-video methods are integrated and invoked as repositories, achieving a fusion of understanding and generation.

3. A video QA annotation engine is obtained using the GPT4 multimodal model, which is centered around both single and multiple videos.

4. Regarding video safety, a benchmark for evaluation is proposed, and safety considerations are taken into account during fine-tuning.
strength_weakness: strength_weakness: 1. Conceptually integrates video understanding and generation, expanding the capabilities and applications of large video models.
2. Proposes a video text annotation engine centered around multiple videos.
3. Introduces the concept of safety and creates a benchmark for safety.
limitations: limitations: 1. The integration of text-to-video is simply done by using existing text-to-video methods as a library, lacking an ablation study on this module.
2. It may be worth considering merging understanding and generation at the feature level, which could be attempted as an ablation study.
3. In the data engine, could GPT4 be replaced with GPT-4V or a deployable vision-LLM?
4. Can comparisons be made with the latest video LLM for effectiveness?
paper_summary: paper_summary: This paper is about how to connect MLLMs and some video generation base models as tools for instruction-followed understanding and safety-aware generation.
strength_weakness: strength_weakness: 1. construct multiturn and caption-interleaved datasets for finetuning and benchmarking MLLMs is a good contribution.
2. safeguarding is one of the important thing in text-to-video generation and needed to be considered.
3. this framework can do both text-to-video generation and video-to-text understanding.
4. using video abstractor to extract the features from video is good.
limitations: limitations: 1. Although the dataset contribution, the method part seems to me is so engineering.
2. Since the author claim the flexible framework contributio, it would be good to see the more about the combination of different mllm and video generation models. (see only two models ZeroScope and VideoCrafter1, but no MLLM ablations found)
3. For caption task, usually we use CIDEr as the main metric, could the author also show that?