Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding

Authors: minghui  wu
anyang  su
donglin  di
tianyu  fu
ping  wang
chenxu  zhao
da  an
min  he
ya  gao
meng  ma
kun  yan
  hypergraph multi
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 07:06:09

Abstract: Understanding of video creativity and content often varies among individuals, with differences in focal points and cognitive levels across different ages, experiences, and genders. There is currently a lack of research in this area, and most existing benchmarks suffer from several drawbacks: 1) a limited number of modalities and answers with restrictive length;  2) the content and scenarios within the videos are excessively monotonous, transmitting allegories and emotions that are overly simplistic. To bridge the gap to real-world applications, we introduce a large-scale Video $\textbf{S}$ubjective $\textbf{M}$ulti-modal $\textbf{E}$valuation dataset, namely Video-SME.  Specifically, we collected real changes in Electroencephalographic (EEG) and eye-tracking regions from different demographics while they viewed identical video content. Utilizing this multi-modal dataset, we developed tasks and protocols to analyze and evaluate the extent of cognitive understanding of video content among different users.  Along with the dataset, we designed a $\textbf{H}$ypergraph $\textbf{M}$ulti-modal $\textbf{L}$arge $\textbf{L}$anguage $\textbf{M}$odel (HMLLM) to explore the associations among different demographics, video elements, EEG and eye-tracking indicators. HMLLM could bridge semantic gaps across rich modalities and integrate information beyond different modalities to perform logical reasoning. Extensive experimental evaluations on Video-SME and other additional video-based generative performance benchmarks demonstrate the effectiveness of our method. The code and dataset are available at https://github.com/mininglamp-MLLM/HMLLM

Main paper: 
 introduction: 
 With the advancement of Large Language Models (LLMs) [72] and Multi-modal Large Language Models [11, 32, 45, 46] , the eld of video understanding has entered a new era. The advanced logical reasoning abilities of multi-modal LLMs facilitate a thorough analysis of explicit elements within videos. Moreover, these models can deduce the underlying implicit content of these explicit factors, leveraging the knowledge and experience acquired by LLMs. Existing benchmarks for video content question-and-answering, such as [31, 53, 75, 75, 79] , provide a rich set of instruction labels. Alternatively, they exhibit several deciencies as illustrated in Table 1: 1) the video content itself is overly simplistic, often only involving objective, explicit factors, which does not support the exploration of deeper levels of video creativity and implicit factors. We utilize the MedScene metric to evaluate this issue, where Med-Scene denotes the median number of scene across all videos in the dataset. A higher number of scenes indicates greater complexity in video content; 2) the number of modalities included in these datasets are limited, generally conned to videos and frames; 3) the instruction labels concerning the length of answers are restricted to certain predetermined options, failing to assess the divergent and analytical abilities of LLMs. We utilize the AvgAnsLen to evaluate this issue, where AvgAnsLen represents the average text length of the answer portion across all Q&A pairs in the dataset. To address the issues mentioned above, we have prepared an extensive collection of content-rich advertisement videos, accompanied by a more comprehensive set of modality labels.In the burgeoning eld of cognitive neuroscience, the exploration of how individuals perceive and interpret video content has opened new avenues for understanding the intricate interplay between brain activity and media interaction [67] . Recent advancements in multi-modal data analysis have underscored the importance of leveraging diverse physiological signals to gain insights into the cognitive and emotional states of viewers [37] . Among these, Electroencephalographic (EEG) signals with their high temporal resolution, provide a direct measure of brain activity [59] , capturing the nuanced and dynamic changes in cognitive states as individuals engage with video content. These signals embody the electrical manifestations of the brain's complex neural dynamics, oering insights into the emotional and cognitive processes underpinning video content interpretation [58] .Inspired by the aforementioned context, we have utilized EEG and eye-tracking apparatus to collect and record the EEG and eye movement responses of individuals across various ages, genders, and professions while watching the same advertisement video. We aggregated this information into modality labels, introducing a novel, large-scale benchmark: Video Subjective Multi-modal Evaluation dataset, namely Video-SME. As illustrated in Figure 1 , our proposed dataset captures the subjective reactions of individuals watching videos through EEG and eye-tracking devices, lls the gaps in the video understanding domain regarding the assessment of video appeal and implicit factors. How to eectively leveraging these multi-modal labels to uncover the latent associations among the modalities becomes the cornerstone for addressing deeper challenges in video understanding.Graph-based methodologies exhibit superiority in exploring the associations among features, particularly hypergraphs, extending beyond traditional graph theory, oer a powerful framework for representing complex relationships in data [6] . In the context of video content analysis, hypergraphs can encapsulate the intricate associations among video elements, EEG signals, and eye-tracking data, allowing for the modeling of higher-order interactions that are not capturable through simple pairwise connections.Utilizing the multi-modal information of the Video-SME dataset, coupled with the superiority of constructing associative features through hypergraph, we proposed a Hypergraph Multi-modal Large Language Model (HMLLM), integrating information from disparate modalities to perform logical reasoning and semantic analysis. By leveraging the rich information encoded in video content, along with EEG and eye-tracking data, HMLLM can bridge semantic gaps across modalities, oering a comprehensive understanding of the cognitive processes involved in video content interpretation.The main contributions can be summarized as follows:1. Introduction of a novel large-scale benchmark dataset: the Video Subjective Multi-modal Evaluation (Video-SME) dataset, a large-scale benchmark that captures real-time EEG and eye-tracking data from a diverse demographic while they watch advertisement videos. This dataset lls a signicant gap in the eld of video understanding by providing rich modality information and a comprehensive set of question-and-answer (Q&A) pairs that allow for the assessment of video creativity and implicit factors.2. Development of the Hypergraph Multi-modal Large Language Model (HMLLM): we have developed a novel HMLLM that leverages the complex relationships among video elements, EEG signals, and eye-tracking data encapsulated in hypergraphs.3. Extensive experimental evaluations demonstrating our method's eectiveness: through rigorous experimental evaluations conducted on the Video-SME dataset and additional video Q&A datasets, we have demonstrated the eectiveness of our HMLLM. 
preliminaries 2.1 video understanding: 
 Video understanding aims to create algorithms that allow machines to interpret videos with the same expertise as humans. Meanwhile, video emotion recognition [44, 57, 84] emphasizes the interplay between the emotions conveyed by the video and the viewer responses, collectively forming a critical component of video understanding. Most existing works focus on modeling objective and tangible visual properties of videos [16] , particularly in action recognition [3, 7, 10, 17, 19, 20, 54, 61, 69, 71] and temporal action localization/detection [18, 49, 86] . However, the need for content recommendation systems has spurred research into subjective and intangible aspects (e.g. the appeal and memorability of content [14] ), where various semantically rich information are considered [5, 13, 29, 55, 85] .Compared with the above work, we present a new large-scale dataset lled with content-rich advertisement videos. This dataset includes a wider range of labels that cover both tangible and intangible aspects of content. Leveraging this dataset, we introduce an advanced hypergraph multi-modal large language model. This model is designed to simultaneously process various modalities, enabling it to conduct logical reasoning and perform in-depth semantic analysis of video content.Electroencephalography (EEG) signals provide detailed insights into brain activity related to emotions, oering spatial information on specic brain regions involved [8] . The Arousal-Valence model [60] is a key framework for classifying emotions along two dimensions. Xiaolin et al [63] explored various features to enhance the emotion recognition model. However, there's a shift towards deep learning due to the limitations of machine learning. The dynamical graph convolutional neural network (DGCNN) [62] was proposed to learn discriminative EEG features and interrelationships among EEG channels. Some works have moved towards multimodal learning for robust results in EEG signal recognition tasks, such as integrating physiological signals in the multi-modal framework to enhance emotion recognition accuracy [74] , and employing proper windowing and channel selection to avoid relying on the full length of EEG and EOG signals for classication [9] . Furthermore, advancements in neuromorphic computing led to the use of Spiking Neural Networks (SNN) [52] for classifying spatiotemporal EEG data with lower computational requirements [35] .Multi-modal Large Language Models (MLLMs), primarily serving as vision-language models, transform images or videos into texts. These models are mainly divided into two categories: traditional large-scale pretraining [39, 40, 68] and instruction tuning using pretrained LLMs [50, 78, 87] . The rst category comprises models that blend a visual encoder with a language model, either developed from scratch or based on pre-existing models, possibly including a trainable module to bridge the two modalities. Utilizing auto-regressive loss for text generation, these models are training on extensive image-text datasets, including image-text pairs [27, 39, 40, 68] and image-text sequence instances [2] . The second category, drawing inspiration from instruction-tuning techniques used in MLLMs [1, 56] , incorporates instruction-following data to enhance MLLMs' zeroand few-shot learning abilities [15, 50, 78, 87] . A notable example is LLaVA [50] , which employs a simple projection matrix to link a pre-trained visual encoder with an LLM, focusing initially on pretraining for feature alignment before comprehensive end-to-end ne-tuning. Some other works extend to video understanding by connecting video encoders to MLLMs [41, 47, 77, 81] . In addition to models that focus on combining images or videos with text, there are projects that incorporate even more types of data, like speech, audio, and sensor information [25, 64, 73, 80] .A hypergraph includes vertices and hyperedges, where hyperedges can connect multiple vertices. This structure is more adaptable and eective for representing complex relationships in data than traditional graphs [24] . Methods for creating hypergraphs fall into two groups: explicit and implicit. Explicit methods directly use the data structure to form hyperedges, like connecting vertices with shared attributes [28, 34] . Implicit methods, however, infer hyperedges from data without clear high-order links, utilizing approaches based on distance [22] or representations [33, 48, 51, 70] . Unlike static structures, some methods allow for hypergraph structure optimization, adjusting it during the learning phase. This involves adaptively changing weights on hyperedges [23] or sub-hypergraphs [83] to improve learning outcomes. Recent advancements have introduced deep hypergraph representation learning, a new approach that mainly divides into spectral [21, 76] and spatial [4, 26] categories based on how hypergraph convolution operator is dened. 
video-sme dataset: 
 In this section, we present the Video Subjective Multi-modal Evaluation (Video-SME) dataset. The Video-SME dataset not only focuses on the Objectivity Task typically found in traditional video Q&A datasets but also meticulously collects Subjective Response Indicators (SRI) to enhance the richness. It encompasses a wide array of advertisement videos across dierent industries. To capture a diverse set of responses, we enlisted participants from various cities throughout Mainland China. These participants are equipped with EEG devices, enabling us to monitor their brainwave activities and eye motion ratios (EMR) in real-time while watching the advertisements. The collected data is subsequently analyzed to establish a benchmark for the classication of brainwave and EMR responses, which is elaborated in Sections 3.1 and 3.2.Advertisement VideoScene 1 and instruction for language models. The middle section depicts the video preprocessing with Frame Sequence for Video Representation (FSVR) by scene detection and Automatic Speech Recognition (ASR) for videos. On the right side, we present our proposed semi-automated video Q&A generation process, which leverages both video storyboarding from FSVR and dialogue text from ASR. This integration enriches video content comprehension, facilitating both Subjectivity and Objectivity Tasks.Additionally, the Video-SME dataset includes an extensive video Q&A section to provide objective insights into the ads, facilitating model training and subjective index assessment. The task denition and protocol of our dataset are outlined in Section 3.3 and 3.4.The Video-SME dataset features Chinese advertising videos from diverse elds such as food and beverages, household items, consumer electronics, cultural tourism, software, and automobiles. It comprises 498 curated landscape videos sourced from online platforms and TV commercial ads, each running for 15-30 seconds.In this study, we introduce the Frame Sequence for Video Representation (FSVR) strategy to preprocess advertisement videos, as depicted in the middle part of Figure 2 . We enhance the video scene sensitivity by integrating the AdaptiveDetector 1 for FSVR with specic parameters: adaptive_threshold = 2, min_scene_len = 10, window_width = 2. In the case of advertisement videos with frequent scene changes, the scene detection algorithm captures more information compared to average frame capture methods. Moreover, it is invaluable in minimizing redundant frames in videos primarily composed of static scenes.By employing FSVR, we are able to deconstruct the temporal sequence of advertisement video frames, achieving capabilities including modality signal alignment, video content understanding, and semi-automated Q&A instruction generation.1 https://www.scenedetect.com/We developed a sophisticated system for collecting subjective indicators. Each participant watches a series of advertisement videos using the device described in the appendix. During this process, we synchronously gather EEG and eye-tracking data, along with anonymized demographic details. Our study includes over 4,600 participants, ensuring a wide demographic representation. The participant base spans white-collar workers, civil servants, students, and freelancers across various age groups and income brackets.The raw EEG signals are characterized by parameters such as U 1 , U 2 . . . V 2 , V 3 [36, 38] , which is detailed in the appendix. Given the unique demands of advertisement video analysis, we pinpointed two pivotal EEG metrics: engagement and emotion, as delineated by Equation 1 and Equation 2, respectively.EQUATIONEQUATIONwhere ⇢# C and ⇢" C represent the engagement and emotion of the individual user at the sampling moment, respectively. Furthermore, we tracked eye movement data, dening the Eye Movement Ratio (⇢"' C ) as the proportion of time the participant's gaze xates on the display relative to the total video duration.The SRI Collection & Determine workow, depicted on the left of Figure 2 , captures sub-second high-frequency raw signals data. To align with video content's scene-based evolution, Video Storyboard-Timed SRI Synchronization was adopted, producing time-averaged and participant-specic SRIs. Demographic characteristics then grouped these SRIs into units of 5-20 same-gender participants with a maximum age dierence of 5 years, such as {female, <20}, {male, 26-30}, and {female, 46-50}, as Demographic-Based SRI Aggregation in Equation 3.EQUATIONwhere -? 8 ,C 9 denotes the original SRI such as ⇢# C , ⇢" C , and ⇢"' C . Each indicator associated with discrete values for participant ? 8 at specic timestamps C 9 , where C 9 signies the eective sampling moment instances within the video storyboard timeframe from FSVR in Section 3.1.For quantitative analysis, we meticulously examined data distribution across various Audience Prole segments. Engagement was categorized into two groups using the Leuven Engagement Scale (LES) and its distribution. Emotion and EMR indicators, which followed normal distributions, were divided into three equal categories. For detailed data distribution, refer to the appendix. The SRI Instruction Generation protocol is detailed in Table 2 .In addition to subjective indicators from Audience Proles, we developed a semi-automated annotation pipeline for ChatGPT4-Vision (GPT4V) to obtain Objective Video Q&A, depicted in Figure 2 . Although GPT4V cannot process videos, it supports multiple consecutive key-frames simultaneously. Based on FSVR in Video Preprocessing, we extracted middle frames from each shot as keyframes that eectively represent the entire video. During each invocation of GPT4V to automatically generate answers, questions are selected randomly from the Random Question Set to enhance the diversity of Q&A sessions, along with providing ASR text and FSVR key-frames. Lastly, annotators were carefully selected to manually rene objective Q&A instruction from Automated Answer Generation, addressing issues like advertisement branding, expression errors, and factual inaccuracies.Based on the processing presented in Sections 3.2 and 3.3, Video-SME is categorized into subjectivity and objectivity tasks. The subjectivity task examines the SRI, whereas the objectivity task is dedicated to the qualitative analysis of video content and audience perception. As shown in Table 2 , we present the tasks, protocols, and instructions associated with the Video-SME dataset.Task 1, entitled Subjectivity, is formulated as a classication task, aimed at examining the inuence of video content and user characteristics on the SRI. We develope two experimental protocols to guide this investigation. The rst protocol (P1) is designed to assess the SRI ability of a broad audience, involving the analysis of average responses across dierent videos. This approach is relatively straightforward. The second protocol (P2) introduces a layer of complexity by focusing on the SRI discernment of particular user demographics. This necessitates a comprehensive examination of how response patterns uctuate among diverse user cohorts.Task 2, designated as Objectivity, mirrors the video Q&A tasks prevalent in prior datasets, as described in Section 3.3. Building on the method outlined in [53] , this study conducts a supervised analysis of the answers generated, assessing their accuracy and allocating scores. This approach is designed to objectively ascertain the narrative coherence of the advertisement content and its ecacy in captivating the target audiences. 
method: 
 This section elaborates on the Hypergraph Multi-modal Large Language Model (HMLLM), an approach designed to intelligently process video clips and textual prompts for generating contextually relevant text, including Subjective Response Indicators (SRI). Central to our methodology are several key components as depicted in Figure 3 : Visual Encoder, Query Former (Q-Former), SALM Projector, SRI-Aware Language Model (SALM), and SAL-HL Module. All components mentioned above synergistically orchestrated across two primary phases: SALM Warm-Up and SAL-HL Fine-Tuning, as depicted in our model architecture (refer to Figure 3 ). The pseudocode in the appendix illustrates the detailed training process.We begin by detailing the initial stage. The approach ingests brief video clips and corresponding textual prompts, extracting key frames from the videos using a predened, static extraction strategy, which can be either random or uniformly distributed. These key frames are represented as = {5 0 , 5 1 , . . . , 5 # }, with # signifying the number of extracted frames. These key frames are then pre-processed to form the initial data matrix, denoted by X 0 2 R ⌫⇥⇠ ⇥# ⇥⌘⇥F , where ⌫, ⇠, # , ⌘, and F correspond to the batch size, color channels (RGB), the number of keyframes, and the resized dimensions of the frames, respectively. The initial data matrix X 0 is fed into a pre-trained visual encoder to yield initial visual representations, expressed as F E 2 R ⌫⇥# ⇥ ! ⇥ ⇠ , with ! and ⇠ representing the length and channels of features, respectively. During the rst training phase, the "Hypergraph Learning Gate (HL-Gate)" remains inactive while the Q-Former and SALM are warmed up. The visual features F E are then input into the frozen Q-Former as the Key (K 2 R ⌫⇥ (# ⇥ ! ) ⇥ ⇠ ) and Value (V 2 R ⌫⇥ (# ⇥ ! ) ) for the attention mechanism. The Query in the Q-Former is initialized as either a random or null set, represented by Q 2 R ⌫⇥ ( Q⇥ C @ ) , where Q ⇥ C @ are the predened hyper-parameters for the length and channels of the query. Subsequently, we introduce an "SALM Projector", a multi-layer perceptron that follows the Q-Former, capable of reshaping the data and introducing additional learning parameters into the model. The output of projector is denoted Figure 3 : Overview of the HMLLM. The architecture comprises a suite of pre-trained models, including a "Visual Encoder", "Q-Former", and the "SRI-Aware Language Model (SALM)", which are initially frozen and subsequently ne-tuned through strategic training procedures. More importantly, our model incorporates a designed "SRI-Aware Language Hypergraph Learning (SAL-HL)" module that is trained de novo via a combined loss function. During inference, the HMLLM generates SRI and Q&A responses tailored to the video content, thereby providing a deeper level of engagement and comprehension.as F ? 2 R ⌫⇥ ( Q⇥ C ? ) , with C ? being another predened hyperparameter. The SRI-Aware Language Model (SALM) is then engaged, taking the output of the SALM Projector (F ? ) and the corresponding textual instructions as inputs during the initial warm-up training stage. The SALM is trained using the Image-grounded Text Generation (ITG) loss function [40] (L )⌧ ), which instructs the Q-Former to generate text conditioned on the input images. The goal of the ITG loss is to minimize the dierence between the generated caption e Y @0 SALM(F ? ,) ) and the ground-truth caption Y 6C . This is typically achieved using a cross-entropy loss computed over the words or tokens in the caption. The ITG loss function can be mathematically represented as:L )⌧ = ' ⇣ log P(Y 6C 8 |Y 6C 1 , • • • , Y 6C 8 1 , F E ) ⌘ (4)whereP(Y 6C 8 |Y 6C 1 , • • • , Y 6C 8 1 , F E )denotes the probability of generating the 8-th word in the caption given the previous words and the visual features extracted from the image. The summation encompasses all words or tokens in the ground-truth caption.In our approach, we integrate specic strategies from BLIP2 [40] to address the limitation of Q-Former architecture on direct interactions between the image encoder and text tokens. Following the aforementioned training procedure, the SALM Projector and SALM are adequately warmed up, preparing them for subsequent ne-tuning optimization.In the subsequent ne-tuning phase, the hypergraph learning gate (HL-Gate) is activated, and the hypergraph learning module (SAL-HL) undergoes training in tandem with the ne-tuning of the SRI-Aware Language Model (SALM). As delineated in Figure 3 , the SAL-HL module receives the initial visual features (F E ) and the representations of the projected frames (F ? ) produced by the warmed SALM Projector as inputs.The SAL-HL module initiates the process by merging these two feature sets (i.e., F ? , F E ) and then pooling them to generate framelevel representations (F 5 A0<4_;4E4; ). This process is formulated as:EQUATIONThe Feature_Mixer denotes the mixing operation between two feature matrices, which can be implemented as a multi-layer perceptron (MLP). Each frame, denoted as 5 8 for 8 2 [0, # ], is considered a vertex (V) within the hypergraph structure (G), which facilitates the establishment of high-order relationships among the frames. The construction of the hypergraph entails the application of a clustering algorithm that links frames with similar latent visual features. After constructing the hypergraph, we proceed to train the Hypergraph Neural Network (HGNN) [21] in parallel with the Structured Attention Layer Mechanism (SALM). This process is mathematically formulated as follows:eEQUATIONwhere e Y BA8 represents the predicted output from the SALM-enhanced HGNN, and f denotes a non-linear activation function, which introduces the necessary non-linearity into the model for capturing complex patterns. D 4 2 R ⇢⇥⇢ , D E 2 R # ⇥# , and W 2 R ⇢⇥⇢ denote the diagonal degree matrix of hyperedges, the degree matrix of vertices, and weight matrix of hyperedges, respectively. H 2 R # ⇥⇢ signies the incidence matrix that connects hyperedges to their constituent vertices. f (•) denotes the nonlinear activation function (e.g., LeakyReLU(•)). Θ is a diagonal matrix representing the learnable parameters updated by the Cross_Entropy loss function in the EQUATIONwhere _ is a hyperparameter that balances the inuence of the Cross-Entropy loss and the ITG loss on the overall optimization process. This composite loss function ensures that the model not only generates text that is grounded in the visual content but also adheres to the learned high-order relationships within the hypergraph structure. This enhances the model's capability to capture intricate interactions and dependencies among video frames. 
experiment: 
 Metrics. In our study, the Subjectivity Task of Video-SME is structured in a multiple-choice question (MC) format. To evaluate its For its evaluation, we introduce evaluation measures [53] based on assessments using GPT-3.5 Turbo. Implementation Details. We employ UMT-L [43] as the visual encoder and Vicuna-7B-v0 [12] as the base model for the SRI-Aware Language Model (SALM). Following the methodology of BLIP2 [40] , QFormer is congured with 64 queries. Throughout both the inference and training phases, we adhere to the FSVR strategy detailed in Section 3.1, which involves representing each video with 8 key frames. Further details can be found in the appendix. 3 , we present our proposed method HMLLM alongside those from renowned MLLMs such as Geminipro-vision, GPT4V, Video-LLaVA, and Video-Chat2. The results, observed from top to bottom, indicate: 1) For zero-shot inference, GPT4V and Video-LLaVA signicantly outperformed the random across-the-board. Gemini-pro-vision underperformed the Random baseline in predicting the accuracy of Question: Who is the audience of the video?The video appears to be a commercial aimed at consumers who are interested in snacks, particularly those who enjoy chocolate and possibly matcha-flavored treats. The presence of both an adult male and a young girl suggests that the product is being marketed to a wide age range, including families. The product is likely positioned as a treat that can be enjoyed by both adults and children, indicating a universal appeal. The focus on the sensory experience of eating the snack, with close-ups of the man enjoying the food and the vibrant green color of the matcha, suggests that the video is trying to entice viewers with the taste and quality of the product. At the end of the video , the brand name-ORLON is prominently displayed to encourage brand recognition and purchase. …According to the dimensions of the video, the audience for this video is children and parents.The video seems to be targeted towards people who enjoy eating ice cream and desserts. The audience can be assumed to be people who are interested in trying out new flavors and indulging in sweet treats.The video is meant for an Asian audience.The audience of this video is likely to be consumers, especially those who are interested in food or snacks. The video shows a man and a little girl enjoying food, and the scene of the little girl eating food looks very enjoyable, which may be aimed at attracting the attention of children or young people. The video also includes the display of food packaging and brand information, which is usually aimed at promoting the product to the audience. The brand and packaging design of the food may be aimed at attracting the attention of young consumers, and the video may be a commercial advertisement aimed at attracting family audiences. Engagement and Emotion. Video-Chat2's failure to follow instructions made it dicult to obtain meaningful results. The settings and prompts of zero-shot inference can be found in the appendix.2) Upon ne-tuning the models with the Video-SME dataset, we observed notable improvements in performance for both Video-LLaVA and Video-Chat2 across both P1 and P2, compared to their initial zero-shot congurations. Moreover, our proposed HMLLM demonstrated the most substantial advancements, consistently outperforming other state-of-the-art methods across all evaluated metrics and protocols.In detail, within Protocol 1, HMLLM surpassed the leading benchmarks in the categories of Engagement, Emotion, and EMR Duration. The improvements were remarkable, showing enhancements in (accuracy, F1) scores by (3.07, 2.31), (6.08, 2.89), and (1.00, 0.63), respectively. These results underscored the ecacy of our method in accurately capturing and analyzing both engagement and emotional dynamics, as well as predicting EMR duration with high precision. For Protocol 2, the superiority of HMLLM is equally evident. Again, it outshoned the best-existing benchmarks with enhancements in (accuracy, F1) scores by (4.37, 1.34), (3.54, 4.12) , and (7.90, 5.40), respectively. These ndings highlight the robustness and adaptability of our model across dierent protocols, establishing its potential for widespread applicability in real-world scenarios. Objectivity Task. In the exploration of the objectivity task, as detailed in Section 3.3, we meticulously rened the ground truth (GT) by manually correcting annotations initially provided by GPT4V. This meticulous process contributed to the notably high zero-shot inference capabilities observed for GPT4V. Given that Gemini-provision and GPT4V inherently lack support for video inputs, we integrated FSVR to bridge this gap. This adaptation endowed both models with the ability to process video inputs, thus expanding their applicability across a wider range of tasks. As shown in Table 4 , GTP4V became the upper bound in a zero-shot setting because we semi-automatically utilized it for labeling, as described in Section 3.3. When the narrative shifts upon the ne-tuning of our models with the Video-SME dataset. Both Video-LLaVA and Video-Chat2 showcased enhancements in their performance metrics, surpassing their initial zero-shot congurations. This improvement highlights the transformative impact of targeted training on model ecacy. Notably, our proposed HMLLM method emerged as a formidable contender, eclipsing other models in performance across the board. Specically, HMLLM outperformed the best baseline, Video-Chat2, in terms of Acc and the Score [53] by 1.25 and 0.01, respectively.The results not only validate the eectiveness of ne-tuning with the Video-SME dataset but also emphasize that our HMLLM method sets a new benchmark in model performance. 5.1.2 Intra Testing on Video Conversation Benchmark. To further validate the performance of HMLLM, we conducted experiments on other video-based generative performance benchmarks. Following the setup of Video-ChatGPT [53] , we present the performance of our proposed HMLLM, detailed in the last row of Table 5 . Experimental results demonstrate that the HMLLM eectively enhances both Contextual Understanding and Consistency. Given the HMLLM did not overemphasize temporal details, a slight decrease in Temporal Understanding was observed.We further present a qualitative comparison in Figure 4 . HMLLM demonstrates an enhanced ability to generate longer and more comprehensive responses for Objectivity Tasks. This improvement can be attributed to the longer average context length of our dataset, which facilitates a deeper understanding of video content by enabling detailed analysis of advertising plots and visual elements. More detailed qualitative analyses are available in the appendix. 
conclusion: 
 In this paper, we released a large-scale Video-SME dataset with two challenging tasks. We hope it will push cutting-edge research in video understanding. Besides, we proposed a novel HMLLM approach that enhances the language model by constructing a hypergraph feature space across modalities, thereby providing semantically richer associative features. Finally, we conducted a comprehensive set of experiments on both Video-SME and other videobased generative datasets, verifying the signicance of the proposed dataset and method. 
 
 Appendix:  


Decision: The initial reviews were positive with 4x weak accept. After the rebuttal, one reviewer upgraded their review to a strong accept, while one other confirmed their weak accept.

The reviewers agree in this submission including (+) a innovative dataset and (+) having novelty in the use of hypergraph learning. 

While there are some minor concerns on (-) the generalizability of subjective data, including cultural biases through the dataset being Chinese, the overall concerns seem limited hence the positive ratings.

I believe this paper should definitive be accepted.

Reviews: review: paper_summary: paper_summary: The paper introduces the Hypergraph Multi-modal Large Language Model (HMLLM) designed to integrate and analyze multi-modal data, including Electroencephalographic (EEG) and eye-tracking signals, for understanding video content. The authors present a novel large-scale dataset, Subjective Response Indicators for Advertisement Videos (SRI-ADV), which captures real-time EEG and eye-tracking data from diverse demographics watching advertisement videos. This dataset aims to bridge the gap in video understanding by providing rich modality information and a comprehensive set of question-and-answer (Q&A) pairs. The proposed HMLLM utilizes hypergraph learning to represent complex relationships among video elements, EEG signals, and eye-tracking data, enhancing logical reasoning and semantic analysis.
strength_weakness: strength_weakness: The introduction of the SRI-ADV dataset is a significant contribution, offering a rich, multi-modal dataset that includes real-time EEG and eye-tracking data. This dataset addresses the limitations of existing video understanding benchmarks by providing more modalities. 

The use of hypergraph learning to represent and analyze complex relationships among different modalities is a novel approach that enhances the model's ability to perform logical reasoning and semantic analysis.

The experimental evaluations demonstrate the effectiveness of HMLLM, showing improvements over existing models.
limitations: limitations: The paper does not explicitly mention the computational efficiency or complexity analysis of the HMLLM model. However, given the integration of multiple modalities and the use of hypergraph learning, it is reasonable to infer that the model might be computationally intensive, potentially requiring significant resources for training and fine-tuning.

While the model performs well on the SRI-ADV dataset, its generalization to other types of video content and different demographic data is only briefly discussed in Section 5.1.2. Further experiments on varied datasets would strengthen the validation of the model.
paper_summary: paper_summary: The paper presents a comprehensive study on a Hypergraph Multi-modal Large Language Model (HMLLM), developed to enhance video understanding by integrating multiple data modalities, including EEG signals and eye-tracking data. The paper provides these main components:
1.	SRI-ADV Dataset: Introduction of a large-scale dataset, that captures EEG and eye-tracking data from diverse demographics while watching advertisement videos.
2.	HMLLM Algorithm: Hypergraph Multi-modal Large Language Model that employs hypergraph structures to encode complex relationships among video elements, EEG signals, and eye-tracking data.
3.	Results: Extensive experimental evaluations demonstrating the effectiveness of the proposed model on the SRI-ADV dataset and additional video-based generative datasets.
strength_weakness: strength_weakness: 1.	Innovative Dataset: The SRI-ADV dataset fills a gap in video understanding research by providing rich modality information and comprehensive Q&A pairs, allowing for the assessment of video creativity and implicit factors.
2.	Model Architecture: The use of hypergraphs to represent complex relationships among different data modalities is fits well in this problem as its ability to encapsulate logical reasoning and semantic analysis based on different multimodalities.
3.	Comprehensive Analysis: The paper includes thorough experimental evaluations and qualitative comparisons, that demonstrate the model's ability to generate longer, more comprehensive responses for objectivity tasks.
4.	Writing of the paper. The paper is well-written and easy to follow.
limitations: limitations: -	Heterogeneous Data Types: Despite the importance of mixing these modalities for understanding subjects’ responses while watching ads videos, EEG signals, eye-tracking data, and video content are fundamentally different types of data, each with unique characteristics and formats. Integrating these heterogeneous data types into a unified representation is inherently complex. It is  unclear in the paper how the authors achieved the temporal and spatial alignment between these data streams to accurately capture the relationships between them requires sophisticated preprocessing and alignment techniques. Misalignment can lead to incorrect or misleading interpretations of the data.

-	Computational power of large scale data: Constructing hypergraphs for large scale datasets, like the SRI-ADV dataset, requires substantial computational power and memory. I think the authors need to shed some light on this aspect in the paper.
paper_summary: paper_summary: This paper presents a large-scale video dataset, which captures real-time EEG and eye-tracking data, including 498 videos and 178k Q&A pairs. The authors also introduce a multi-modal large language model that leverages EEG and eye-tracking data to assess heterogeneous responses to video content. Extensive experiments on the SRI-ADV dataset and additional benchmarks demonstrate the model's effectiveness.
strength_weakness: strength_weakness: 1.	The proposed large-scale dataset will be useful and benefit the community.
2.	Extensive experiments demonstrate the effectiveness of proposed method.
3.	Ablation studies verify that each component is an optimal solution.
limitations: limitations: 1.	The subjective response indicator is pivotal in this paper; however, there is a need for a clearer definition of what it specifically encompasses. It is unclear whether it includes emotions, engagement levels, or other relevant factors such as user satisfaction, attention span, or cognitive load. Providing a more detailed explanation of the components of the subjective response indicator would enhance the reader's understanding and the overall impact of the study.
2.	Video emotion analysis is a well-studied area, and the proposed method actually involves emotion in dataset construction and methodology design. However, there is no discussion or comparison involving video emotion analysis. This may lead the audience to misunderstand the contribution of this work.

[1] Representation Learning through Multimodal Attention and Time-Sync Comments for Affective Video Content Analysis, ACM MM22

[2] Weakly Supervised Video Emotion Detection and Prediction via Cross-Modal Temporal Erasing Network, CVPR23

[3] Towards Robust Multimodal Sentiment Analysis under Uncertain Signal Missing, SPL23
paper_summary: paper_summary: The paper proposes a novel large-scale dataset, SRI-ADV, and a Hypergraph Multi-modal Large Language Model (HMLLM) that utilizes EEG and eye-tracking data to enhance video understanding. This work is a substantial contribution to cognitive neuroscience and video content analysis, highlighting the importance of multi-modal data for understanding user engagement and emotional responses.
strength_weakness: strength_weakness: 1. The introduction of the SRI-ADV dataset, which integrates EEG and eye-tracking data with video content analysis, addresses a gap in current research. This dataset not only enhances the complexity of the analysis but also provides a comprehensive approach to understanding subjective user engagement.
2. The development of HMLLM, which leverages hypergraph techniques to analyze the relationships among video elements, EEG signals, and eye-tracking data, is interesting. The model's ability to bridge semantic gaps across modalities shows potential for deep insights into user cognition.
limitations: limitations: 1. The paper lacks a detailed discussion on the generalizability of the proposed model to other forms of media content beyond advertisement videos. It's unclear how the model performs in different contexts or with different types of video content, which could limit its application in broader scenarios.
2. The collection of data primarily from viewers in Mainland China might introduce cultural biases in model training and interpretation. Expanding the dataset to include a more diverse global audience could enhance the model's robustness and applicability.
3. Integrating additional modalities, such as acoustic signals or textual annotations, could provide a richer understanding of user interactions with video content.