Hawkeye: Discovering and Grounding Implicit Anomalous Sentiment in Recon-videos via Scene-enhanced Video Large Language Model

Authors: jianing  zhao
jingjing  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 04:20:03

Abstract: In real-world recon-videos such as surveillance and drone reconnaissance videos, commonly used explicit language, acoustic and facial expressions information is often missing. However, these videos are always rich in anomalous sentiments (e.g., criminal tendencies), which urgently requires the implicit scene information (e.g., actions and object relations) to fast and precisely identify these anomalous sentiments. Motivated by this, this paper proposes a new chat-paradigm Implicit anomalous sentiment Discovering and grounding (IasDig) task, aiming to interactively, fast discovering and grounding anomalous sentiments in recon-videos via leveraging the implicit scene information (i.e., actions and object relations). Furthermore, this paper believes that this IasDig task faces two key challenges, i.e., scene modeling and scene balancing. To this end, this paper proposes a new Scene-enhanced Video Large Language Model named Hawkeye, i.e., acting like a raptor (e.g., a Hawk) to discover and locate prey, for the IasDig task. Specifically, this approach designs a graph-structured scene modeling module and a balanced heterogeneous MoE module to address the above two challenges, respectively. Extensive experimental results on our constructed scene-sparsity and scene-density IasDig datasets demonstrate the great advantage of Hawkeye to IasDig over the advanced Video-LLM baselines, especially on the metric of false negative rates. This justifies the importance of the scene information for identifying implicit anomalous sentiments and the impressive practicality of Hawkeye for real-world applications.

Main paper: 
 introduction: 
 In the literature, Multimodal Affective Computing (MAC) [16, 55] routinely focuses on leveraging the explicit information (e.g., acoustic, facial expressions and language) for precisely understanding the sentiment of videos. These studies can be broadly divided into two paradigms: fusion paradigm [8, 46] and interaction paradigm [18, 41, 55] . However, in real world, there exists a vast amount of recon-videos, such as surveillance and drone reconnaissance videos, wherein the above explicit information is often missing, while these videos are always rich in negative sentiments (e.g., criminal tendencies). Under this scenario, it is urgent to leverage the implicit scene information (e.g., actions and object relations) for precisely discovering and grounding anomalous sentiments. In this paper, we define the above recon-videos involved negative sentiments, which require the implicit scene information for identifying, as the implicit anomalous sentiments.With these in mind, we propose a new Implicit anomalous sentiment Discovering and grounding (IasDig) task, which leverages Video-LLMs through a new chat paradigm for implicit anomalous sentiment identification. This task aims at discovering and grounding (i.e., classifying and localizing) anomalous sentiments in recon-videos through implicit scene information, such as actions and object relations. To be specific, this IasDig task discovers and locates the anomalous sentiments segments within the reconvideos through interactions with LLM. For example, in Figure 1 , a man near the door with his back to the camera is shooting at someone during the ground truth timestamps from 15s to 17s in the recon-video, where the LLM is required to locate the timestamps of implicit anomalous sentiment displays by the man. Under this circumstance, LLM should fully use the scene information for precisely identifying implicit anomalous sentiments. In this paper, we seek to leverage the advanced video comprehension capabilities of Video-LLMs (e.g., Video-LLaVA [29] ) to address our IasDig task. However, current LLMs have deficiencies in scene understanding as reported by [27, 35] . In this way, we believe that leveraging Video-LLMs to comprehend implicit scene information at least faces two main challenges.On one hand, it is challenging to model the scene information (i.e., action and object relations) in recon-videos. As shown in Figure 1 , considering the recon-video with obscured facial expressions due to the camera angle, it is significantly difficult to precisely discover and localize anomalous sentiments. In this situation, taking the implicit scene information, such as action (shooting at) and object relations (<man, near, door>) into consideration can help better identifying anomalous sentiments. Therefore, how to model the scene information is rather important. Fortunately, some prior works have recognized this problem and utilize graph structures to model object relations information [10, 48] . Actually, the joints of the movements of human body can also be viewed as graph structures. Inspired by this, this paper believes that a well-behaved IasDig approach should consider leveraging graph structures to model both the action and the object relations scene information for further enhancing the scene understanding ability of Video-LLMs.On the other hand, it is challenging to balance the above two heterogeneous scene information during the alignment phase of Video-LLMs. Still take the recon-video in Figure 1 as an example, from 15s to 17s, the object relations information indicates a man near the door, whereas the action information depicts a man shooting at someone, which is a strong signal of the presence of anomalous sentiments. Assigning equal weight to both two types of scene information could lead to interference from object information, causing the model to mistakenly categorize this video segment as non-threatening. However, existing Video-LLMs process visual features as a whole, lacking the ability to balance the weights of individual elements. Therefore, this paper believes that a betterbehaved Video-LLM approach for IasDig should further consider balancing the two heterogeneous scene information.To this end, this paper proposes a tailored Scene-enhanced Video Large Language Model named Hawkeye, i.e., acting like a raptor (e.g., a Hawk) to discover and locate prey, for the IasDig task to tackle the above two challenges simultaneously. Specifically, this approach designs a Graph-structured Scene Modeling Module to model the scene information, wherein an Action-Sensitive Graph (ASG) and an Object-Relation sensitive Graph (ORG) are proposed to model the action and object relations information respectively. Furthermore, inspired by Mixture-of-Experts (MoE) [15, 19] , this approach designs a Balanced Heterogeneous MoE Module (B-H MoE) along with a scene-balancing loss function to weight the heterogeneity between the two scene information. Furthermore, this paper constructs two scene-sparsity and scene-density IasDig datasets to evaluate the effectiveness of Hawkeye, and detailed experiments demonstrate that Hawkeye achieves significant performance improvements compared to the advanced Video-LLMs. 
related work 2.1 multimodal affective computing: 
 Previous studies on Multimodal Affective Computing (MAC) leverage explicit signals to predict sentiments [20, 65] . They can be broadly divided into fusion paradigm [8, 46] and interaction paradigm [18, 41, 55] . Recent studies focus more on the latter paradigm. SELF-MM [57] leverages self-supervised learning to acquire the consistency and difference of modalities. UniMSE [18] and ConFEDE [55] introduce contrastive learning for mutimodal information representation. However, these studies mainly focus on explicit sentiment analysis. Recently, a few studies pose the Implicit Sentiment Analysis (ISA) while only focusing on language. Zhou et al. [63] propose composition mechanism to event-centric ISA. Fei et al. [12] introduce chain-of-thought to mimic the reasoning process. Different from all the above studies, we propose a new chat-paradigm Implicit anomalous sentiment Discovering and grounding (IasDig) task to identify implicit anomalous sentiments in recon-videos via implicit scene information. To our best knowledge, this is the first attempt to incorporate these scene information in MAC.Video Temporal Localization (VTL) [36] and Referring Video Object Segmentation (RVOS) [23] are two tasks within Video Grounding Answer: A man in suit and sunglasses. A person raises right hand, waves. (VG) [60] . VTL aims to utilize language query to localize temporal segments, while RVOS focuses on tracking objects correspond to the language query. For VTL, it can be broadly divided into two forms: one-stage models [42, 54] and two-stage models [30, 49, 62] . Besides, Chen et al. [4] propose an audio-enhanced approach. For RVOS, it can be divided into offline [3, 51] and online [52] processing. Particularly, Wasim et al. [50] propose an open vocabulary VG task to localize and recognize objects and actions in the video. Considering the sentiment in the video, Zhang and Yang [61] propose a temporal sentiment localization task to simultaneously localize and classify the sentiments in long untrimmed videos. Further, a relevant task for us is video-based anomaly detection [44, 53] , which provides some inspiration for defining our new task, IasDig. However, existing works primarily focus on classifying the objective abnormal events, while overlook the subjective implicit anomalous sentiments especially for localizing these sentiments. Besides, these works struggle to adapt to the interactive demands of the LLM era. Different from all the above studies, this paper aims to classify and localize implicit anomalous sentiments in recon-videos, which is adapt to interactive scenarios in the era of LLMs and seek to leverage implicit scene information to perform better identification.The emergence of ChatGPT [38] has sparked a boom in the field of NLP with LLMs, such as LLaMA [45] and Vicuna [9] . Some studies [5, 6, 25, 31] expand LLMs to multimodal filed. All these studies pave the way for the development of Video-oriented Large Language Models (Video-LLMs). According to the visual encoder, Video-LLMs can be categorized into four types. 1) VideoChat [26] and Video-LLaMA [59] utilize BLIP-2 [25] and use the Q-former to map visual representations to Vicuna. 2) Video-ChatGPT [34] , Otter [24] , Valley [33] , mPLUG-Owl [56] and ChatUniVi [21] take advantage of CLIP [40] to obtain visual features. 3) PandaGPT [43] employs the ImageBind [13] as its backbone for video comprehension. 4) Video-LLaVA [29] leverages LanguageBind [64] to pre-align image and video features into the language feature space. However, all the Video-LLMs have overlooked the scene information in the video. Recently, a few studies [27, 35] consider incorporating scene information in images, while understanding scene in videos is almost nonexistent. Besides, some studies [15, 28] introduce the concept of MoE into LLMs, but they only focus on the efficiency, without considering the balance between different information. In summary, although all the above studies of Video-LLMs have achieved great success in video understanding, the ethical constraints limit their ability to analyze harmful content [37] , therefore have difficulty tackling the IasDig task. This paper proposes a Scene-enhanced Video-LLM called Hawkeye, aiming to elicit the ability of Video-LLMs in scene understanding through scene modeling, and design a Balanced Heterogeneous MoE module to balance the scene information. 
approach: 
 Problem Formulation. Given a recon-video 𝑉 with 𝑇 frames, each frame 𝑡 is labeled with 1 or 0, where 1 and 0 represent whether this frame conveys implicit anomalous sentiments. The goal of IasDig is to interactively discover and ground (i.e., classify and localize) the presence of implicit anomalous sentiments in 𝑉 , and generate a set of segments {(𝑠 1 , 𝑒 1 ), ..., (𝑠 𝑖 , 𝑒 𝑖 ), ..., (𝑠 𝑛 , 𝑒 𝑛 )}, where 𝑠 𝑖 and 𝑒 𝑖 denote the start and end time of an implicit anomalous sentiment segment. Challenges. The IasDig task faces two key challenges: scene modeling and scene balancing. To address the two challenges, we propose two modules: a Graph-structured Scene Modeling Module and a Balanced Heterogeneous MoE Module (B-H MoE). Backbone. Several Video-LLMs are open-sourced currently, we consider using Video-LLaVA [29] and its visual encoder Language-Bind [64] as the backbone. It is optimized using a mixed dataset of images and videos, and achieves the best performance on most of the image and video benchmarks. Therefore, we utilize Video-LLaVA as the backbone to explore the capabilities of Video-LLMs in identifying implicit anomalous sentiments of recon-videos.We design a Graph-structured Scene Modeling Module to model the implicit scene information, which consists of an Action-Sensitive Graph (ASG) and an Object-Relation sensitive Graph (ORG), detailed as follows.Action-sensitive Graph is designed to model the action information of individuals. Specifically, we address two crucial questions: 1) how to capture the action information of individuals; 2) how to make full use of the action information in the IasDig task. We will provide comprehensive answers to these two questions in the subsequent section, formulated as follows.For question 1), we leverage HigherHRNet1 [7] , a well-studied bottom-up human pose estimation network, to obtain the action information. As shown in ASG, given a sequence of video frames, HigherHRNet is able to generate an action graph token 𝒙 𝒂 𝒊 consisting of 17 nodes of human joints for each individual in each frame, where 𝑖 represents the 𝑖-th frame. Then, we will get the full action graph tokens X 𝒂 = {𝒙 𝒂 1 , ..., 𝒙 𝒂 𝒊 , ..., 𝒙 𝒂 𝒏 } of the video sequence 2 , where 𝑛 represents the number of frames in this sequence.For question 2), we leverage Action-aware Graph Attention to enrich video tokens X 𝒗 = {𝒙 𝒗 1 , ..., 𝒙 𝒗 𝒊 , ..., 𝒙 𝒗 𝒏 } from the Video Encoder with X 𝒂 . For each node 𝑒 𝑘 in 𝒙 𝒂 𝒊 , we first compute the attention weights 𝛼 𝑘 𝑗 between it and its neighboring node 𝑒 𝑗 :EQUATIONwhere W is the weight matrix, ℎ 𝑘 and ℎ 𝑗 represent the features of 𝑒 𝑘 and 𝑒 𝑗 , and 𝑑 represents the dimension of features.Then 𝛼 𝑘 𝑗 can be used to aggregate the feature ĥ𝑘 of node 𝑒 𝑘 : ĥ𝑘 = 𝑗 ∈ N (𝑒 𝑘 ) 𝛼 𝑘 𝑗 • ℎ 𝑗 , where N (𝑒 𝑘 ) denotes the set of neighboring nodes of node 𝑒 𝑘 . The final feature of 𝑒 𝑘 is formulated as:ℎ ′ 𝑘 = ReLU(W[ ĥ𝑘 , ℎ 𝑘 ]), where W is the weight matrix and [ ĥ𝑘 , ℎ 𝑘 ] denotes the concatenation of ĥ𝑘 and ℎ 𝑘 .After the graph attention operation, we consider enriching X 𝒗 by leveraging the attention mechanism. Given the query Q 𝒗 , key K 𝒂 and value V 𝒂 , the output of a cross-attention layer is first computed as:X 𝒄 = V 𝒂 softmax Q ⊤ 𝒗 K 𝒂 .We then borrow an idea from the transformer encoder layer [47] where each sub-layer is put into aEQUATIONEQUATIONwhere LN (•) indicates layer normalization, F MSA (•) is the multihead self-attention layer, and F FFN (•) represents the feed forward network. Then, we can obtain the action-aware video tokens X 𝒄+1 .Object-relation Sensitive Graph is designed to model the object relations information. Specifically, there are also two questions to be answered: 1) how to effectively capture the object relations information from videos; 2) how to leverage the object relations information to help better implicit anomalous sentiment identification in the IasDig task. Next, we will answer the two questions, formulated as follows.For question 1), we leverage RelTR 3 [10] , a well-studied onestage object-relation graph generation method, to obtain object relations information. As shown in ORG, for a given sequence of frames, RelTR is able to generate a sequence of graphs 𝐺 = {𝐺 1 , ...𝐺 𝑖 , ..., 𝐺 𝑛 }, where 𝐺 𝑖 = (𝑅 𝑖 , 𝐸 𝑖 ) represents the object-relation graph of the 𝑖-th frame, 𝑅 𝑖 = { 𝑐 𝑖,1 , 𝑏 𝑖,1 , ..., 𝑐 𝑖,𝑘 , 𝑏 𝑖,𝑘 } is a set of 𝑘 detected objects with the class 𝑐 and the corresponding bounding box 𝑏, 𝐸 𝑖 denotes a set of directed edges of the form {𝑐 𝑖,𝑝 , 𝑟 𝑖,(𝑝,𝑞) , 𝑐 𝑖,𝑞 }, assigning two directional edges from 𝑐 𝑖,𝑝 to 𝑟 𝑖,(𝑝,𝑞) and from 𝑟 𝑖,(𝑝,𝑞) to 𝑐 𝑖,𝑞 , where 𝑟 𝑖,(𝑝,𝑞) denotes a relationship categories. Take the last frame of the video in ORG as an example, one object can be expressed as (man, <0.28, 0.13, 0.62, 1.36>), and one edge can be (man, near, door).For question 2), we leverage object-aware masking together with Masked Graph Transformer Networks (MaskGTN) to fully use the object relations information. We first design object-aware masking, which masks out the unimportant object parts of the frame according to the bounding box information for each frame representation 𝒙 𝒗 𝒊 ∈ X 𝒗 . Then, we will get a sequence of masked video tokens X 𝒎 = {𝒙 𝒎 1 , ..., 𝒙 𝒎 𝒊 , ..., 𝒙 𝒎 𝒏 }. For enriching each region representation in each frame, we propose MaskGTN by following Wang et al. [48] , which aggregates the information of its local neighbors through a graph transformer layer (GT) [58] . Given an input graph 𝐺 𝑖 of region classes and edges, MaskGTN computes new vectors of each region and edge. Supposing that we use 𝐿 GTs, let 𝐻 ℓ be the feature representations of the ℓ-th layer in GTs, the forward propagation becomes:EQUATION)where 𝜎 is an activation function on the graph, Ã is the adjacency matrix of the graph 𝐺 𝑖 , i.e., a matrix from 𝐸 𝑖 , D is the degree matrix of Ã, i.e., D𝑖𝑖 = 𝑖 Ã𝑖 𝑗 and 𝑊 (ℓ ) is a trainable weight matrix. After the last propagation, a feed forward network (FFN) will be introduced to get the final object-aware video tokens X 𝒔 = FFN 𝐻 (𝐿) .After modeling the scene information, we design a Balanced Heterogeneous MoE Module (B-H MoE) for better implicit anomalous sentiment identification. In this section, we address two crucial 3 https://github.com/yrcong/RelTR questions: 1) how to obtain the routing weights of the two scene information; 2) how to balance each type of the scene information. For question 1), inspired by Mixture-of-Experts (MoE) [15, 19] , we hope to introduce two projection experts (PE), allowing them to dynamically adjust the weights of the two heterogeneous pieces of the scene information.As shown in Figure 2 , in contrast to other methods that introduce several FFNs within LLMs, B-H MoE consists of two PEs outside the LLM, which dynamically adjust the weights of information from ASG and ORG. Each PE is a stack of transformer layers, and a dynamic Modality Router R is introduced to control the contribution of each PE. The router R is structured as a straightforward MLP that receives input tokens and calculates the routing weights for each expert, i.e., a soft router [39] . Formally, for the output ℎ of ASG and ORG, the output of B-H MoE can be represented as follows:EQUATIONwhere 𝑅(ℎ) 𝑖 and 𝐸 𝑖 (ℎ) denote the corresponding weight and the output of the 𝑖-th PE, respectively. 𝑁 is the number of PEs. The router 𝑅(•) can be written as: 𝑅(•) = softmax(ℎW 𝑔 ), where W 𝑔 is the trainable weight matrix for router 𝑅(•).For question 2), inspired by LoRAMoE [11] , we design a new balancing constraint loss L 𝑏𝑐 to optimize the soft router in B-H MoE. By minimizing the L 𝑏𝑐 loss, Hawkeye can encourage the router R to dynamically adjust the contributions among all experts. Formally, we define the importance matrix M, and M 𝑖,𝑚 denotes the sum of importance value of the 𝑖-th PE in B-H MoE for the 𝑚-th training sample in a batch, which can be represented as follows:EQUATIONwhere 𝑇 𝑚 and 𝜔 𝑗 𝑖 denote the token numbers of the 𝑚-th training sample, the hidden output of the 𝑗-th token. 𝜏 is a hyper-parameter.We then define a coefficient matrix P with the same size of M, corresponding to the importance matrix M, P 𝑖,𝑚 denotes the importance coefficient of M 𝑖,𝑚 , which can be written as follows:EQUATIONwhere 𝑅 𝑎 𝑖 and 𝑅 𝑠 𝑖 are routing weights of the action information and the object information from HeteroMoE, Type(𝑚) is the type of the 𝑚-th training sample, i.e., action or object relations information.The weighted importance matrix T can be T = P • M. In order to constrain the scales of the scene information and meanwhile keep a balance between them, we introduce the Coefficient of Variation (CV) [1] , and the new loss L 𝑏𝑐 is formulated as:L 𝑏𝑐 = CV = 𝜎 2 (T) 𝜇 (T) ,where𝜇 (T) = 𝑚 𝑖=1 𝑛 𝑗 =1 𝑡 𝑖 𝑗 𝑚•𝑛 and 𝜎 2 (T) = 𝑚 𝑖=1 𝑛 𝑗 =1 (𝑡𝑖𝑗 -𝜇) 2 𝑚•𝑛represent the variance and mean of T; 𝑛, 𝑚 are the rows and columns of T; and 𝑡 𝑖 𝑗 is element in the matrix T. The variance represents the dispersion of the distribution of expert importance, while the mean provides an indication of the central tendency of the distribution. L 𝑏𝑐 provides a dimensionless measure, allowing for comparisons between different scene information.We observe that Video-LLaVA lacks the ability to comprehend the scene information, which is important for our study. To address this, we take a two-stage training process.Stage 1. Object-relation and Action Enhanced Pre-Tuning. As shown in Figure 2 , we first pre-tune Video-LLaVA using two high quality and manually annotated datasets, i.e., RefCOCO [22] and HumanML3D [14] , which aims at enhancing Video-LLaVA with the scene understanding ability. The model is asked to "Describe the image region and judge the action of the characters in the video".Stage 2. IasDig Tuning. At the second stage, our goal is to make the model able to tackle the IasDig task through instruction tuning. We construct an IasDig-tuning dataset and the pre-tuned Video-LLaVA is asked to "Analyze the following video and locate the timestamps when the individuals in the video convey implicit anomalous sentiments". The instruction will undergo Text Tokenization to obtain the textual tokens X 𝑡 . The input of the LLM will be "𝑦 from Eq.( 5) + X 𝑡 ". The overall loss of Hawkeye can be represented as L 𝑡𝑜𝑡𝑎𝑙 = L + 𝜆L 𝑏𝑐 , where L is the next-token prediction loss of LLMs, and 𝜆 controls the strength of the balancing constraint loss. 
experimental settings 4.1 instruction data construction for hawkeye: 
 The training pipeline of Hawkeye contains two stages and each stage needs an instruction dataset, detailed as follows.For Stage 1. As shown in Figure 3 , to enhance the scene understanding ability w.r.t actions and object relations of Video-LLMs, we construct a dataset based on RefCOCO and HumanML3D. Specifically, based on these datasets, we manually construct the instruction for each video, for instance: Instruction: "Describe the image region <objs> and judge the action of the characters in the video." Answer: "A man in suit wearing sunglasses. & A person raises right hand, waves.", where <objs> denotes the coordinates of the region. As HumanML3D has 25K videos with an average duration of 1 seconds, and we take 8 frames per second. For the data balance, we randomly select 200K images from RefCOCO to form the dataset.For Stage 2. As shown in Figure 3 , we construct an IasDigtuning dataset consisting of 1K videos. We take 8 frames per second, resulting in 1160K frames. This dataset consists of a real-word Scene-sparsity IasDig dataset and a real-word Scene-density IasDig dataset. Scene-sparsity (short for S-S) is based on TSL-300 [61] , which includes only daily life scenes for temporal sentiment localization (thus called scene-sparsity). To fit the IasDig scenario, we have manually filtered out videos have facial expressions and removed the audio of all videos. Each video in S-S contains 0.33 anomalous segments on average. Scene-density (short for S-D) is based on UCF-Crime [44] , a real-world surveillance video dataset containing 13 crime scenes (thus called scene-density). As IasDig mainly focuses on the anomalous sentiments, we only use the abnormal videos with negative sentiments. Each video in S-D contains 1.09 anomalous segments on average. Further, we construct the instruction for each video manually, for instance: Instruction: "Analyze the following video and locate the timestamps when the implicit anomalous sentiments are present. " Answer: "From <s1> to <s2>, there exists ..., as ... shooting at ... ", where <s1> and <s2> are the start and end time of an anomalous segment. Traditional methods cannot be directly applied to the IasDig task, while general Video-LLMs can, so we choose several of them as baselines. VideoChat [26] utilizes Q-Former to map visual features to Vicuna [9] . Video-ChatGPT [34] combines LLM with CLIP [40] . Valley [33] uses a temporal modeling module. PandaGPT [43] utilizes ImageBind [13] to demonstrate cross-modal capabilities. mPLUG-Owl [56] uses a visual abstractor to align different modes. Chat-UniVi [21] merges visual tokens with semantic meanings. Video-LLaVA [29] conducts joint training stages on images and videos. Because the above approaches are for different tasks and have different experimental settings, for a fair and thorough comparison, we re-implement these approaches using their released codes and all the LLM size is 7B in our experiments.IasDig focuses on discovering and grounding implicit anomalous sentiments, so it needs two evaluation metrics, detailed as follows.For the localization performance, we adopt the commonly used mAP@tIoU metric [61] . Under different intersection over union (IoU), the metric is calculated by mean Average Precision (mAP). We set thresholds ranging from 0.1 to 0.3, with an interval of 0.1.For the classification performance, as IasDig mainly focuses on precisely discovering the anomalous sentiments, we prefer falsenegative rates (FNRs), denoting the rates of "mistaking a frame labeled positive (i.e., 1) for negative (i.e., 0)", which is of great importance for the IasDig task. For the example in Figure 1 , we'd rather identify all timestamps as anomalous than miss a single timestamp with anomalous sentiment. If a frame is missed, it could result in a serious criminal event, so this is a typical FNRs problem. We also prefer Recall over Precision and report F2 [61] as another classification metric. The two classification metrics are denoted as:EQUATIONEQUATIONwhere # denotes the frame numbers and 𝛽 is 2. Moreover, 𝑡-test5 is used to evaluate the significance of the performance.We utilize open-source codes to obtain experimental results of all the baselines on S-S and S-D dataset. The hyper-parameters of these baselines remain the same setting reported by their public papers. The others are tuned according to the best performance. For Hawkeye, we take 8 frames per second of each video. In training stage, we use AdamW [32] as the optimizer. The initial learning rate is 2e-5, with a warmup ratio of 0.03. For stage 1 and stage 2, we fine-tune Video-LLaVA (7B) 6 using LoRA [17] , with the dimension, scaling factor, dropout rate of the LoRA matrix set to be  
contributions of each key component: 
 To delve deeper into the impact of the key components of Hawkeye, we conduct a series of ablation studies, the results of which are detailed in Table 1 ). This is reasonable and again confirms that the backbone lacks the ability to comprehend the scene information. This further demonstrates the necessity and effectiveness of Pre-tuning, and encourages us to use more high quality datasets to facilitate the scene understanding ability of Video-LLMs before tuning on IasDig.To study the practicality of Hawkeye, we compare the FNRs of Hawkeye with other Video-LLMs. From Table 1 you can see that: Hawkeye performs the best on the metric of FNRs. On S-S, it outperforms Video-LLaVA by 8.5% (p-value < 0.01), and on S-D, it outperforms mPLUG-Owl by 8.47% (p-value < 0.01). This indicates that Hawkeye is effective in reducing the rates of FNRs, which is of great importance in practical applications. Moreover, recognizing that the S-D dataset encompasses 13 distinct real-world crime scenes, we perform a detailed analysis of each crime scene on the performance of FNRs, which is shown in Table 2 7 . From this table you can see that: Hawkeye significantly outperforms all other Video-LLMs across the 13 anomalies. This further demonstrates the effectiveness of Hawkeye and its robust ability to reduce the FNRs, thereby reducing the likelihood of misclassifying anomalously sentimented samples as normal. This also underlines the necessity to consider the scene information in the IasDig task, especially in the case of Vandalism.Besides, we also study the inference time of Hawkeye and other Video-LLMs, from Table 1 , you can see that: Hawkeye does not perform much differently from the other models in terms of inference time. The inference time of Hawkeye for a one minute video is 66s on S-S and 74s on S-D. While the fastest inference time onhttps://github.com/HRNet/HigherHRNet-Human-Pose-EstimationEach token contains up to 5 people, and we average each token after graph attention. residual structure, and layer normalization[2] is performed after the residual connection. The input is X 𝒄 , and the procedure can be:Code and dataset are released at https://github.com/Zhao-Jianing-SUDA/Hawkeyehttps://docs.scipy.org/doc/scipy/reference/stats.htmlhttps://github.com/PKU-YuanGroup/Video-LLaVAAs FNRs can truly reflect the practicality of the IasDig task, and limited by the length of the article, we do not illustrate F2 and mAP@tIoU on these 13 crime scenes. Actually, Hawkeye is still the best performing approach. 
 
 Appendix:  


Decision: The reviewers have come up with the following strengths and limitations of the paper

STRENGTH
- Reasonable Pipeline with Promising Results
- Effective Visualization
- Novel Task and Dataset, and Extensive Experiments on Custom-Built Datasets
- Clear and Well-Written
- Sufficient Ablation Experiments
- Innovative Use of Implicit Scene Information
- New Research Direction with IasDig Task
- Integration of Graph-Structured Scene Modeling
- Comprehensive Evaluation

LIMITATIONS
- High Complexity and Low Efficiency
- Lack of Detailed Dataset Description
- Similarity to Existing Tasks (Video Grounding)
- Limited Comparisons, Lack of Comparison with Traditional Methods
- Insufficient Implementation Details
- Generalization and Real-World Validation
- Technical Detail Deficiency

The rebuttal did not change the rating of the reviewers. There is a high consensus among reviewers' rating. The average rating is 4.0.

Reviews: review: paper_summary: paper_summary: - The paper proposes a new Scene-enhanced Video Large Language Model named Hawkeye for the task of discovering and grounding anomalous sentiments in recon-videos (IasDig). The proposed method consists of two modules: a graph-structured scene modeling module and a balanced heterogeneous MoE module, designed to address the tasks of scene modeling and scene balancing in the IasDig task. Extensive experiments demonstrate the effectiveness of the Hawkeye method in proposed IasDig task and dataset, and the practicality for real-world applications.
strength_weakness: strength_weakness: - The overall pipeline of this paper, utilizing large language models, is reasonable and has yielded promising experimental results.
- The visualization of experiments in this paper contributes to a better understanding of the method design and experimental results.
limitations: limitations: - The method proposed in this paper, which includes introducing two feature extraction networks and implementing a two-stage process, potentially suffers from high complexity and low efficiency. Improving final performance might come with significant additional computational costs. I expect the authors to discuss this issue in the rebuttal.
- This paper does not clearly describe the IasDig dataset constructed for studying the video large language model. It lacks detailed introductions of video instances, language information, and comparative analyses with datasets used by other video language models.
paper_summary: paper_summary: The paper introduces a novel task called Implicit anomalous sentiment Discovering and grounding (IasDig), which is designed to identify and ground anomalous sentiments in surveillance and drone reconnaissance videos.  The paper proposes a new Scene-enhanced Video Large Language Model named Hawkeye.  The paper reports extensive experimental results on specially constructed datasets, demonstrating Hawkeye's superior performance over advanced VideoLLM baselines, particularly in reducing false negative rates.
strength_weakness: strength_weakness: This paper introduces a novel task and dataset.

This paper is well-written and easy to understand.

This paper has done sufficient ablation experiments to verify the proposed model.
limitations: limitations: It seems that the proposed task is similar to Video Grounding, the difference between the two tasks is only the format of the output.

Can the proposed method solve Video Grounding? Or the Video Grounding method can also solve the IasDig task? I think it is appropriate to make a comparison between the two types of methods.

Since this paper is about anomalous emotion detection, why do we need to capture actions? From this perspective, this task is very similar to Temporal Grounding of Human Motion. I don’t quite understand the difference between tasks.
paper_summary: paper_summary: This paper introduces Hawkeye, a novel system designed to discover and ground implicit anomalous sentiments in recon-videos. 
It addresses the absence of explicit sentiment cues like language, audio by leveraging implicit scene facial expressions by leveraging implicit scene information. 
The authors propose a new task, Implicit anomalous Sentiment Discovering and Grounding, and develop a Scene-enhanced Video Large Language Model (Hawkeye) to execute this task efficiently.
strength_weakness: strength_weakness: - This paper pioneers the use of implicit scene information for sentiment analysis in recon-videos, filling a gap where traditional sentiment analysis methods are insufficient due to the lack of explicit cues. 
- The work includes extensive experiments on custom-built datasets that demonstrate Hawkeye's superiority over sota video-llms in identifying and localizing implicit anomalous sentiments. 
- The Balanced Heterogeneous MoE module and scene-balancing loss function show technical sophistication in handling the complexity of scene information, enhancing model adaptability and performances.
limitations: limitations: - While the paper compares Hawkeye to several video-llms, a more comprehensive comparison with established models in related fields, such as anomaly detection or multimodal sentiment analysis, would strengthen its position. 
- The paper might benefit from more detailed implementation details, especially for the B-H MoE module and scene-balancing loss, to ensure researchers can replicate and build upon the work. 
- When constructing S-S and S-D datasets, what criteria were used to ensure their representativeness and diversity for evaluating Hawkeye's performance? 
- Are there any plans or ongoing work to validate Hawkeye's performance on real-world recon-videos?
paper_summary: paper_summary: This paper introduces a novel task—Implicit Anomalous Sentiment Discovery and Grounding (IasDig), which utilizes implicit scene information (such as actions and object relations) to identify anomalous sentiments in reconnaissance videos. The paper develops a Scene-enhanced Video Large Language Model named Hawkeye, which combines a graph-structured scene modeling module and a balanced heterogeneous MoE module to accurately discover and locate anomalous sentiments in videos. Through extensive experiments conducted on specially constructed scene-sparsity and scene-density IasDig datasets, the paper demonstrates the advantages of the Hawkeye model over advanced video LLM models in terms of accuracy and practicality.
strength_weakness: strength_weakness: 1. The paper proposes a new IasDig task, providing a new research direction for automatic recognition of implicit anomalous sentiments in reconnaissance videos.

2. By integrating graph-structured scene modeling with a balanced heterogeneous MoE module, the paper effectively addresses the modeling and balancing of scene information.

3. The authors have constructed specific datasets and comprehensively evaluated the proposed model through comparisons with multiple advanced video LLM models.

4. The structure of the article is clear, and both the methodology and experimental design are well articulated, facilitating understanding and reproducibility.
limitations: limitations: **1. Lack of Technical Detail:** Although the paper mentions graph-structured scene modeling and balanced heterogeneous MoE modules, the descriptions of specific implementation details (such as parameter settings and algorithm optimizations) are not sufficiently detailed, which may affect the transparency and reproducibility of the model.

**2. Limitations:** Although the effectiveness of the model is validated on the custom-built datasets, it has not been tested on publicly available standard datasets, which may limit the validation of the model's generalization ability.

**3. Experimental Design:** There is a lack of comparison with traditional non-LLM methods, which could help more comprehensively showcase the advantages and innovations of Hawkeye against the existing technology backdrop.