Predicting the Unseen: A Novel Dataset for Hidden Intention Localization in Pre-abnormal Analysis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 12:08:45

Abstract: Our paper introduces a novel video dataset specifically designed for Temporal Intention Localization (TIL), aimed at identifying hidden abnormal intention in densely populated and dynamically complex environments. Traditional Temporal Action Localization (TAL) frameworks, focusing on overt actions within constrained temporal intervals, often miss the subtleties of pre-abnormal actions that unfold over extended periods. Our dataset comprises 228 videos with 5790 clips, each annotated to capture fine-grained actions within ambiguous temporal boundaries using a Joint-Linear-Assignment methodology. This comprehensive approach enables detailed analysis of the evolution of abnormal intention over time. To address the detection of subtle, hidden intention, we developed the Intention-Action Fusion module, an creative approach that integrates dynamic feature fusion across 11 behavioral subcategories, significantly enhancing the model's ability to discern nuanced intention. This enhancement has led to performance improvements of up to 139\% in specific scenarios, dramatically boosting the model's sensitivity and interpretability, which is crucial for advancing the capabilities of proactive surveillance systems. By pushing the boundaries of current technology, our dataset and methodologies foster the development of proactive surveillance systems capable of preemptively identifying potential threats from nuanced behavioral patterns, encouraging further exploration into the complexities of intention beyond observable actions.

Main paper: 
 introduction: 
 Temporal Action Localization (TAL) focuses on analyzing untrimmed long videos to identify and categorize periods of key action, distinguishing them from non-action backgrounds by marking the start and end times of these actions. Common datasets such as THU-MOS14 [11] , ActivityNet [7] and Kinetics [13] have been important in advancing this field, leading to the development of sophisticated models such as Actionformer [35] , TriDet [20] [19], and Video-Mamba [3] . While these models excel in recognizing actions, they often struggle to interpret the hidden intention of these actions. This challenge becomes more apparent in situations requiring a nuanced understanding of the participants' motives. Based on the above challenges, our research pioneers the concept of Temporal Intention Localization (TIL), aimed to identify and locate segments within untrimmed videos that potentially hidden intention, focusing on forecasting from action clues rather than merely recognizing ongoing actions.Unlike traditional temporal action localization tasks that locate actions as they occur based on singular behavior, TIL anticipates future actions by analyzing intention combined with sequences of related behavior, thus requiring a higher degree of interpretability. The evolution of TIL underscores not only the advancements in action localization but also highlights the ongoing need to bridge the gap between recognizing actions and understanding intention. As shown in figure 1 , in security-related contexts, our research tries to discover hidden abnormal intention to make intervention measures more precise and effective, facilitating pre-abnormal analysis.The limitations of traditional coarse-grained datasets are particularly pronounced when addressing the nuanced detection of hidden abnormal intention. For instance, behavior preceding theft, like looming, may seem harmless in everyday life but can signify premeditated abnormal intention in specific contexts. Relying solely on singular, overt actions for judgment falls short in accurately detecting hidden intention. This shortfall underscores the inadequacy of existing coarse-grained labels, which lack the nuance and specificity needed to discern the subtle cues of premeditated abnormal actions. Hence, the challenge lies in understanding the context and the myriad of small, seemingly innocuous behavior that, when pieced together, reveal a larger, more concerning picture of potential abnormal behavior like figure 2 .Building upon the identified gaps in understanding the hidden intention behind actions in abnormal scenarios, our research introduces the Hidden Abnormal Intention (HAI) Dataset. Our dataset is designed not just to capture actions but to delve into the hidden intention that precede abnormal behavior. Unlike datasets that prioritize overt action localization, HAI maps a series of multiperspective, multi-level behavior to their underlying abnormal intention, creating a connected narrative that highlights the causal relationship and cumulative effect between actions enhancing interpretability. Meanwhile, we value the diversity of annotator perspectives and align them with expert knowledge for a comprehensive understanding. This progression entails equipping these models with the ability to differentiate between normal behavior and those imbued with covert motives, thereby furnishing a more refined methodology for elucidating the hidden intention of humans prior to committing the abnormal.In our paper, we undertake a detailed study to uncover hidden abnormal intention by carefully creating a dataset and refining models. Leveraging the expertise of professionals, we have constructed a fine-grained, multi-level labeling system to explore the feature of hidden abnormal behavior. Our dataset comprises 228 video, carefully annotated to reflect the gradation of abnormal intention across different scenarios, utilizing a method known as Joint-Linear-Assignment. This approach links annotated segments end-to-end before applying a linear scoring of suspicion levels to behavior, highlighting the increased suspiciousness of repetitive actions. This layered approach to labeling facilitates the development of a nuanced understanding of pre-abnormal actions.Our detailed annotation framework directs our model improvement efforts towards locating 11 specific subcategories of behavior with abnormal intention, instead of directly identifying hidden intention segments. By aggregating predictive results for these subcategories using a weighted combination, our model finely discerns the complex web of intention preceding abnormalities. This dual approach enhances action localization precision and provides novel insights into preemptive abnormal behavior identification, significantly advancing behavioral analysis and security measures.In our study, we advance temporal intention localization in preabnormal analysis with 3 key innovations:• Beneficial Dataset. We introduce a specialized dataset, named HAI, aimed at understanding actions and their underlying intention related to the abnormal, enriched with expert annotations for comprehensive behavioral insights. • Specific Annotation. We employ a Joint-Linear-Assignment method for nuanced abnormal intention analysis across video segments, enhancing behavioral understanding through a focus on repetitive actions. • Solid Method. Our approach refines predictive models by focusing on pre-abnormal actions, improving accuracy in locating hidden intention for effective security applications. 
related work: 
 Temporal Video Localization Datasets. Video datasets are crucial for advancing action recognition and temporal localization.To enhance model generalizability, many datasets include a wide range of everyday action categories, tracing back to early datasets like KTH [17] and Weizmann [5] [1] . With technological advancements, more challenging datasets have been introduced, such as UCF101 [22] , Kinetics [13] , ActivityNet [7] , and FineAction [15] . These datasets expand action variety, enhance scene diversity, and include annotations for temporal locations and spatiotemporal bounding boxes. Rising demands for temporal localization in specific domains and unique action categories have led to the emergence of domain-specific video datasets. For example, the UCF igure 3: Visualization of the similarities and distinctions among fine-grained labels under "Suspicious Gaze". All three fine-grained labels prioritize suspicious pre-abnormal gaze, yet diverge in range of gaze and duration.Sports [16] [23] and Sports-1M datasets [12] , by focusing on complex sports actions, provide richer and harder research material. In more detailed action recognition (AR) research, the FineGym [18] dataset focuses on gymnastics, and the EPIC-KITCHENS [4] dataset focuses on actions and object interactions in kitchen environments, contributing to domain-specific research. Despite efforts to improve annotation content and accuracy, most domain-specific datasets still use traditional methods for annotation. These datasets typically employ broadly related labels for annotating target actions (e.g. True/False), lacking in-depth annotations tailored to specific tasks within a given domain. In contrast, our Hidden Abnormal Intention Dataset (HAI) creatively adopts a more targeted and domain-task-aligned annotation approach: (1) HAI identifies underlying intention through sequences of causally linked actions, rather than directly annotating target behavior; (2) To capture hidden intention, we employ associative labels that cumulatively contribute to illustrating and understanding these covert objectives. (3) HAI incorporates diverse annotator perspectives to counter the subjectivity in interpreting hidden intention, enhancing accuracy through algorithmic and expert validation. (4) HAI employs a unique hierarchical labeling system that breaks down abnormal intention into 11 specific behavior analyzed from four angles, enabling precise assessment of intention intensity.Methods for Temporal Action Localization. In Temporal Action Localization (TAL), deep learning has driven significant advancements. Techniques range from anchor-based methods like SSN [37] to actionness-guided approaches like UTS [28] for pinpointing actions. Additionally, graph-based models like G-TAD [34] and Transformer-based systems such as Actionformer [35] have further refined the understanding of complicated video relationships. Alongside, two-stage methods and end-to-end learning, including I3D [2] [8], offer robust frameworks for action proposal and feature extraction. Weakly supervised learning and 3D Convolutional Networks, like C3D [25] , also contribute by minimizing annotation needs and capturing dynamic spatiotemporal patterns, marking a comprehensive evolution in TAL.Hidden Intention Detection. In traditional intention detection, "intention" is equated with undisguised behavior, which has led to the development of numerous intention detection algorithms. E3D-LSTM [30] predicts human intention by learning explicit spatiotemporal visual representations, while EM-base [32] and HAO [31] infer human intention by combining visual representations with attention. Automatic planning techniques, generating plans from sequences of past actions to predict possible future actions for intention identification, have been successfully applied in human intention recognition, such as the RACNN algorithm [29] . Not limited to this, HFD [33] introduces non-verbal cues as signals for intention recognition. However, these works have not explored the subtle, concealed intention in human activities, where the actor hides certain behavior-namely hidden intention. It wasn't until the introduction of the Hidden Intention Discovery (HID) task [38] that a deeper exploration into the recognition of hidden intention was highlighted. Our HAI dataset provides more fine-grained, systematic annotation materials for the recognition of hidden intention, supporting the development of more precise algorithm research, improving existing algorithms, and introducing temporal localization methods into hidden intention recognition, thus achieving a more accurate quantification and localization of hidden intention. 
key definitions and concepts: 
 Our research focuses on the complex concept of hidden abnormal intention and temporal intention localization.Hidden Abnormal Intention is defined as a series of preparatory or exploratory actions exhibited by individuals before committing abnormal actions. These actions, like subtle and indirect body language or environmental interactions, are designed to conceal true intention. For example, a potential thief may pretend to browse goods while looking for opportunities to steal. By enabling security personnel to take preventative measures beforehand, identifying these hidden intention is crucial for preventing abnormal actions.Temporal Intention Localization specifically targets untrimmed video to locate and identify segments indicative of abnormal intention. Given an untrimmed video, denoted as 𝑉 , the objective is to identify and delineate a set of segments, represented as {𝑠 1 , 𝑠 2 , ..., 𝑠 𝑛 }. Each identified segment 𝑠 𝑖 is defined by a tuple (𝑡 𝑠𝑡𝑎𝑟𝑡 , 𝑡 𝑒𝑛𝑑 , 𝑙), where 𝑡 𝑠𝑡𝑎𝑟𝑡 and 𝑡 𝑒𝑛𝑑 indicate the segment's commencement and conclusion times within the video 𝑉 , respectively. Additionally, 𝑙 denotes the segment's hidden abnormal intention level, categorized into three distinct levels: 1 for low, 2 for moderate, and 3 for high, each reflecting the inferred degree of hidden abnormal intention. This task aims for precise localization and accurate intention assessment, enhancing the analysis of video data for security and surveillance purposes. 
hai dataset: 
 Abnormal actions such as theft pose a global threat to societal security and personal property, yet tracking and investigating these acts is challenging. Effective prevention and timely identification of abnormal intention are crucial to reducing crime. This dataset is developed to assist researchers and technology developers in creating systems that can recognize and locate abnormal intention, particularly during the crucial hidden phase before an incident. By leveraging data-driven insights, this initiative aims to enhance security measures and reduce risks associated with abnormal actions, thus promoting safer communities and protecting personal assets.To capture the complexity and subtlety of HAI, we combined expert knowledge with public insight, collaborating with frontline police officers and security experts. This collaboration not only enriched our dataset with practical accuracy and value but also enhanced its broad applicability by integrating public understanding. We established a hierarchical labeling system with multi-grained labels.Coarse-grained labels identify broader behavioral categories preceding abnormal actions, such as "Suspicious Gaze, " which depict general patterns during the preparatory phase. Fine-grained labels offer detailed insights, specifying actions indicative of hidden intention like "Surveying the Area" or "Staring for an Extended Period". This approach helps illuminate the complicated behavior suspects use to conceal their intention, as shown in figure 3 .The labels logically arrange the abnormal process, enhancing our understanding and detection of HAI. For example, under the coarse category of "Suspicious Gaze", we might find fine-grained labels like "Quick Glance to the Side" or "Surveying the Area" which provide insights into the specific actions a suspect might take to assess their surroundings before committing. This annotation system aims to cover the entire process from preparation to attempted actions comprehensively, revealing the complexity of abnormal intention.In the initial of the HAI dataset, we have carefully curated and amalgamated 11 fine-grained behavior categories from a broad spectrum of hidden abnormal intention. These categories include: quick glance to the side, surveying the area, staring for an extended period, bending over to touch, concealing an object, taking out a tool, mask action, looming, deliberately slowing down, tools, and mask. These fine-grained categories are organized under four coarse-grained categories that represent different aspects of suspicious behavior: suspicious gaze, suspicious posture, suspicious behavior, and suspicious objects. This organizational framework forms the semantic backbone of our dataset, as illustrated in figure 4 .Our dataset development pipeline is a structured process that spans from the initial video collection to the final integration of annotated data. The subsequent figure 5 provides a high-level overview of the process, highlighting the systematic approach taken at each stage to ensure the creation of a robust dataset for Hidden Abnormal Intention (HAI) analysis. In the following subsections, we will delve into the details of the dataset collection and the annotation steps that form the backbone of our methodology.Collection. We sourced our videos from publicly datasets like UCF-Crime [24] and videos collected from the internet. Our dataset cover a variety of scenarios, including indoor and outdoor abnormal actions, abnormal actions involving vehicles, both individual and collaborative abnormal actions, and incidents occurring at different times throughout the day. This diverse collection is crucial because the abnormal doesn't happen in just one kind of place or time but occurring under numerous conditions.To create an inclusive and representative dataset, we included videos with varying resolutions and frame rates, ranging from lowresolution videos common in less economically developed regions to high-definition footage from more prosperous areas. This approach ensures our dataset reflects the universal nature of abnormal actions, thereby enriching our insights.Upon assembling the videos, we strictly excluded any footage that did not accurately capture acts of abnormal behavior or attempted incidents. This selective approach ensures the integrity and relevance of our dataset, focusing exclusively on behavior that merits further analysis.The entire annotation process, emphasizing multiple checks and expert feedback, ensures the dataset's high quality and utility. We believe this meticulous and strict annotation work provides researchers and practitioners with a powerful tool for better understanding abnormal behavior.Annotation Preparation. To guide our annotation team, we designed a comprehensive guideline, ensuring everyone clearly understands the annotation process, the specific meanings and scenarios for each label. This guideline covers everything from video selection, importing information, to examples of each label, aiming for a standardized execution as much as possible.Preliminary Annotation. In this phase, annotators apply predefined labels to the video content by identifying key behavior and accurately marking relevant time segments. Each video is thoroughly annotated by at least three individuals to ensure comprehensive label application and understanding.Annotation Review and Integration. After the initial annotation, a dedicated team reviews the work to ensure accuracy and consistency. Once all annotations and reviews are complete, we compile all the information and start the integration process. We take both the union and intersection of the data annotations, preserving clips that any annotator considered suspicious (forming annotation L) and those all annotators agreed were suspicious (forming annotation S). This method allows us to simulate different levels of sensitivity to hidden abnormal intention, ensuring our dataset offers multiple angles on observing hidden intention.Joint-Linear-Assignment for pre-abnormal intention analysis. In developing the annotations of hidden abnormal intention, we initiated by assigning a suspicion score to each category label per frame, adhering to a linear scale to indicate increased suspicion with repeated behavior. We then aggregated the suspicion scores from all categories within a single frame, summing these values and normalizing, to accurately gauge the intensity of hidden abnormal intention for each frame. Frames with scores ranging from 0 to 0.3 are categorized as having an abnormal degree of uncertain. The true intention is not fully determined but observation should be remained. Those between 0.3 and 0.6 as suspicious, there are signs or evidence to arouse initial suspicion so increased surveillance may be conducted. Scores exceeding 0.6 are designated as alert, evidence is sufficient to indicate a high risk of abnormal so we need issue an alert or take action to prevent abnormal. This multi-level framework of hidden abnormal intention annotations provides a better understanding of the severity and potential risk associated with each observed incident.In addressing the complex task of identifying hidden abnormal intention, our project faces nuanced challenges that distinguish it from standard behavior analysis tasks. These challenges highlight the complexity of our work and emphasize the need for a dataset specifically designed to capture the subtleties of abnormal behavior. Below, we delve into these unique challenges. Integrated Causality and Cumulative Labeling. By annotating behavior causally related to abnormal intention and recognizing label interconnections, our dataset highlights the cumulative effect of actions, including repeated behavior and simultaneous actions. This approach reveals how behavior accumulate to manifest HAI, providing deeper analysis than traditional datasets. Multi-Level Intention Localization and Granular Decomposition. Our dataset offers an unprecedented level of detail, classifying hidden abnormal intention across multiple levels and dissecting these into 11 distinct behavior from four different viewpoints. This not only allows for a nuanced understanding that surpasses the simple binary categorizations found in traditional datasets but also provides an complex mapping of the gradual buildup of covert actions. Such granularity is crucial for accurately interpreting the complex interplay of micro-behavior that collectively signal abnormal intention, ensuring a comprehensive analysis of HAI.Inter-Annotator Variability with Expert Comparison. We acknowledge and preserve the diversity in annotators' judgments, providing a richer, context-dependent interpretation of behavior. By comparing these diverse perspectives with expert knowledge, our dataset permits a more refined assessment of the most appropriate understanding of HAI.Behavior-Intention Association. Distinguishing between simple, overt actions and hidden abnormal intention is essential. As Shown in figure 6 , HAI involves interpreting micro-behavior and intention, a complex task not commonly addressed in conventional behavior studies. These challenges highlight the complexity and necessity of developing a dataset tailored to HAI, aiming to advance our understanding and detection capabilities in the area.Our HAI dataset Precisely annotates 228 videos, yielding 5 annotation files with a total of 5,790 clips spanning 8 hours of video from global datasets, the internet, and surveillance footage. It includes segments with 28 to 320 instances across 11 categories, and temporal durations from 8 to 2,223 seconds. Each video averages 122.65 seconds, includes 3.61 categories, and contains about 6 instances per clip, with instances averaging 7.4 seconds and ranging from 0.1 to 312 seconds. The dataset features diverse global locations and resolutions, authentically representing abnormal incidents. Figure 7 provides foundational data, and the heatmap calculates the Jaccard [9] similarity between label pairs in the HAI, showing more frequent label combinations in darker cells. We used the Apriori algorithm to identify common action combinations, with Table 2 listing the top three by frequency and their support values, indicating the likelihood of these combinations occurring. This analysis reveals potential patterns in hidden abnormal intention, enhancing our understanding of behavior subtly indicating abnormal intention and focusing on high-frequency action combinations.Our annotation framework employs a three-tier hierarchical structure with cumulative effect tagging to decode intention behind abnormal activities. In video annotations, each frame can show up to 7 behavior instances, with a cap of 17 recurrences of the same behavior in a single video. Table 1 provides statistical data, comparing our dataset with other related video datasets. 
empirical studies: 
 In the empirical study, we introduce and evaluate our novel Intention-Action Fusion (IAF) module, alongside the pioneering HAI dataset, within the realm of temporal intention localization. Through a series of rigorous experiments, including SOTA baseline comparisons and a specific ablation study, we aim to underscore the effectiveness of our methodological advancements and the intrinsic value of our dataset. This comprehensive examination not only showcases the precision and interpretability of our model in temporal intention localization but also provides a clear demonstration of how fine-grained and causality labels and dynamic feature fusion significantly contribute to the understanding of hidden intention. The empirical evidence gathered from these studies serves to highlight the crucial role of our dataset and the IAF module in advancing the field of intention analysis.We present an creative Intention-Action Fusion (IAF) Module based on underpinned by the insight that the detection of surreptitious abnormal behavior transcends the mere analysis of actions or the prediction of intention in isolation. It receives feature information from the temporal localization model on 11 types of behavior indicative of hidden abnormal intention. It extracts temporal information about key actions from the videos and then performs dynamic feature fusion on these key behavior based on predefined prior knowledge. Specifically, for each segment of video features, a weighted sum 𝑆 𝑏,𝑡 = 11 𝑘=1 𝑊 𝑘 • 𝑇 𝑏,𝑘,𝑡 is computed for the first 11 types of behavior, where 𝑊 𝑘 represents the weight for the 𝑘-th behavior and 𝑇 𝑏,𝑘,𝑡 represents the feature value of the 𝑘-th behavior at time 𝑡 for the 𝑏-th video. This weighted sum is then dynamically added to the features of the intention categories of each segment across all time points, modifying the original feature tensor to 𝑇 ′ 𝑏,𝑙,𝑡 = 𝑇 𝑏,𝑙,𝑡 + 𝑆 𝑏,1,𝑡 for the intention level 𝑙, applicable across all time points 𝑡. The fused information is subsequently incorporated into the feature set for hidden abnormal intention, allowing the model to flexibly optimize intention predictions tailored to the specific circumstances of different video contents. This methodology ensures that the model's predictions remain highly accurate and strongly interpretable across diverse scenarios.In this section, we initially evaluate our HAI dataset by benchmarking it against representative methods in the task of temporal action localization, demonstrating its efficacy. Additionally, we conduct an illustrative study using the Intention-Action Fusion module for temporal localization to further establish the dataset's utility. Our focus centers on the application of fine-grained labels for in-depth In our study, we aim to detect and locate hidden abnormal intention, thus enabling the proactive identification of abnormal incidents within video. To achieve this, we conduct a detailed evaluation of leading temporal recognition approaches at various levels of granularity, leveraging two SOTA : TriDet [20] [19], ActionFormer [35] . We conducted experiments on a dataset with five distinct sets of annotations. These sets include annotation 1, annotation 2, annotation 3, annotation S, annotation L. Annotation 1 was contributed by the least experienced annotator, annotation 2 by the most experienced, and annotation 3 by an annotator with moderate experience. Additionally, we utilized the annotation integration method to obtain strict annotation (annotation S) and lenient annotations (annotation L). The performance of the models on these annotations gives us insight into how experience in annotation may influence the outcome of machine learning models. The dataset is divided into training and testing sets with a 9:1 ratio. We utilize VideoMaeV2 [26] [27] to extract video features.In our analysis, we measure the accuracy of located action intervals using Intersection over Union (IoU), a key metric that quantifies the overlap between the predicted action 3 with a more moderate level of experience that aligns with the general public, Avg mAP has improved by 91.21%.In concert with the findings from the Actionformer [35] model, the TriDet [20] [19] with the IAF module showcases a span of Avg mAP values ranging from 1.44% to an impressive 5.15%, culminating in a dataset Avg mAP of 3.16%. The incorporation of the IAF module has led to a substantial enhancement in model performance, with Avg mAP improvements oscillating between 15% and a remarkable 139% across the five annotations. Mirroring the trends observed with the Actionformer [35] , the deployment of the IAF module within the TriDet [20] [19] model notably excelled in annotation 3, underlining the module's adeptness in optimizing localization in scenarios with a balanced complexity that mirrors the collective cognitive baseline of the general populace.The use of the IAF module in various annotations has demonstrated how deep learning models can effectively reflect common human experiences through data annotation. Particularly in annotations that combine expert knowledge with everyday experiences, the IAF module effectively refines the model, improving its ability to recognize complex intention. This discovery highlights the value of creating data annotations that connect with a wide range of human experiences. By enhancing the model's performance on annotations that incorporate professional expertise and everyday details, the IAF module makes significant progress in narrowing the divide between artificial intelligence and the complex world of human actions and intention.To further elucidate the vital contribution of our proposed Intention-Action Fusion (IAF) module to the model's performance, we designed a specific ablation study. This process involved using the TriDet [20] [19]model as our baseline and integrating the IAF module with an essential modification: during the model's actual learning phase, the features used and optimized did not pass through the IAF module. However, the model still maintained its focus on learning pre-abnormal actions, effectively simulating the scenario where the IAF module's advanced feature processing capabilities were bypassed. As vividly illustrated in figure 8b , the outcomes of this study were telling. There was a noticeable decrement in performance post-ablation, a phenomenon that was particularly pronounced in the dataset's Avg mAP metric. This stark contrast underscores the indispensable role of the Intention-Action Fusion module in amplifying the model's acumen for locating hidden intention. It is a step forward in our endeavor to refine and push the boundaries of hidden intention localization methodologies, aiming to achieve unprecedented levels of accuracy and efficiency in understanding and interpreting complex, nuanced intention embedded within actions. 
conclusion: 
 In this paper, we propose a new task-temporal intention localization, which involves the chronological positioning of individuals' intention, and introduce an novel dataset named the Hidden Abnormal Intention Dataset (HAI). Unlike other existing datasets, the HAI creatively employs a marking scheme more tailored to temporal intention localization, utilizing associative labels with a cumulative effect, diversity in annotation, and hierarchical labeling. This provides a solid foundation for algorithmic research in temporal intention localization. Building upon HAI, we have also developed a unique algorithm to address the challenges of temporal intention localization, and the results highlight the significant challenges posed by our dataset for this task. These challenges underscore the difficulty and importance of recognizing hidden abnormal intention in natural settings. We hope our efforts will pave the way for new advancements in the field of temporal intention localization, thereby contributing to the assurance of public safety and societal well-being. 
: 
 Traditional datasets solely concentrate on actions that are visibly occurring, but our HAI dataset analyzes the same video sample according to the structure from 4 coarse-grained dimensions as "Suspicious Gaze", "Suspicious Posture", "Suspicious Object" and "Suspicious Behavior" and 11 fine-grained perspectives to provide comprehensive guidance on abnormal intention and abnormal behavior. (b) and (d) show the comparison between traditional Temporal Action Localization (TAL) task and our Temporal Intention Localization (TIL) task. TAL predicts actions from actions. Models are provided with video input to locate the start and end times of actions by predicting action feature scores. However, our TIL predicts abnormal intention and rates intention degrees from pre-abnormal actions with analysing their cumulative effect. TIL does not directly locate isolated visible actions, instead, it focuses on locating a series of associated behavior related to abnormal intention, predicting abnormal degree (uncertain, suspicious, alert) to facilitate the judgment and prediction of abnormal behavior.Our paper introduces a novel video dataset specifically designed for Temporal Intention Localization (TIL), aimed at identifying hidden abnormal intention in densely populated and dynamically complex environments. Traditional Temporal Action Localization (TAL) frameworks, focusing on overt actions within constrained temporal intervals, often miss the subtleties of pre-abnormal actions that unfold over extended periods. Our dataset comprises 228 videos with 5790 clips, each annotated to capture fine-grained actions within ambiguous temporal boundaries using a Joint-Linear-Assignment methodology. This comprehensive approach enables detailed analysis of the evolution of abnormal intention over time.To address the detection of subtle, hidden intention, we developed the Intention-Action Fusion module, an creative approach that integrates dynamic feature fusion across 11 behavioral subcategories, significantly enhancing the model's ability to discern nuanced intention. This enhancement has led to performance improvements of up to 139% in specific scenarios, dramatically boosting the model's sensitivity and interpretability, which is crucial for advancing the capabilities of proactive surveillance systems. By pushing the boundaries of current technology, our dataset and methodologies foster the development of proactive surveillance systems capable of preemptively identifying potential threats from nuanced behavioral patterns, encouraging further exploration into the complexities of intention beyond observable actions.Multimedia, Hidden Intention, Pre-abnormal, Temporal Localization 
 
 Appendix:  


Decision: The paper introduces a new task called Temporal Intention Localization, along with a corresponding dataset named Hidden Abnormal Intention (HAI) that includes detailed action categories. Additionally, it presents an Intention-Action Fusion module designed to improve the detection of subtle intentions by dynamically fusing features across multiple behavior subcategories. The paper received positive scores from 3 out of 4 reviewers. While some negative comments focused on the annotation accuracy due to unclear annotation procedures, the overall contribution of the proposed task and dataset surpass the acceptance threshold.

Clarity: The paper is well-written and is easy to read and follow. 

Originality: The novelty of this work is sufficient for acceptance.

Significance： The proposed task can motivate research of Interpretable abnormal detection, and has the potential to inspire other video understanding tasks. The constructed dataset is a contrbituion to the community.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel video dataset designed for Temporal Intention Localization (TIL) aimed at identifying hidden abnormal intentions in densely populated and dynamically complex environments. Traditional Temporal Action Localization (TAL) frameworks focus on overt actions within constrained temporal intervals and often miss the subtleties of pre-abnormal actions unfolding over extended periods. The dataset comprises 228 videos with 5790 clips, each annotated using a Joint-Linear-Assignment methodology to capture fine-grained actions within ambiguous temporal boundaries. By integrating dynamic feature fusion across 11 behavioral subcategories, the proposed method significantly enhances the model's ability to discern nuanced intentions, thereby advancing proactive surveillance systems.
strength_weakness: strength_weakness: Innovative Dataset Design: The paper introduces the Hidden Abnormal Intention (HAI) dataset, which not only captures actions but also delves into the underlying abnormal intentions. The dataset includes rich fine-grained annotations and multi-dimensional behavior labels, providing a comprehensive framework for behavior analysis.

New Temporal Intention Localization Task: The introduction of the TIL task aims to predict abnormal intentions by analyzing sequences of related behaviors rather than just recognizing ongoing actions. This extends the research scope beyond traditional TAL tasks and improves proactive analysis capabilities in security systems.

Feature Fusion Method: The development of the Intention-Action Fusion (IAF) module, which dynamically integrates features from 11 behavioral subcategories, significantly enhances the sensitivity and interpretability of the model in identifying hidden intentions.
limitations: limitations: In section 5.1, even if this paper describes a dataset, the corresponding model figure should be shown, especially for models proposed for a new dataset.

For the experimental results in Table 3, even though the model proposed in this article effectively enhances the performance of each method on the new dataset, such a low average performance raises doubts about its practical application significance.
paper_summary: paper_summary: The paper introduces the Hidden Abnormal Intention (HAI) dataset, designed to identify hidden abnormal intentions in complex environments. The dataset emphasizes pre-abnormal actions and their cumulative effects, leading to abnormal behavior. The authors propose the Temporal Intention Localization task, which focuses on predicting abnormal intentions rather than visible actions. The paper also presents the Intention-Action Fusion module, enhancing the model's ability to detect nuanced intentions through dynamic feature fusion across multiple behavioral subcategories.
strength_weakness: strength_weakness: 1. The paper introduces the Hidden Abnormal Intention (HAI) dataset, designed to identify hidden abnormal intentions in complex environments. The dataset is meticulously annotated, capturing fine-grained actions and cumulative effects, offering a detailed understanding of abnormal behavior.
2. The HAI dataset fills a significant gap by focusing on hidden abnormal intentions, providing a rich resource for future research.
3. Experimental validation and benchmarking against state-of-the-art models showcase significant performance improvements.
limitations: limitations: 1. Some paragraphs are poorly structured, with single-sentence paragraphs that disrupt the flow of the manuscript, such as lines 616.
2. Table 1 should explicitly compare the dataset size (e.g., number of videos) with similar datasets to provide a fuller comparison.
3. More references to related research on intention prediction, such as pedestrian intention, should be included to provide context and background for the work.
[2023] Pit: Progressive interaction transformer for pedestrian crossing intention prediction
[2023] Faster pedestrian crossing intention prediction based on efficient fusion of diverse intention influencing factors
4. The paper should clarify if the dataset will be publicly accessible upon acceptance, and provide more details on how it will be shared (e.g., via Google Drive or a project homepage) and any access restrictions.
5. The paper should address potential ethical risks, particularly regarding the inclusion of abnormal behaviors and personal biometric information in the videos, and provide more details on how these issues are handled.
6. The authors should include more examples of the dataset in the supplementary materials to give readers a fuller understanding of the data.
If the authors address my concerns, I would be happy to rejust my score.
paper_summary: paper_summary: In this paper, the authors collect a new video dataset annotated with fine-grained temporal labels regarding intention prediction. They proposed a model based on a behavior category fusion module to detect subtle intentions.
strength_weakness: strength_weakness: The authors provide a clear description of the motivation behind the Hidden Abnormal Intention （HAI） problem and have conducted extensive temporal and class annotations. 

The focus of this dataset differs significantly from existing mainstream datasets on action detection and video content understanding, which serves as a potential basis for many research topic.
limitations: limitations: Although the authors annotate the data with fine-grained temporal information, multi-level categories, and continuous abnormal intention values, the annotations are not sufficient in terms of the number of videos reflecting the overall scene distribution, the number of clips reflecting local intention trends, and the duration of instances. This can lead to models learning biases in the distribution (such as imbalances in fine-grained category labels). Particularly regarding the duration of instances, in the intention prediction task proposed by the authors, richer contextual information has a more comprehensive indicative capability for intentions.
paper_summary: paper_summary: This paper introduces a dataset for the Temporal Intention Localization (TIL) task, designed to identify hidden abnormal intentions in videos of complex environments. Compared to the traditional Temporal Action Localization (TAL) task, TIL can proactively identify potential threats. Additionally, this paper proposes the Intention-Action Fusion module, which can be integrated into existing TAL methods to enhance TIL performance. Notably, the author suggests that if the dataset could be open-sourced, it would have significant implications for the further development of TIL tasks.
strength_weakness: strength_weakness: 1. This paper is easy to follow, the motivation and formulation of the paper are sound
2. The HAI dataset, annotated using Joint-Linear-Assignment, provides multi-level classification of intentions, aiding in the proactive identification of potential threats
3. Extensive ablation experiments have demonstrated the effectiveness of the IAF module
limitations: limitations: 1. In the Dataset Collection section, the authors describe, "We sourced our videos from publicly datasets like UCF-Crime [24] and videos collected from the internet." However, further description of the data from the internet is needed, along with consideration of user privacy in the videos
2. The lack of detailed description regarding data annotation, such as the mention of generating 5 annotation files, including 2 from post-processing, raises questions. Does this imply that annotation was conducted by three groups of annotators with different expertise distributions? Within each group, were multiple annotations made for the same video? If only one person annotated, does it introduce the possibility of individual bias
3. Reviewing annotated data is equally important for assessing the quality of the dataset. The paper lacks information on the proportion of relevant reviews and the pass rates of annotated data through review. Additionally, are multiple rounds of annotation and review conducted during the annotation process? If multiple rounds of annotation exist, when is the annotation considered complete
4. Although the effectiveness of the IAF module has been demonstrated on ActionFormer and TriDet, its motivation is unclear