SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation Metric In the classification task, we measure the performance of the network by calculating the accuracy of the output corresponding to the labels. However, misjudgment of violent events can have severe consequences and cause irreparable harm. Violent scenarios occur less frequently than non-violent ones, implying a substantial class imbalance in real-world situations. Relying solely on accuracy as an evaluation metric may not adequately reflect the model's ability to predict violent events accurately. To address this issue, we have employed the Precision-Recall (PR) curve [54]. The PR curve allows us to examine the model's predictive capabilities across varying judgment thresholds.
Results and Analysis Table 2 presents a detailed comparison of the performance of several widely used action recognition models on the Bullying10K dataset. It includes the backbone architectures of each model and their respective operational configurations. We conducted experiments using frame intervals of 0, 2, and 4, with three different temporal step lengths of 4, 10, and 16. The results demonstrate that increasing the frame interval and temporal step length contributes to improved precision of the models. However, even models that exhibit strong performance in traditional visual classification tasks did not yield satisfactory results on the Bullying10K dataset, indicating the dataset's unique challenges. Furthermore, performed a visual analysis of the corresponding PR curve of the X3D model on the Bullying10K dataset. The PR curve reveals significant variations in performance across different action categories, underscoring the dataset's complexity and the need for robust recognition models.
We implemented different privacy protection methods on the corresponding RGB frames of the Bullying10k dataset. We observed that after employing privacy protection with RGB data, there is a decline in performance. However, DVS data demonstrated slightly superior results compared to RGB data. DVS inherently offers advantages such as resilience to significant illumination changes, motion blur resistance, and reduced data redundancy. Therefore, for tasks like action recognition that rely on motion characteristics, the features captured by DVS are particularly beneficial.
4.2 Temporal action localization
Experimental Setting In surveillance videos, violent scenarios often occur irregularly and sporadically. Due to the few background disturbances and noise in event data, we can naturally concatenate multiple samples to form extended sequences. In our task, we input a video frame sequence that encompasses segments from different action categories randomly extracted from the dataset and
Table 4: Performance of different temporal action localization models on Bullying10k.
Model	Feature	AUC	AR@1	AR@5	AR@10	AR@100
BSN [67]	TSN [63]	75.9	31.1	70.5	75.8	77.6
BSN	TSM [64]	75.5	31.8	71.2	75.8	77.2
BMN [68]	TSN	80.3	34.6	74.3	81.0	82.0
BMN	TSM	81.0	36.0	75.3	82.4	82.7
Table 5: Performance of different pose recognition models on Bullying10k.
Model	SimpleBaseline [69] (ResNet-50)	SimpleBaseline [69] (ResNet-101)	HRNet [70] (HRNet-ws32)	HRNet [70] (HRNet-ws48)	SimpleBaseline [69] (Spiking ResNet-50)
AP	62.6	63.6	62.7	62.8	54.1
\(AP^{50}\)	88.3	88.2	88.2	87.8	84.6
\(AP^{75}\)	67.4	67.7	67.5	67.2	57.6
\(AP^{M}\)	58.3	59.3	58.6	59.5	49.3
\(AP^{L}\)	73.4	74.6	73.9	74.9	64.9
AR	65.9	67.0	66.3	66.7	58.5
integrate them into longer video sequences. And aim to predict the action label along with its corresponding start and end times. The ground truth annotations are constructed based on the respective time intervals. For training on the Bullying10K dataset, we have chosen commonly used temporal action localization models. To achieve more accurate testing results, we provided action recognition data for the pre-trained features at the same ratio. Additionally, we trained the model for the localization task at a 1:3 ratio. Initially, we pre-trained the TSN [63] and TSM [64] models on the action recognition task, which serve as the feature extractors for the localization models. We extract features from the dataset and store them for subsequent analysis and processing.
Evaluation Metric Average recall [65] and Area Under the Curve (AUC) [66] are commonly used metrics to evaluate action localization. Concurrently, we employ the AR@N indicator for assessment, representing the recall rate under the condition of N proposals. This study considers N to be 1, 5, 10, and 100. Additionally, we calculate the AUC for the AR-AN curve.
Results and Analysis We present the accuracy of several commonly used temporal action localization algorithms on Bullying10K for comparative evaluation. They exhibit varying performances under different features. On the other hand, performance significantly diminishes with the reduction of proposals. It is worth exploring a model designed for processing event datasets that can maintain higher precision with fewer proposals.
4.3 Pose estimation
Experimental Setting Human pose estimation typically involves providing an image or video containing one or multiple individuals and outputting the corresponding locations of various keypoints for each person. These keypoints include the head, shoulders, arms, and legs. Different datasets may have different numbers of joints, and in the case of Bullying10K, we use 26
Figure 6: Taking the prediction results of the SimpleBaseline model with ResNet101 as the backbone, we visualize the skeleton and heatmap of the prediction results, respectively.
keypoints. Pose estimation can provide additional information for action recognition. We split the dataset at an 8:2 ratio and have provided the corresponding pose annotations along with the data to ensure consistency. We evaluate the performance of Bullying10K using several commonly used pose estimation models.
Evaluation Metric The standard metric used to measure pose estimation is usually based on Object Keypoint Similarity (OKS) [71]. OKS is a scale-invariant measure of localization accuracy and is utilized to evaluate how closely the predicted keypoints of a model match the ground truth keypoints.  $OKS = \frac{\sum_{i}[\exp(-d_{i}^{2} / 2s^{2}k_{i}^{2})\delta(v_{i} > 0)]}{\sum_{i}[\delta(v_{i} > 0)]}$ , where  $d_{i}$  is the distance between the position of predicted keypoints and the ground truth.  $v_{i}$  denotes the visibility of related keypoints.  $s$  indicates the object scale.  $k_{i}$  is the predefined constant that controls falloff.Average Precision (AP) and Recall (AR) are used, including  $AP$ ,  $AP^{50}$  (OKS is at 0.50),  $AP^{75}$ ,  $AP^{M}$ ,  $AP^{L}$ , AR to measure the accuracy of different models on the Bullying10K dataset.
Results and Analysis Table 5 presents the results of multiple commonly used pose estimation models on the Bullying10K dataset. However, these models exhibit relatively low accuracy on the dataset, indicating that Bullying10K poses substantial challenges for pose estimation tasks. Additionally, although HRNet has shown superior capabilities to SimpleBaseline on other RGB datasets, it achieved lower accuracy on our dataset. This discrepancy may be attributed to the unique nature of our event-based dataset, which significantly differs from RGB images. Furthermore, we investigated the potential of SNN in posture estimation by testing the SNN backbone using the SimpleBaseline algorithm. To enhance the interpretability of the results, we visualized relevant output images and presented heatmaps for selected samples.
This research introduces a novel event-driven dataset called Bullying10K, which utilizes Dynamic Vision Sensor (DVS) cameras to detect instances of violent behavior while preserving individual privacy. This approach offers a novel possibility for privacy preservation, distinct from traditional image surveillance methods. It has an influence in the field of privacy protection and security surveillance.
However, typically, the kind of private information that cameras can capture is varied and encompasses aspects like facial details, gait patterns, and even individuals' habits of daily life. Some recent technologies can identify specific individuals using non-facial information [72, 73, 74]. Our dataset might struggle to prevent leaks of non-facial information, such as gait data. Still, we wish to emphasize that facial data is the most commonly used and has been incorporated into many crucial applications. This type of privacy is our primary focus in this paper.
Moreover, for the reason for clarity in labeling and for comparability during model validation, our dataset defines specific, commonly observed actions. It's challenging to cover every possible violent or non-violent action a person might exhibit comprehensively. However, an incomplete category set might increase the risk of classification errors in the model, which could adversely affect judgments regarding violence.
The dataset addressed the limitations of existing event-driven datasets by featuring complex, rapid movements and overlapping figures, presenting higher complexity and challenges. By offering a large-scale dataset, Bullying10K enables researchers to explore complex actions and contributes to advancements in violence detection and privacy-preservation techniques.
This paper is funded by the National Key Research and Development Program (Grant No. 2020AAA0107800).
[67] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[68] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Boundary-matching network for temporal action proposal generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3889-3898, 2019.
[69] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 466-481, 2018.
[70] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5693-5703, 2019.
[71] Qi Dang, Jianqin Yin, Bin Wang, and Wenqing Zheng. Deep learning based 2d human pose estimation: A survey. Tsinghua Science and Technology, 24(6):663-676, 2019.
[72] Changsheng Wan, Li Wang, and Vir V Phoha. A survey on gait recognition. ACM Computing Surveys (CSUR), 51(5):1-35, 2018.
[73] Zhengjie Wang, Kangkang Jiang, Yushan Hou, Wenwen Dou, Chengming Zhang, Zehua Huang, and Yinjing Guo. A survey on human behavior recognition using channel state information. IEEE Access, 7:155986-156024, 2019.
[74] Maria De Marsico and Alessio Mecca. A survey on gait recognition via wearable sensors. ACM Computing Surveys (CSUR), 52(4):1-39, 2019.