# Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians
## 3 Proposed Method
### Pedestrian Detection and Tracking Method  
Since abnormal behavior analysis should work with real-time streaming images as input, object detection does not perform at every frame, and a tracking algorithm estimates the coordinate of the object in the skipped frames. More specifically, the object detection network detects an object at every 12 frames, and the coordinate of the detected object are used as the input of the tracker. The reason for not using detection every frame is that it takes a lot of computation to perform detection every frame of a video. In the context, we performed the detection periodically and used the tracking method in between. A certain period was set to 12 frames showing the best performance when tracking and detection were merged according to the experimental results. It is necessary to update newly detected or disappeared at a specific point in the input image. Figure 2a,b, respectively, show the situation where the object goes out of the image and the situation where the object enters the house through the window. Existing trackers keep tracking the most similar patch even after the object disappears. Since object detection is performed every 12 frames, the tracker is updated according to the detected person in the corresponding area to determine whether to track the object or not. Figure 1: Proposed abnormal behavior analysis network. \[ConfidenceScore=P_{r}(Object)\times IOU^{truth_{area}} \tag{1}\]  
In the proposed method, the object detection network uses YOLOv4 as the baseline network. The unified detection increases the processing speed compared with two-stage methods such as Regions with Convolutional Neuron Networks features (R-CNN) [22]. The input image is divided into grid cells and the score for each object is calculated. The score is calculated by Equation (1), and if there is no object in the image, the score becomes 0. \(IOU^{truth_{area}}\) represents the intersection divided by union of the correct answer box and the predicted box, and \(P_{r}(Object)\) represents the probability that an object exists within the bounding box. \[ConditionalClassProbability=P_{r}(Class|Object) \tag{2}\]  
The probability corresponding to each class is expressed as the following Equation (2). \(P_{r}(Class_{i}|Object)\) represents the classification probability of the object in the grid cell. The detector divides the input image into an S X S grid. Each grid cell has the confidence score defined in (1) for the bounding box. If there is no object in the cell, the confidence score is 0. In addition, a class-specific confidence score is obtained by multiplying the confidence score of (1) and the Conditional Class Probability of (2). Finally, the output of the detector include the center and size of the object, and the probability of matching the learned class. Object tracking is then performed by setting the coordinates of the object to be detected as the input of the tracker. The KCF tracker has two advantages: (i) efficient computation by replacing the correlation operation in the spatial domain with the element-by-element operation in the Fourier domain and (ii) robustness against shape changes using online training. Its performance can be further improved by incorporating the weight of the color histogram in the target patch. The formula for generating a color space by weighting the main colors is as follows:  
\[I_{G}=\frac{\sum\mu^{c}I_{P}^{c}}{\sum\mu^{c}},for\,c\in\{R,G,B\}, \tag{3}\]  
where \(I_{G}\) is the output of the grayscale image, \(\mu^{c}\) represents the relative ratio of \(RGB\) color values, and \(I_{P}\) is the patch of the target object. To improve the tracking performance in a scene with objects having different color values, the proposed tracking method considers the color distribution of the object, and adaptively weight the color components. To detect a very small object, we used five YOLO layers, and additional learning was performed with images having small objects.