# Real-Time Surveillance System for Analyzing Abnormal Behavior of Pedestrians
## 3 Proposed Method
### A Set of Behavior Analysis Modules  
An object of interest is initially detected by the detector, and its location is tracked every frame by the tracker. To analyze the abnormal behavior of the interesting object, we developed a module, denoted as ObjectManager, that systematically manages the information obtained from the detector and tracker. ObjectManager composes information of each object as shown in Figure 3, where (x, y) represents the latest coordinate of the interesting object. ID is object label information for managing each object. The first-seen time means the time when the interesting object was first detected by the detector, and the last-seen time means the last time detected or tracked by the detector or tracker. Hue histogram is color histogram information for managing the same object. Information on a single object is composed of these six items. Figure 4 shows the flowchart of ObjectManager operation. ObjectManager first receives the coordinate of the interesting object from detector and tracker. Next, It determines whether the object is the same object by comparing with the object information currently managed in the Object List, and it is determined based on the distance of the coordinates. We additionally compare the color histogram information extracted from the objects to determine whether they are the same object [24]. The similarity of two histograms, denoted as \(H_{1}\) and \(H_{2}\), is defined as  
\[r(H_{1},H_{2})=\frac{\sum_{i}(H_{1}(i)-\bar{H_{1}})(H_{2}(i)-\bar{H_{2}})}{ \sqrt{\sum_{i}(H_{1}(i)-\bar{H_{1}})^{2}\sum_{i}(H_{2}(i)-\bar{H_{2}})^{2}}} \tag{4}\]  
where \(H_{1}\) and \(H_{2}\) represent the mean of the two histograms. Figure 5 shows an example of object tracking results using histogram comparison. It shows accurate tracking results even in the occlusion situation. Figure 4: ObjectManager Flowcharts. Figure 3: Object List. Where index represents the object similarity, if the index is determined to be the same, the information of the object is updated. The current input object information is then added as a new object information if the same object does not exist anymore. If the time of the last detected or tracked object is not updated for a certain period, the object is no longer observed in the input image and is removed from the Object List to manage only valid objects. The abnormal behavior analysis module acquires several close-up images of a specific interesting object on a continuous frame and uses them as input data. In this context, we need a function, denoted as ObjectCropper, to crop the image of the interesting object being tracked every frame. ObjectCropper manages the cropped image of the same object as a bundle, and then provides it to the abnormal behavior analysis module. ObjectCropper receives coordinates and size information of the interesting object. As a result, an appropriately cropped region is calculated for the interesting object. The image of each interesting object is separately saved for efficient management. When the number of cropped image reaches a pre-specified quantity, a sub-thread for dumping the image saves the image in the specified folder. The abnormal behavior analysis module takes the stored image as an input to analyze the abnormal behavior. Figure 6 shows the overall operation flow of ObjectCropper. The processing of recording the cropped image to the auxiliary storage device causes a serious bottleneck and adversely affects the performance of the overall system. Therefore, by operating a sub-thread for storing the cropped image, we can reduce the number of main-thread operations. The 'continuity' is an important condition to collect and manage cropped images. ObjectCropper collects cropped images of the interesting objects. In checking cropped Images, we determined whether the same object comes in consecutive frames. The continuity determination method used the assigned ID through ObjectManager and the last detection time.