# Temporal Saliency-Based Suspicious Behavior Pattern Detection
## Materials and Methods
### Description of Proposed Method
#### 2.2.3 Reactivity Map Generation  
(1) The distinction between ordinary and suspicious behavior  
The statistical value of the magnitude of the motion vector is used to distinguish ordinary motion from suspicious motion in the feature map represented by the polar coordinate system. The statistical value in the region where suspicious behavior occurs is different from the statistical value in the region where ordinary behavior occurs. To determine the statistical value, experiments were performed. Four video sequences were selected among No. 4, No. 5, No. 7, and No. 9 data, which were mentioned in Section 2.1 and were edited to include actual suspicious behaviors such as falling, jumping, violence, and suddenly running behaviors in the 40 th frame of each video. Selected video sequences all contain suspicious behaviors. However, for testing purposes, these suspicious behaviors have been edited to appear in the 40 th frame. Edited videos are 15 s long, and there are about 15 people in the first video, eight people in the second video, three people in the third video, and about 13 people in the fourth video. After that, motion vectors were calculated, and the results were analyzed. Figure 4 shows the graphs that show the statistical values of motion vectors of the four kinds of videos after editing the image frame to contain suspicious behavior in the 40 th frame. A legend 'value' shown in Figure 4a means the minimum, maximum, and average value of the motion vector, and Figure 4b means the standard deviation value of the motion vector. After the motion vector was extracted from these edited videos, the statistical information of the motion vector was analyzed. As we can see in Figure 4a, the motion vector value was maximized near 40 frames of each video. This was because each video contained suspicious behavior (crashes, drowsiness, falls, etc.) near the 40 th frame. Since the sizes of objects showing suspicious behavior are slightly different for each dataset image, there could be a slight difference in the average value. Figure 4b also shows that there is a large increase in the standard deviation near 40 frames. As a result of extracting and experimenting motion vectors for several videos, the average value is different according to the size of the moving region included in the video, while the value of the standard deviation changes a lot when suspicious behavior appears. Through these experiments, the standard deviation is selected as the criterion value for judging the abnormal behavior, and value 1.2 was selected. Figure 4: Statistical value of the motion vector. (**a**) Maximum, minimum, average; (**b**) standard deviation. Figure 3: Magnitude and gradient feature map. (**a**) original frame; (**b**) \(F_{\text{mag}}\); (**c**) \(F_{\text{grad}}\). (2) Magnitude Reactivity Map Generation: \(R_{\text{mag}}\)  
The frame was considered that it contains suspicious behavior when a frame with a standard deviation of the motion vector at the time (t) is greater than 1.2, which was calculated experimentally. The region where the magnitude of the vector is calculated by optical flow at the point (x, y) is larger than the summation of the average, and the variance of the whole image was considered to be the suspicious behavior region. Since the movement is slightly different for each person even when performing the same ordinary behavior, the region was detected based on the summation of the average and the variation. Based on these facts, the magnitude reactivity map (\(R_{\text{mag}}\)) is calculated as follows. \[ \begin{gathered} R_{mag}=F_{mag}(x,y,t)\times\sigma(F_{mag}(x,y,t ))\\ \text{where, }\sigma\Big{(}F_{mag}(x,y,t)\Big{)}>1.2\text{ and }F_{mag}(x,y,t)>\mu \Big{(}F_{mag}(x,y,t)\Big{)}+\sigma\Big{(}F_{mag}(x,y,t)\Big{)}\end{gathered}  \tag{2}\]  
Figure 5 shows the magnitude reactivity map generated by the proposed method. Figure 5a shows the video of people gathered at the center of the park moving in various directions simultaneously with any signal.