## 1 Introduction  
With the advance of artificial intelligence, pedestrian loitering detection has been more and more widely applied in many fields. As one case of abnormal behavior detection, it plays an important role in the urban video surveillance system. According to IMS Research's research prediction, in the next five years the market demand of domestic security software will increase at the speed of 30% to 50%, which is enough to show that today's video surveillance and management software platform has huge potential market [1; 2; 3]. The demand of video surveillance systems in public places is large [4; 5]. Pedestrian loitering detection is mainly applied to safety monitoring in public places such as public transportation areas to ensure the urban safety [6]. Automatic loitering detection can help recognize vulnerable people who needing attention and potential suspects that might be harmful to public security [7]. For example, older patients with Alzheimer's disease in vulnerable groups are prone to be victims of unexpected security event [8; 9]. Nowadays, people are paying more and more attention to this. Alzheimer's patients are often in a state of loitering when they need attention. In addition, public safety has always been a matter of great concern. The presence of dangerous suspicious individuals in public places poses a great threat [10; 11; 12]. Some scholars made an example about human behavior detection in transit scenes told us that detecting specific activities like loitering needs scarce human resources [13]. Even with costly human resources, facing such a mass of data, operators may be unfocused because of fatigue, distraction, or interruption [14]. It means that traditional loitering detection is achieved by manual monitoring through observation or with assistance of videos, which is not only time-consuming, but also hard in operation. During recent years, different kinds of approaches have been proposed to detect loitering behavior, but there are not explicit definition of loitering in the existing research literatures. Some researchers define loitering by time, others define loitering by classify trajectories by curvature or grid [15; 16], these studies consider single scenario, without considering the different needs of multiple scenarios and external environments. [PERSON] [17] compared the stationary time of boats on a given area with the time that boats normally stay to determine whether the boats are loitering. Our paper detects pedestrian loitering in different situations. It uses the change of pedestrian activity area size to detect pedestrian loitering. The common loitering detection methods mainly include the feature optical flow method and the motion trajectory method [18]. Trajectory detection is one of the most common and accurate methods. There are also some motion trajectory-based detected methods [19; 20]. The abnormal behavior was analyzed by combining the motion tracking and shape aspect ratio [21]. [PERSON] [22] used the trajectory distribution and the probability of the trajectory feature to determine loitering. The loitering detection methods in literatures [23; 24] were performed by capturing the trajectories of the pedestrian area. Although there is large amount of research on trajectory analysis, using trajectory analysis to detect loitering is still a challenge. Authors like [PERSON] [25] detected loitering by analyzed the simple trajectory but ignored the reality, loitering is usually random and arbitrary. Another loitering detection method based on trajectory tracking in literature [26] needed use historical traces as an assistive model. A method in the literature [27] analyzed the trajectory of object through the duration time and angles variation between the initial point and remaining points on the trajectory, but is not suitable for practical complex scenarios. Target missing, occlusion, detection errors, and trajectory terminations all belong to complex scenarios. In order to achieve satisfactory trajectory detection results, complex scenarios must be handled [28].