# CNN-Based Crosswalk Pedestrian Situation Recognition System Using Mask-R-CNN and CDA
## 1 Introduction  
Researchers initially used CNN [1] to classify objects in images. Later, they found a way to differentiate and detect each object in the image. In addition, they conducted their studies on object relationships or situation recognition between detected objects. Our research aimed to perform situation recognition between objects by detecting pedestrians and crosswalk objects in images, respectively. Through deep learning training, our system can distinguish whether a pedestrian in a crosswalk is safe from the driver's point of view. Many researchers studied object detection and object relationships. [PERSON] et al. [2] devised the object relationship module using original and geometric weights to understand the dependence between objects for object detection [3]. [PERSON] et al. [4, 5] used YOLO (You Only Live Once) for object detection. [PERSON] et al. [6] suggested extracting objects and labeling visual semantic roles in the image using VGG (Visual Geometry Group) for situation recognition. [PERSON] et al. [7] developed a system for situation recognition in the image that applied the object detection stage using Fast R-CNN and a mutual recognition stage combining the pair filtering system for a subject feature and DR-Net. [PERSON] et al. [8] proposed GNN (Graphic Neural Network) that predicted the relationship between objects in images. They achieved this by analyzing the objects in the image on a graph. Furthermore,Shi et al. [9] recently suggested a new gait recognition system with a deep learning network. They used multimodal inertial sensors for their system.  
[PERSON] et al. [10], [PERSON] et al. [11], and [PERSON] et al. [12] proposed that color was an essential factor in CNN learning. Therefore, we devised a new method to color objects extracted from the image to distinguish the relationship between two objects of different colors. After learning the objects' colors, we demonstrated that our new method accurately classified the relationship between the objects. Therefore, we aimed to classify the location of the pedestrian on the crosswalk with the method. We trained CNN by extracting the image's crosswalks and pedestrians in different colors. We classified the pedestrian as safe when their feet were inside the crosswalk but dangerous outside. When tested experimentally, these learned CNN could accurately classify test images.  
For the safety of pedestrians, it is critical to determine whether a pedestrian is walking inside or outside the crosswalk area. According to the National Highway Traffic Safety Administration of the U.S.A. [13], among the different types of traffic accidents, there are many fatal ones involving pedestrians at crosswalks. We detected a pedestrian on the crosswalk using Mask R-CNN [14]. We then developed and applied the CDA to detect crosswalks. We colored pedestrians extracted in red and crosswalks in black. Among CNNs [15; 16; 17; 18; 19], we trained Xception [18] and ResNet50 [19] using the image data created in colors. Through experiments, we achieved 96-98% accuracy in classifying whether the test data were safe or dangerous with learned CNN. The contributions of this paper are as follows:  
Firstly, we proposed a CNN-based crosswalk-pedestrian situation recognition system that detects crosswalks and pedestrians in images and determines whether people are safe or in danger, depending on their location. We present further details of the method in Section 3.  
Secondly, we developed and applied the CDA to the system to detect crosswalks in images. We provide further details of the algorithm in Section 3.2.  
Thirdly, we created colored simple box shapes and used them for CNN learning. The experiment showed little difference in accuracy between actual field photo images and simplified boxes. We expect that somebody can prepare training datasets with efficient time and economic cost to train CNN. We present further details about simple box shapes in Section 4.2.