## 3 Position-Information Feature Added Two-Stream CNN with Pedestrian Detector and Tracker
### Fusion with a Detector and Tracker without a Ground-Truth Boundary Box  
Recently, research on pedestrian-action recognition has trained the model by cropping the pedestrian image based on the ground-truth boundary box. However, the network model trained using the ground-truth boundary box has limitations in applying it to a vehicle driving in an actual outdoor environment. Research on the network model trained by cropping the image of the pedestrian through a detection algorithm without using the  
Figure 4: Position-information feature image representing the pedestrian position in the RGB image. Figure 5: Position-information feature-image added two-stream CNN structure. ground-truth boundary box has been studied recently to verify the performance of the recognition algorithm even in an actual driving vehicle [31]. In this paper, YOLO v3 [36] was selected to confirm that the performance of the proposed algorithm is guaranteed when applied to a moving vehicle in a real-world environment. A pedestrian action is a set of motions over time; thus, the detection algorithm should be continuously executed for the serial time of \(N\) frames. However, because the time and memory cost are inefficient when detection is performed per frame, detection is used only in the first frame, and the tracking algorithm is applied to the other frames [37]. In this case, the Siamese fully convolutional tracker [38] was selected because the Siamese tracker has a score map representing the target position in the image that can be used as the position-information feature image, as depicted in Figure 6. The position-information feature image generated with the Siamese tracker is used as input to the position-information feature added binary descriptor dense SIFT flow two-stream CNN, as presented in Figure 7.  
Figure 6: Position-information feature image obtained using a score map of the Siamese tracker output. Figure 7: Position-information feature-image added two-stream CNN structure with a Siamese tracker score map. JAAD and PIE Datasets  
Datasets: Many datasets classify human behavior, but few datasets classify pedestrian behavior [15]. Although there are few datasets, the representative datasets of pedestrian behavior are listed in Table 1. The dataset with the greatest number of frames is the STIP dataset [39], but this dataset only has cross attributes and no action attributes. Therefore, the STIP dataset is inappropriate for evaluating the proposed method. The JAAD [40] dataset is currently the most widely used in pedestrian behavior recognition research and consists of five attributes. Each attribute is composed as follows: action attributes (walking and standing), cross attributes (crossing and not crossing), look attributes (looking and not looking), hand-gesture attributes (great, yield, and right of way), and reaction attributes (clear path, speed up, and slow down). The numbers of training and testing samples expected from the JAAD dataset are provided in Table 2, where a sample is selected when the same attribute appears for 10 frames in a series. As presented in Table 2, the look, hand-gesture, and reaction attributes are too biased toward one label. Therefore, only action and cross attributes are used for pedestrian behavior recognition. However, because the number of JAAD samples is few for training the proposed method, PIE [41], created in JAAD creator, was also used. Unlike the JAAD dataset, because PIE has no distinction between the training and testing datasets, the entire PIE dataset was used as the training dataset. Thus, the training dataset consists of the JAAD and PIE datasets, whereas only the JAAD dataset was used for evaluation with other research as the testing dataset. Finally, the dataset is listed in Table 3.  
Multitask Learning: As presented in Table 3, the action and cross attributes were selected for pedestrian behavior recognition. Recent research has focused on single-task learning, which recognizes only action attributes [26] or only cross attributes [39; 42; 43; 44].