###### Abstract  
Pedestrian behavior recognition in the driving environment is an important technology to prevent pedestrian accidents by predicting the next movement. It is necessary to recognize current pedestrian behavior to predict future pedestrian behavior. However, many studies have recognized human visible characteristics such as face, body parts or clothes, but few have recognized pedestrian behavior. It is challenging to recognize pedestrian behavior in the driving environment due to the changes in the camera field of view due to the illumination conditions in outdoor environments and vehicle movement. In this paper, to predict pedestrian behavior, we introduce a position-information added two-stream convolutional neural network (CNN) with multi task learning that is robust to the limited conditions of the outdoor driving environment. The conventional two-stream CNN is the most widely used model for human-action recognition. However, the conventional two-stream CNN based on optical flow has limitations regarding pedestrian behavior recognition in a moving vehicle because of the assumptions of brightness constancy and piecewise smoothness. To solve this problem for a moving vehicle, the binary descriptor dense scale-invariant feature transform (SIFT) flow, a feature-based matching algorithm, is robust in moving-pedestrian behavior recognition, such as walking and standing, in a moving vehicle. However, recognizing cross attributes, such as crossing or not crossing the street, is challenging using the binary descriptor dense SIFT flow because people who cross the road or not act the same walking action, but their location on the image is different. Therefore, pedestrian position information should be added to the conventional binary descriptor dense SIFT flow two-stream CNN. Thus, learning biased toward action attributes is evenly learned across action and cross attributes. In addition, YOLO detection and the Siamese tracker are used instead of the ground-truth boundary box to prove the robustness in the action- and cross-attribute recognition from a moving vehicle. The JAAD and PIE datasets were used for training, and only the JAAD dataset was used as a testing dataset for comparison with other state-of-the-art research on multitask and single-task learning. p edestrian-action recognition; two-stream convolutional neural network (CNN); binary descriptor dense scale-invariant feature transform (SIFT) flow; position-information feature +
Footnote †: journal: Medical Image and Vision  
future pedestrian behavior, current pedestrian behavior must be recognized. However, according to a recent survey on recognizing pedestrian behavior [15], few studies have addressed pedestrian behavior recognition. There has been considerable research on recognizing human-action datasets [16; 17; 18]. The OpenPose based convolutional neural network (CNN) has been used for spatial features [19; 20] and long short-term memory (LSTM) has been used for temporal features [21]. In recent years, the flow-image-based two-stream CNN has been widely used to deal with spatial and temporal features [22; 23; 24; 25]. Similar to human behavior, pedestrian behavior is also recognized by the OpenPose and LSTM [26; 27; 28]. The existing algorithms that recognize human behavior are difficult to apply in the driving environment with many obstacles such as buildings, trees or traffic poles due to the problematical conditions in the driving environment. Although there has been some research on the CNN and recurrent neural network (RNN), there is still no research on the flow-based two-stream CNN for pedestrian behavior recognition that is robust in moving vehicles in an outside environment. The most widely used two-stream CNN is the optical flow-based algorithm [22]. However, the optical flow has limitations concerning the brightness constancy and piecewise smoothness assumptions, making recognizing pedestrian behavior using optical flow challenging in moving vehicles and outside environments [29; 30].