## 4 Pedestrian Dense Optical Flow Extraction  
Optical flow refers to the instantaneous velocity of the motion of each pixel in an image plane during the movement of an object. Correlation of two frames in this moving sequence is necessary to compute the offset of the object in two consecutive frames. Dense optical flow is an image comparison method that uses point-to-point matching of images. Compared with sparse optical flow, which compares only several feature points in an image, dense optical flow computes all pixel offsets in an image. To eliminate excessive  
Figure 3: Misaligned image [1]. information that the NN does not require such as background and pedestrian appearance--including colors of clothes and shifts between light and shadow--the present study set pedestrian movement as the focal point. In particular, gait sequences were described by capturing dense optical flow from two color images. In this study, the DNN used to capture dense optical flow was a variant of the fully convolutional network (FCN) framework called FlowNet2.0. The FCN was first introduced in [15] to solve a problem that the original CNN framework could not solve, namely recognizing an object at a particular location or a particular part of an image. Although the CNN framework is suitable for classifying a complete image, it cannot identify possible sub-elements in the image. Therefore, the FCN was introduced to solve such problems. The fundamental CNN usually flattens an entire image into a one-dimensional vector after classifying the convolution layer, followed by classification at the fully connected layer by using the Softmax function. The fundamental concept of the FCN is to classify each pixel in an image to achieve pixel-level image segmentation. Specifically, the FCN replaced the fully connected layer with a convolution layer. Therefore, the end result of the NN was an unflattened high-dimensional feature map. Another benefit of replacing the fully connected layer with a convolution layer is the possibility of sliding on large input images; the outcomes obtained from each location where the convolution kernel slid were the classification results of said locations in this study. FlowNet [16] operates by training an FCN and obtaining pixel-level optical flow in an image sequence. This method is based on two basic NNs, namely FlowNetS and FlowNetC, as well as a refinement. The main difference between FlowNetS and FlowNetC is that FlowNetS concatenates two consecutive frames for input, whereas FlowNetC separately inputs two image frames through three convolution layers before conducting correlation analysis on the two frames and calculating the relationship between the two feature maps. For single network operations, FlowNetC is superior to FlowNetS. The refinement section is a network that uses deconvolution to obtain the final optical flow, and employs bilinear interpolation, which requires relatively few computational resources yet does not cause considerable differences in results. Moreover, to not lose details, the deconvolution process concatenates the convolution result with the previous deconvolution prediction result. Because the results of using only FlowNetS or FlowNetC for training could still be improved, a follow-up study [11] concatenated these networks. Consequently, FlowNet2.0 with enhanced performance was developed; its network structure is presented in Figure 4 which was redrawn based on the information in [11]. Because FlowNetC is more accurate but FlowNetS can more easily replace an input, the first network was set as FlowNetC. Subsequently, bilinear interpolation was applied to the second frame and the result of the first network to calculate warping. Then, the brightness error was added as the input for the next network, namely FlowNetS. Because bilinear  
Figure 4: Concatenated network of FlowNet2.0. interpolation is suitable for computation of the gradient, the aforementioned three networks could be combined; this combination was named FlowNet-CSS.