A specific perspective: Subway driver behaviour recognition using CNN and time‐series diagram

IET Intelligent Transport SystemsVolume 15, Issue 3 p. 387-395 ORIGINAL RESEARCH PAPEROpen Access A specific perspective: Subway driver behaviour recognition using CNN and time-series diagram Shize Huang, Shize Huang Shanghai Key Laboratory of Rail Infrastructure Durability and System Safety, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this authorLingyu Yang, Corresponding Author Lingyu Yang 1931294@tongji.edu.cn Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. China Correspondence Lingyu Yang, Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, 4800 Caoan Road, Shanghai, P. R. China. Email: 1931294@tongji.edu.cnSearch for more papers by this authorWei Chen, Wei Chen Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this authorTing Tao, Ting Tao Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this authorBingjie Zhang, Bingjie Zhang Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this author Shize Huang, Shize Huang Shanghai Key Laboratory of Rail Infrastructure Durability and System Safety, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this authorLingyu Yang, Corresponding Author Lingyu Yang 1931294@tongji.edu.cn Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. China Correspondence Lingyu Yang, Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, 4800 Caoan Road, Shanghai, P. R. China. Email: 1931294@tongji.edu.cnSearch for more papers by this authorWei Chen, Wei Chen Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this authorTing Tao, Ting Tao Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this authorBingjie Zhang, Bingjie Zhang Key Laboratory of Road and Traffic Engineering of Ministry of Education, Tongji University, Shanghai, P. R. ChinaSearch for more papers by this author First published: 28 January 2021 https://doi.org/10.1049/itr2.12032Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Urban rail transit, especially the subway, has been booming in China for a decade, imposing safety challenges on all related parties. Drivers' behaviours are particularly crucial. Typically, drivers' actions are recorded by cameras, and the surveillance videos are evaluated manually. Current driver behaviour recognition methods mostly target the bus or car drivers and can hardly be implemented for subways, because subway drivers follow a rigid working code that needs a time sequence of movements to describe. In this study, we propose a recognition model to automatically recognise behaviours from single-frame images that are extracted from surveillance videos; second, we convert the recognition results into time series diagrams, thus the recognised behaviours can be interpreted and analysed statistically and effectively. The validation experiments demonstrate that the convolutional neural network model can recognise 96.20% driver behaviours, and time series diagrams add time information to the behaviours, providing a convincible reference for subway driver evaluation. 1 INTRODUCTION Rapid urbanisation in China has caused a huge surge in rail transit [1], and it, in turn, has created concerns of safe operations. Among the influential factors, drivers' behaviours are crucial and can directly impact the safety of passengers [2]. Typically, a surveillance camera is deployed in the cab to record driver's activities, and drivers' behaviours will be evaluated through analysing those videos. This process, however, has to be completed manually, which is laborious and ineffective. This job has become particularly burdensome in China, given the increasing amount of trains and drivers in recent years. Therefore, a system specifically for subway trains that can intelligently identify and evaluate drivers' behaviours is quite tempting, and algorithms of human behaviour recognition exhibit the potential to fulfil this task. Human behaviour recognition has received extensive attention in transportation systems. For instance, Wang et al. proposed sampling motion trajectory [3] to identify human behaviours and then proposed an improved version of the dense trajectory extraction method [4], which once occupied a leading position in the field of behaviour recognition. Ohnbar et al. [5] proposed a framework for leveraging three cues obtained from videos capturing car drivers' hands and heads in order to provide activity recognition and prediction. Yang et al. [6] collected images that record the movements of car drivers' upper-body joints from surveillance videos and identified their distractions and dangerous behaviours. A similar method was proposed by Chao Yan et al. [7] that monitors drivers' hand movements and considers them as a predictor of safe driving. These methods can accurately identify human behaviours and extract movement trajectories, but their cameras hold a broader and clearer view and can capture detailed movements of drivers. Subway trains' cabs, however, are more cramped and views of the surveillance cameras are more limited, where current trajectory-based methods can barely function. Some researchers found another way that does not need a clear view as before [8]. They classified images before sending them to behaviour recognition. For example, Poppe et al. transformed the problem of behaviour recognition into a classification of image sequences and discussed various representation and classification methods for images [9]. Fortunately, convolutional neural network (CNN) model can lead this way further, because it can find the most effective features directly from the input data [10]. CNNs can classify and detect two-dimensional images through convolution and pooling operations [11, 12]. Many scholars have noticed CNNs' expertise and applied it to drivers' behaviours recognition. Behera et al. integrated the body posture into the existing deep networks, and the performance of driver activity recognition was significantly improved [13]. Yan et al. [14] and Xing et al. [15] used the Gaussian mixture model (GMM) to extract images of car drivers' skin area. Then they adopted deep learning methods, either R*CNN or Alexnet, to classify behaviours. Pang et al. [16] also directly classified car drivers' behaviours into eight categories and constructed a deep CNN framework to identify and classify them. Chen et al. [17] processed the raw RGB images with a GMM-based segmentation algorithm and proposed a driving-related activity recognition system based on the deep CNN model. Although these deep learning-based methods are effective, they all performed on separate frames without time-series information, and their targets, that is, car drivers, follow more flexible operation codes. Subway drivers' working specification, however, is more rigid and only sequences information being incorporated can feed safety requirements. Therefore, we utilised CNN model's recognition proficiency in single-frame images, specifically targeting at videos with certain angles taken from subway trains' cabs. Then we adopted time-series diagrams to further investigate the previous recognition results and provided a direct and statistical reference for subway drivers' behaviour evaluation. 2 METHODS 2.1 Technical route The proposed approach works in three steps as shown in Figure 1. First, a surveillance video is decomposed and a sequence of frames is extracted; second, these frames are input into a CNN model and behaviours of these frame can be recognised. Third, the recognition results are converted into a time-series diagram, where the drivers' behaviours can be further calculated and analysed, and the judgement whether the driver follows the rules can be achieved. FIGURE 1Open in figure viewerPowerPoint Technical route for subway driver behaviour recognition 2.2 Subway driver behaviour recognition model based on CNN 2.2.1 Preprocessing of input data A video recording a driver driving a subway train is obtained as input data. First, the video is converted into a sequence of consecutive frames, which are used to construct the dataset of subway driver behaviours. Second, some sequential frames are selected from that dataset as test samples, and the rest are taken as training samples. Training samples are not necessarily sorted in a sequence but need to be classified and labelled according to the manual of 'Daily Operation for Drivers'. The manual defines four types of the driver job: Mainline driving, inbound operation, platform operation and outbound operation. Their detailed descriptions are shown in Figure 2. FIGURE 2Open in figure viewerPowerPoint Driver's operations in each type of standard operation Thus, driver behaviours can be classified into six categories accordingly: Standing on the platform, pointing call, hanging gear, pressing the start button, driving between two stations and leaving the seat, and images in training samples can be labelled and classified into the six categories. We assign each category a number as its label as illustrated in Table 1. TABLE 1. labels corresponding to the six categories Category Label Standing on the platform 0 Pointing call 1 Hanging gear 2 Pressing the start button 3 Driving between two stations 4 Leaving the seat 5 2.2.2 CNN architecture CNNs are a class of feedforward neural networks with convolutional computation and deep structure. A typical CNN contains three layers as illustrated in Figure 3 [18]. The first is the input layer; the second, the combination of the convolutional layer and the pooling layer; and the third, the fully connected output layer. FIGURE 3Open in figure viewerPowerPoint Architecture of a typical convolutional neural network (CNN) First, the input layer accepts data including original audios, RGB images, and so forth. Then the convolution layer uses a convolution kernel to convolve the local image area and obtains local information of the image, that is, to extract features. The calculation for each convolution operation is as follows: y m ′ , n ′ = f ∑ j = 0 J − 1 ∑ i = 0 I − 1 x m + i , n + j ∗ w i j + b , × 0 ≤ m ≤ M , 0 ≤ n ≤ N where J, I are the width and the height of the convolution kernel, respectively; M, N are the width and the height of the input image, respectively; w is the weight; b is bias; X is the pixel value at (m, n) position in the input image, and y is the calculation result of the convolution operation. Second, the convolution layer utilises the pooling layer for feature mapping. The obtained feature maps are subsampled, so both the data size and the amount can be reduced. Third, the final layer, the fully connected layer, acts as a classification to map features to the tag space of the data. Among these layers, the convolved and the fully connected layers contain neurons whose weight is learned and adjusted. Thus, the input data can be better represented after training. 2.2.3 CNN model establishment Considering the excellent performance of CNN in image recognition, this study takes the behaviour data of subway drivers as the input of a CNN model and the behaviour category as the output, and thus obtains the mapping relationship between them as shown in Figure 4. FIGURE 4Open in figure viewerPowerPoint Roadmap using CNN to identify subway driver behaviour We choose Alexnet model to identify behaviours of subway drivers. Alexnet network structure is shown in Figure 5. First, the first convolutional layer filters the 150 × 150 × 3 input image pixels with 96 kernels of size 11 × 11 with a stride of 4, resulting in 96 feature maps of size 148 × 148. Second, a max pooling with a kernel size of 3 × 3 and a stride of 2 is used to filter maximum activations from the output 96 feature maps of the first step. Third, another convolution layer with a filter size of 3 × 3 and a stride of 1 is applied, followed by another max pooling layer with a kernel size of 3 × 3 and a stride of 2. By this analogy, the Alexnet network used in this study can be established as explained in Table 2, where 'f' is the size of a convolution kernel or a pooling kernel, 's' the step size, and 'd' the number of convolution kernels in a layer. FIGURE 5Open in figure viewerPowerPoint Alexnet network structure TABLE 2. The network structure constructed in this study Layer (type) Parameter information Output shape Input shape – (None, 150, 150, 3) Conv2d_1(Conv2D) f = 11; s = 4; d = 96 (None, 148, 148, 96) Max_pooling2d_1(MaxPooling2D) f = 3; s = 2 (None, 73, 73, 96) Conv2d_2(Conv2D) f = 3; s = 1; d = 256 (None, 73, 73, 256) Max_pooling2d_2(MaxPooling2D) f = 3; s = 2 (None, 36, 36, 256) Conv2d_3(Conv2D) f = 3; s = 1; d = 384 (None, 36, 36, 384) Conv2d_4(Conv2D) f = 3; s = 1; d = 384 (None, 36, 36, 384) Conv2d_5(Conv2D) f = 3; s = 1; d = 256 (None, 36, 36, 256) Max_pooling2d_3(MaxPooling2D) f = 3; s = 2 (None, 36, 36, 256) Dense_1 f = 3; s = 1; d = 4096 4096 Dense_2 f = 3; s = 1; d = 4096 4096 Dense_3 f = 3; s = 1; d = 4096 6 Thus, the Alexnet network is ready for training. We have to ensure that the mapping relationship is stable, that is, any input can correspond to a certain output that is infinitely approaching the expected value. During training, one can adjust the parameters to improve accuracy, reduce the loss function value and converge loss function gradually. 2.3 Time-series diagram of subway driver behaviours The CNN model can identify drivers' actions through single-frame images, but it is not enough. Individual actions cannot reveal actual behaviours if they are not connected by time. Therefore, we convert the recognition results output by the CNN model into time-series diagrams and recommend decisions based on them. First, the recognition results of the CNN model are converted into a time-series diagram. Second, through analysis on the diagram, driver behaviours information and parameters of the running train are obtained, including the number of behaviours of each category, duration of behaviours of each category, and status of the train. Third, this information, combined with the standardisation operation manual, is evaluated and decisions on whether the subway drivers' behaviour is qualified are recommended. 2.3.1 Constructing a time-series diagram First, the trained CNN model output recognition results of driver images, and these images must be in a time sequence, like those test samples are selected. The recognition result of each frame is an array of probability as [a,b,c,d,e,f], where 'a', 'b', 'c', 'd', 'e', 'f' indicate the probabilities of the six labels '0', '1', '2', '3', '4', '5', respectively, and the sum of a, b, c, d, e, and f equals 1. In order to clearly describe the process of converting the probabilities array of each image into an intuitive and accurate recognition result, we use the analytical calculation formula: y = argmax R i , i = 1 , 2 , 3 , … where i is the number of frames of the image, and Ri is an array of probabilities of each image. The argmax function is to find the label that corresponds to the maximum probability in the array, and the label is taken as the classification result. The number of frames of each type can then be calculated as follows: s u m a = enumerate i | y = a , a = 0 , 1 , 2 , 3 , 4 , 5 where i represents the number of frames, and y the classification result. We take the number of frames i as the abscissa and the obtained result y as the ordinate. Thus, a time-series diagram is constructed. Figure 9 exemplifies a time series of certain frames. 2.3.2 Time-series graph analysis The behaviour of a subway driver is a continuous process. Only the number of frames of each type of behaviours can hardly describe the driver's actions. Therefore, we adopt the number of occurrences of each type of behaviours and the duration of occurrence to further analyse drivers. For example, Figure 6 illustrates how to calculate the number and the duration of the behaviour of 'standing on the platform', labelled as class 4. There are two occurrences with the starting points f1, f3 and the endpoints f2, f4. FIGURE 6Open in figure viewerPowerPoint Time-series diagram analysis The number of occurrences is defined as F = m 2 where f m is the endpoint of the last occurrence of certain behaviour. And the duration of each occurrence can be calculated as T = f b − f a F P S where f b , f a are the endpoint and the starting point of an occurrence, respectively; and FPS is the frame rate per second. 3 EXPERIMENT AND VALIDATION 3.1 CNN training process 3.1.1 Dataset of subway driver behaviours We took a video clip from a cab camera installed in a train on Beijing metro line 9. The clip was shot on 7 February 2018, from 8:25:26 to 9:00 A.M., lasting 35 minutes with a total of 64,651 frames. During this period, the train passed 13 stations, and the running between two stops lasted for 1 to 2 min. As shown in Figure 7, the camera was installed above the head of the driver; thus, the driver's face was partly out of the frame while his hand movements and other behaviours could be clearly recorded. The monitoring screen also displayed time and the train number. FIGURE 7Open in figure viewerPowerPoint A typical frame of a subway driver video Given the same standards for the operation that all the subways in China use, the same uniform that all the drivers wear during work, and the same view that surveillance cameras in the cabs capture, our model could be easily applied to any other subway system in China. Therefore, choosing a video containing all the behaviours of a driver as a training dataset is sufficient. Given the small difference between two adjacent frames, we extracted every five consecutive frames at regular intervals and included these frames into a dataset. In order to balance the sample size of each type of behaviours, we took 1830 frames as CNN training samples and 1801 frames as CNN test samples. We classified and labelled images for training by using the method described in Section 2.2.1 and took 1290 of them into a training set and the rest 540 into a validation set. Table 3 shows the grouping of network training samples and test samples. TABLE 3. Grouping of network training samples and test samples Category 0 1 2 3 4 5 Training set 215 215 216 217 219 214 Validation set 92 89 91 84 78 93 Training samples 307 304 307 301 297 307 Test samples 1801 3.1.2 CNN model accuracy We established a CNN model on keras 2.1.5 platform by following the rules described in Section 2.2 and trained it by using the images in the training set, which were classified and labelled. We compared the parameter adjustment technique with the multiple training results and set the batch size as 16 and the epochs as 10, which improved the training accuracy. Figure 8 exhibits the accuracy of the training process. The accuracy of validation is 0.9620, and the accuracy of training is 0.9790. Therefore, the final model is satisfying. FIGURE 8Open in figure viewerPowerPoint Accuracy of the training process 3.2 Time-series diagram analysis result We selected a sequence of frames that recorded the train passing through four stations from the test dataset and input them into the trained CNN model. The recognition output was then converted into a time-series diagram as shown in Figure 9. FIGURE 9Open in figure viewerPowerPoint The time-series diagram of driver behaviours Based on the diagram, we calculated the number of occurrences of each behaviour and the duration of each occurrence, and the results are listed in Table 4. TABLE 4. Statistical analysis on the time series diagram of driver behaviours Category 1st 2nd 3rd 4th 5th 6th 7th 8th Total number of times Standing on the platform(s) 21 7 6 14 14 16 4 – 7 Pointing call (s) 0.6 1.2 0.6 1.2 0.6 0.6 0.6 – 7 Hanging gear (s) 1.8 1.8 0.6 1.2 1.2 1.8 0.6 – 7 Pressing the start button (s) 3 4.2 2.4 5.4 – – – – 4 Driving between two stations (s) 110 119 85 74 – – – – 4 Leaving the seat (s) 7 13 10 4 15 6 12 18 8 Table 4 shows that behaviours of 'pointing call' and 'hanging gear' last shorter, while 'pressing the start button' lasts 3 s and the driving time is the longest. When the driver is 'standing on the platform', indicated by category '0', the train stops at the platform; when the driver is 'driving between two stations', indicated by category '4', the train is running. Therefore, the diagram also tells the durations when the train stops and runs as listed in Table 5. TABLE 5. Statistical analysis on the time series diagram of train movements Number of times 1st 2nd 3rd 4th 5th Train stop time (s) 48 97 60 52 27 Train running time (s) 132 198 105 71 – 4 DISCUSSIONS 4.1 Validation of subway driver behaviour recognition model The test samples, a total of 1801 images, were input to the subway driver behaviour recognition model. The result of each frame was an array of probabilities [a, b, c, d, e, f] of the six kinds of behaviours the driver might conduct. Table 6 exemplifies part of the results. TABLE 6. Test sample output Standing on the platform Pointing call Hanging gear Press the start button Running between stations Leave the seat 52,560 1.0347593 × 10–8 4.3404917 × 10–8 7.1643971 × 10–9 5.2520575 × 10–11 9.5856615 × 10–7 9.9999893 × 10–1 52,565 1.8748904 × 10–910 1.4004886 × 10–10 2.1892687 × 10–11 1.2578310 × 10–12 1.6192482 × 10–8 1.0000000 × 10 52,570 1.5068697 × 10–7 6.0993983 × 10–8 1.5047421 × 10–8 2.8598182 × 10–10 7.7750713 × 10–7 9.9999905 × 10–1 52,575 2.3944835 × 10–7 3.4055816 × 10–7 6.3143261 × 10–8 1.0548872 × 10–9 1.0711810 × 10–6 9.9999833 × 10–1 52,580 8.5007224 × 10–7 8.7707485 × 10–7 1.1776500 × 10–7 1.4590531 × 10–9 9.7298043 × 10–6 9.9998844 × 10–1 52,585 1.8127248 × 10–6 5.9039947 × 10–5 5.4842672 × 10–5 1.1641970 × 10–6 7.8987988 × 10–4 9.9909329 × 10–1 52,590 3.1076308 × 10–7 7.1637833 × 10–1 8.9139576 × 10–5 5.5969855 × 10–7 2.6287390 × 10–2 2.5724414 × 10–1 52,595 7.3294188 × 10–8 1.8052933 × 10–7 9.9856746 × 10–1 1.8926667 × 10–6 1.3433982 × 10–3 8.7041539 × 10–5 52,600 9.3588049 × 10–8 1.9473011 × 10–7 9.9975902 × 10–1 2.3187745 × 10–6 1.7079091 × 10–4 6.7666362 × 10–5 52,605 6.1261344 × 10–6 1.6782607 × 10–5 9.7602046 × 10–1 3.3859874 × 10–4 1.3506545 × 10–2 1.0111445 × 10–2 52,610 2.8831559 × 10–7 5.8356549 × 10–11 4.0319763 × 10–5 9.9990058 × 10–1 2.2366801 × 10–7 5.8489386 × 10–5 52,615 1.1395276 × 10–6 2.3271424 × 10–10 5.3103351 × 10–5 9.9259627 × 10–1 1.5984773 × 10–5 7.3335408 × 10–3 52,620 2.0357131 × 10–6 3.8383455 × 10–10 1.0603770 × 10–4 9.8621935 × 10–1 3.4591860 × 10–5 1.3638013 × 10–2 52,625 3.9712544 × 10–9 3.5478795 × 10–9 3.9483064 × 10–5 2.1154327 × 10–7 9.9807274 × 10–1 1.8874322 × 10–3 52,630 3.1996948 × 10–12 2.2094734 × 10–10 3.5052326 × 10–5 1.6667391 × 10–10 9.9996412 × 10–1 7.7541409 × 10–7 52,635 2.6510291 × 10–12 8.6592367 × 10–10 4.9461931 × 10–5 1.3541204 × 10–10 9.9994981 × 10–1 7.1673668 × 10–7 The numbers in red in Table 6 are the most possible categories they belong to. We manually checked each image to validate behaviour recognition. Among the 1801 frames, only 77 false positives occurred. The accuracy of the proposed network reached 95.7%, which is consistent with its training results and also reflects its superior performance. 4.2 Analysis of the behaviour of subway drivers The manual of 'Daily Operation for Drivers' defines the sequence of actions a subway driver should take: Mainline driving, inbound operation, platform operation and outbound operation. We interpreted this sequence into the order of the behaviours as 4→1→2→5→0→5→1→2→3→4, that is, driving between two stations → pointing call →hanging gear → leaving the seat →standing on the platform → leaving the seat → pointing call →hanging gear → pressing the start button →driving between two stations. This sequence of behaviours should repeat during the driver's whole working time. Therefore, we used this sequence as a standard to analyse drivers' behaviours. The results of those test samples are shown in Table 7. When all the driver's actions are in compliance, we can determine that the driver follows the rules; otherwise, the relevant personnel will be informed. TABLE 7. Evaluation on subway driver behaviours Number Category Station one Station two Station three Station four 1 Running between two stations √ √ √ √ 2 Pointing call √ √ √ √ 3 Hanging gear √ √ √ √ 4 Leave the seat √ √ √ √ 5 Station stand √ √ √ √ 6 Leave the seat √ √ √ √ 7 Pointing call √ √ √ √ 8 Hanging gear √ √ √ √ 9 Press the start button √ √ √ √ 10 Running between two stations √ √ √ √ 4.3 Accuracy comparison To further validate the superiority of our CNN model, we compared it with other deep learning methods proposed in [3, 4, 14, 16, 17], and the results are shown in Figure 10. Methods in [3, 4] using positions of hands or joints achieved accuracies of 82.4% and 89.62%, respectively, which are within a reasonable range but not very high. Methods in [14, 16, and 17] used deep learning for behavioural classification and achieved accuracies of 97.76%, 94%, and 98.89%, respectively, which are a huge leap from those in [3, 4]. FIGURE 10Open in figure viewerPowerPoint Comparison of our method with other deep learning methods proposed in refs. [3]: Dense trajectory extraction; [4]: Improved trajectories; [14]: DenseNet; [16]: DedistractedNet; [17]: CNN model Our approach, the proposed Alexnet network, holds an accuracy of 96.20%, not the highest, but it goes further. It converts the recognition results into a time-series diagram for later analysis and decision making. 5 CONCLUSION In this study, we proposed a behaviour recognition model to intelligently identify the behaviours of subway drivers. Our approach combines a CNN model and time-series diagrams, targeting at the restricted angles of surveillance videos and strict operation codes of subway drivers. Through the validation experiments, conclusions can be drawn as follows: The CNN model we construct can achieve a high accuracy of 96.20%, which is impressive, especially when the drivers' joints and hands movements are not clear. Time-series diagrams we borrow for better interpreting behaviours that are recognized by the CNN model can provide statistical and time-related information for drivers' actions, which is a solid reference for driver evaluation. Our approach is the first to add time information to behaviour recognition, which provides not only a widely required automatic tool but also an effective reference for a safe driving evaluation. We intend to further analyse drivers' behaviours and divide them into more categories so that the behavioural evaluation model will be more convincing. And we are also interested in improving the practicality of our approach in multiple fields, for example, in cities and on highways where drivers' movements are more flexible. ACKNOWLEDGEMENTS This research is supported by the National Key R&D Program of China (2016YFB1200402), and the National Natural Science Foundation of China (Grant No. 61703308) and Sichuan Province Science and Technology Support Program (2019YFG0040). The authors gratefully acknowledge the invaluable contribution of the reviewers. REFERENCES 1Xu, R., et al.: Empirical analysis of traveling backwards and passenger flows reassignment on a metro network with automatic fare collection (AFC) data and train diagram. Transp. Res. Rec. 2672(8), 230– 242 (2018) 2Brage-Ardao, R., et al.: Determinants of train service costs in metro operations. Transp. Res. Rec. 2534(1), 31– 37 (2015) 3Wang, H., et al.: Action recognition by dense trajectories. In: CVPR 2011. IEEE, Providence, RI, USA, pp. 3169– 3176 (2011) 4Wang, H., Schmid, C.: Action recognition with improved trajectories. In: Proceedings of the IEEE International Conference on Computer Vision, Sydney, NSW, Australia, pp. 3551– 3558 (2013) 5Ohnbar, E., et al.: Head, eye, and hand patterns for driver activity recognition. In: 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden (2014) 6Xing, Y., et al.: Identification and analysis of driver postures for in-vehicle driving activities and secondary tasks recognition. IEEE Trans. Comput. Social Syst. 5(1), 95– 108 (2017) 7Yan, C., et al.: Video-based classification of driving behaviourusing a hierarchical classification system with multiple features. Int. J. Pattern Recognit. Artif. Intell. 30(05), 1650010 (2016) 8Poppe, R.: A survey on vision-based human action recognition. Image Vision Comput. 28(6), 976– 990 (2010) 9Herath, S., et al.: Going deeper into action recognition: A survey. Image Vision Comput. 60, 4– 21 (2017) 10Krizhevsky, A., et al.: ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, Lake Tahoe, Nevada, USA, vol. 60, pp. 1097– 1105 (2012) 11Luo, C., et al.: Image classification by selective regularized subspace learning. IEEE Trans. Multimedia 18(1), 40– 50 (2015) 12Ji, S.W., et al.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221− 231 (2013) 13Yan, S., et al.: Driver behavior recognition based on deep convolutional neural networks. In: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD). Changsha, China, pp. 636– 641 (2016) 14Behera, A., Keidel, A.H.: Latent body-pose guided densenet for recognizing driver's fine-grained secondary activities. In: 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). Auckland, New Zealand (2018) 15Xing, Y., et al.: End-to-end driving activities and secondary tasks recognition using deep convolutional neural network and transfer learning. In: IEEE Intelligent Vehicles Symposium, Changshu, China, pp. 1626– 1631 (2018) 16Pang, Y., et al.: An advanced deep framework for recognition of distracted driving behaviours. In: 2018 IEEE 7th Global Conference on Consumer Electronics (GCCE), Nara, Japan, pp. 802– 803 (2018) 17Yang, X., et al.: Driver activity recognition for intelligent vehicles: A deep learning approach. IEEE Trans. Veh. Technol. 68(6), 5379–5390 (2019) 18Muhammad, K., et al.: Convolutional neural networks based fire detection in surveillance videos. IEEE Access 6, 18174– 18183 (2018) Citing Literature Volume15, Issue3March 2021Pages 387-395 FiguresReferencesRelatedInformation