# Automatic Traffic State Recognition Based on Video Features Extracted by an Autoencoder

Authors: Xiaoyu Cai; Qiongli Jing; Bo Peng; Yuanyuan Zhang; Yuting Wang; Ju Tang
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Video surveillance has become an important measure of urban traffic monitoring and control. However, due to the complex and diverse video scenes, traffic data extraction from original videos is a sophisticated and difficult task, and corresponding algorithms are of high complexity and calculation cost. To reduce the algorithm complexity and subsequent computation cost, this study proposed an autoencoder model which effectively reduces the video dimension by optimizing structural parameters; thus several traffic recognition models can conduct image processing work based on dimension-reduced videos. Firstly, an optimal autoencoder modelA∗ with five hidden layers was constructed. Then, it was combined with a linear classifier, support vector machine, deep neural network, DNN linear classification method, and the k-means clustering method; thus, five traffic state recognition models were constructed: A∗-Linear, A∗-SVM, A∗-DNN, A∗-DNN_Linear, and A∗-k-means. Train and test results show that the accuracy rate and recall rate of A∗-linear, A∗-SVM, A∗-DNN, and A∗-DNN_Linear were 94.5%–97.1%, and the F1 score was 94.4%–97.1%; besides, the accuracy rate, recall rate, and F1 score of A∗-k-means were all approximately 95%, which suggests that the combination of the autoencoder A∗ and common classification or clustering methods achieve good recognition performance. Comparison was also implemented among the five models proposed above and four CNN-based models such as AlexNet, LeNet, GoogLeNet, and VGG16, which shows that the five proposed modes achieve F1 scores of 94.4%–97.1%, while the four CNN-based models achieve F1 scores of 16.7%–94%, indicating that the proposed light weight design methods outperform more complex CNN-based models in traffic state recognition.


---

## Body

## 1. Introduction

With the rapid development of urbanization and the sustainable development of a national economy, traffic congestion has increasingly become a key issue in large and medium-sized cities. To effectively control traffic operations and alleviate traffic congestion, it is very important to achieve traffic state recognition in a timely and effective manner. Currently, extensive studies have been conducted on traffic state recognition using a variety of methods, such as fuzzy logic in [1], the fuzzy C-means clustering algorithm in [2, 3], machine learning in [4, 5], and the K-means clustering algorithm in [6]. Data used in these studies were mainly from fixed surveillance systems such as cameras and vehicle detectors, as well as vehicle GPS in [7, 8], etc. With the enrichment of urban traffic surveillance video resources and the progress of image processing technology, traffic state recognition based on video images has gained wide attention.A large number of studies have been conducted on traffic state recognition based on video images. Traditionally, traffic state recognition was achieved by establishing certain algorithms to extract the traffic image characteristics in [9, 10]. Since these algorithms had high complexity, they were not able to cope with the rapid increase in mass traffic surveillance videos. To avoid the extraction, analysis, and modeling of complex image features, deep learning methods have recently been used to achieve rapid and effective image pattern recognition using large-scale image training. The convolutional neural network (CNN) is one of the most common deep learning methods and has achieved great success in image classification and object detection tasks in [11, 12]. However, to the best of our knowledge, a few studies have used the CNN in traffic state recognition based on video images.Video images often have high dimensionality and contain complex and redundant information. To recognize the traffic states timely and effectively, attempts have been conducted to reduce the dimensionality of video images, thereby performing image pattern recognition based on the dimension-reduced data in [13]. Encoders, which are nonlinear and unsupervised neural network models that include an input layer, a hidden layer, and an output layer, can effectively reduce the dimensionality of video image data and realize image classification in [14].Traffic data extraction from original videos is a sophisticated and difficult task because of the complex and diverse video scenes, and corresponding algorithms are of high complexity and calculation cost. In this study, to reduce the algorithm complexity and subsequent computation cost, an autoencoder model was proposed which effectively reduces the video dimension by optimizing structural parameters. Then, a traffic state recognition method was established by integrating the autoencoder with common classification and clustering methods. The train and test result shows that the proposed method has the advantages of a lightweight model structure and low calculation cost, which outperforms CNN models such as AlexNet and GoogLeNet. Therefore, it is suitable for traffic state recognition from videos.


## 2. Background

This section explains the relevant background of the video-based traffic state recognition, and related work are reviewed, including traditional traffic video processing, convolutional neural networks, and autoencoders.The research results of video-based traffic state recognition are very fruitful. Among them, the earlier research mainly achieved traffic state recognition by establishing certain algorithms to extract the traffic image characteristics. By extracting a discrete cosine transform and its movement characteristics based on traffic surveillance videos, the hidden Markov model of fuzzy C-means was used to learn and recognize the traffic congestion state. A fuzzy logic method for traffic state recognition based on camera images was proposed. A traffic state recognition method was proposed, using traffic monitoring videos based on the fuzzy C-means clustering algorithm. A method recognizing three traffic states (free traffic flow, steady traffic flow, and forced traffic flow) based on images and video data was achieved in [15], and an image-based traffic density estimation method was proposed, in which a support vector machine classifier was used to classify traffic states into heavy, medium, and light densities. In general, this type of algorithms has high complexity and poor real-time video processing ability.The convolutional neural network (CNN) has achieved rapid and effective image pattern recognition using large-scale image training, which can avoid the extraction, analysis, and modeling of complex image features. It is one of the most common deep learning methods and has achieved great success in image classification and object detection tasks. In 2012, Krizhevsky et al. brought out AlexNet, which ranked the first in the ImageNet image classification competition in [16]. Since then, the network structure has been improved, and other CNN models, such as visual geometry group (VGG) and GoogLeNet, have been proposed. A UAV image vehicle recognition method based on CNN and SVM was proposed in [17]. A satellite image vehicle detection method based on CNN and hard case mining was put forward in [18]. The authors applied MIT and Caltech vehicle datasets, and proposed a vehicle detection method in traffic scenes based on improved Faster RCNN in [19]. The CNN has gained extensive interest in large-scale image classification, but less study has used the CNN in traffic state recognition based on video images.Encoders, which are nonlinear and unsupervised neural network models, can effectively reduce the dimensionality of video image data in [20, 21]. A supervised stacked autoencoder to extract image features was constructed in [22]. In [23], the authors integrated a random forest classifier into the stacked sparse autoencoder for hyperspectral image classification, which produced promising generalization performance, prediction accuracy, and operation speed. A dual adversarial autoencoder for image clustering was proposed in [24], which achieved comparable clustering accuracy to CNN models. In [25], an image compression architecture based on energy compression using a convolutional autoencoder was established, which achieved high coding efficiency. In [26], the authors used an autoencoder to project high-dimensional image feature vectors into low-dimensional potential space and proposed an image selection method based on autoencoder neural networks and then applied it to semisupervised image classification. An enhanced fast NSGA-II based on a special congestion strategy and an adaptive crossover strategy, namely ASDNSGA-II, was proposed to improve the selection strategy in [27]. The studies above demonstrate that autoencoders can effectively extract the image features through dimension reduction and show good image recognition results when combined with classification or clustering methods.


## 3. Methodology

### 3.1. Methodological Framework

In this study, an autoencoder with multiple hidden layers was used to compress the image features. Based on the compressed low-dimensional data, classification and clustering methods were used for traffic state recognition. The proposed method consisted of four steps (Figure1), described as follows:(1)
Establishment of image datasets. Traffic images were generated based on traffic surveillance videos. The traffic state was determined manually for each frame, and an image dataset with three traffic states (i.e., free traffic flow, steady traffic flow, and congested traffic flow) was established.(2)
Image data preprocessing. The image matrix was transformed into row vectors, and the vector element values were normalized and taken as the input of the autoencoder(3)
Construction of the autoencoder. The influence of the main structural parameters, such as the dimensionality of the input data, the number of hidden layers, and the dimensionality of the dimension-reduced data, on the performance of the autoencoder was evaluated, and as a result, an autoencoder suitable for traffic image processing was constructed. The autoencoder was trained, tested, and optimized based on feedback.(4)
Traffic state recognition. The support vector machine (SVM), deep neural network (DNN), and k-means clustering method were used to classify the feature data obtained using the autoencoder, thereby achieving traffic state recognitionFigure 1
Flowchart of the proposed method.


### 3.2. Autoencoder Construction

The autoencoder was composed of an encoder and a decoder. The encoder extracted the features of the original image data, and the decoder restored the feature data. The smaller the difference between the restored data and the original data was, the better the autoencoder, and the more effective the extracted feature data.Traffic surveillance video images were taken as the input of the autoencoder. Both the encoder and decoder hadN hidden layers, and the network had a symmetrical structure. There was a fully connected weighting structure between each pair of layers, as shown in Figure 2. Taking the qth image in an image dataset with a sample size Q as an example, the operation of the autoencoder is as follows:(1)
Theqth image is converted into a 1×τ0 vector(1)X0q=x1q,x2q,…,xiq,…,xτ0q,whereq∈1,Q. Here, τ0=l×h and l,h are the width (pixels) and height (pixels) of the image, and i is the element number of X0q.(2)
Encoding:N encoding hidden layers were used to perform a nonlinear transformation of X0q to obtain N encoding vectors X1q,X2q,…,Xnq,…,XNq, with the dimensions of 1×ϒ1,1×ϒ2,…,1×ϒn,…,1×ϒN, respectively. The calculation equation is as follows:(2)Xnq=ΓwnqXn−1q+znq,whereΓ is the encoding nonlinear activation function, which was a hyperbolic tangent function tanhx in this study; wnq is the encoding weight matrix of the nth encoding hidden layer in the qth image; XNq represents the encoding vector of the qth image; znq represents the encoding offset matrix of the nth encoding hidden layer in the qth image; n represents the encoding hidden layer number, and n∈1,N.The calculation equation oftanhx is as follows:(3)tanhx=sinhxcoshx=ex−e−xex+e−x,wherex is equal to wnqXn−1q+znq.(3)
Image decoding:N decoding hidden layers were used to perform a nonlinear transformation of XNq, to obtain N decoding vectors, with the dimensions of X˜Nq,…,X˜φq,…,X˜2q,X˜1q, respectively. The decoding vectors were restored to the decoding vector X˜0q with a dimension of 1×τ0. The calculation equation is as follows:(4)X˜φ−1q=Γ˜w˜φqX˜φq+z˜φq,whereΓ˜ is the decoding nonlinear transformation function, which was a hyperbolic tangent function tanh() in this study; w˜φq is the decoding weight matrix of the φth decoding hidden layer in the qth image; z˜φq is the decoding offset matrix of the φth decoding hidden layer in the qth image; φ is the decoding hidden layer number, and φ∈1,N.(4)
Feedback optimization of encoding-decoding training: with the least mean square (LMS) as the objective, through continuous feedback propagation training, the autoencoder minimizes the error between the inputX0q and output X˜0q, to obtain the optimal autoencoder structure and encoding data. The error loss calculation equation is as follows:(5)minLX0q,X˜0q=1τ0∑xiq−x˜iq2s.t.X0q=x1q,x2q,…,xiq,…,xτ0q.Figure 2
Autoencoder structure.


### 3.3. Traffic State Recognition

In this study, the dimension-reduced dataX^N obtained using the optimal autoencoder A∗ was used as the input of the classification and clustering methods for traffic state recognition, including linear classifier in [28], SVM in [29], DNN in [30], and DNN linear combined classifier in [28].(6)X^N=X^N1,X^N2,…,X^Nq,…,X^NQ.Here,X^Nq is the encoding vector of the qth image obtained using A∗, X^Nq=x^1q,x^2q,…,x^iq,…,x^ϒNq, x^iq is the value of the ith element, and Q represents the sample size of the dataset, i.e., the number of images.(1)
The linear classifier, a typical supervised learning model, finds the optimal hyperplane that separates the two classes of samples according to the characteristics of the samples. TakingX^Nq=x^1q,x^2q,…,x^iq,…,x^ϒNq as an example, let the optimal hyperplane be equation (14).(7)κ1x^1q+κ2x^2q+⋯+κix^iq+⋯+κϒNx^ϒNq+κ0=0.The left part of the equation can be defined as a linear discriminant function, as shown in (15).(8)gx^q=∑i=1ϒNκix^iq+κ0.Let the weightsκ=κ1,κ2,…,κτN and κ0 be the offset. Then, the inner product is gx^q=κX^NqT+κ0. The linear classifier gx^q divides the feature space into two regions: gx^q>0 and gx^q<0; in other words, the data points are divided into two categories: σ1and σ2. The optimal hyperplane can be obtained by training the sample data.(9)gx^q=kXNq∧T+κ0=>0,x^q∈σ1,0,beyond recognition,<0,x^q∈σ2.(2)
As a linear classifier, SVM is widely used in image processing, face detection, speech recognition, and so on. The dimension-reduced data that corresponds to each image inX^N was labeled with a category label π, π=−1,1, which indicates that the data are to be divided into two categories. Then, a training set U=X^N1,π1,X^N2,π2,⋅⋅⋅,X^Nq,πq,⋅⋅⋅,X^NQ,πQ was established to train the SVM, thereby obtaining the optimal hyperplane to classify the samples.(10)μTX^N+b=0.Here,μ∈Rn,b∈R represent the normal vector and intercept of the hyperplane, respectively. By introducing a slack variable α, the matrix form of the SVM can be expressed as follows:(11)minμ,b,α12X^NTμ+eb2+νeTαs.t.−X^NTμ++eb≥e−αα>0.whereX^NT is the transpose of the matrix X^N, ν is the penalty parameter, and e is the vector in which each element is 1. By solving the optimization problem of (18), μ and can be solved.(3)
The DNN mainly includes an input layer, a hidden layer, and an output layer. There is a fully connected structure of weights between each pair of layers. This method is widely used in image recognition, natural language processing, speech recognition, and so on. Its structure is shown in Figure3.The input layer hasϒN neurons, which are connected with the element values of ϒN and correspond to X^Nq=x^1q,x^2q,…,x^iq,…,x^ϒNq. After being processed by M hidden layers (the dimension of each layer is 1×Ω1, 1×Ω2,…, 1×ΩM), the probability of the ς category is obtained. The sample category is obtained using the softmax function.(4)
The DNN linear combined classifier consisted of two parts: a linear model and a DNN model. The linear model could receive inputs of continuous features or sparse discrete features and quickly converge to effective feature vectors usingL1 regularization. The principle of the DNN is the same as described above. The DNN Linear combined classifier used the joint training method to simultaneously feedback the error to the linear model and the DNN model to update the parameters. The sigmoid function was used to stack the outputs of the linear model and the DNN model to obtain the final output.Figure 3
DNN network structure.


## 3.1. Methodological Framework

In this study, an autoencoder with multiple hidden layers was used to compress the image features. Based on the compressed low-dimensional data, classification and clustering methods were used for traffic state recognition. The proposed method consisted of four steps (Figure1), described as follows:(1)
Establishment of image datasets. Traffic images were generated based on traffic surveillance videos. The traffic state was determined manually for each frame, and an image dataset with three traffic states (i.e., free traffic flow, steady traffic flow, and congested traffic flow) was established.(2)
Image data preprocessing. The image matrix was transformed into row vectors, and the vector element values were normalized and taken as the input of the autoencoder(3)
Construction of the autoencoder. The influence of the main structural parameters, such as the dimensionality of the input data, the number of hidden layers, and the dimensionality of the dimension-reduced data, on the performance of the autoencoder was evaluated, and as a result, an autoencoder suitable for traffic image processing was constructed. The autoencoder was trained, tested, and optimized based on feedback.(4)
Traffic state recognition. The support vector machine (SVM), deep neural network (DNN), and k-means clustering method were used to classify the feature data obtained using the autoencoder, thereby achieving traffic state recognitionFigure 1
Flowchart of the proposed method.


## 3.2. Autoencoder Construction

The autoencoder was composed of an encoder and a decoder. The encoder extracted the features of the original image data, and the decoder restored the feature data. The smaller the difference between the restored data and the original data was, the better the autoencoder, and the more effective the extracted feature data.Traffic surveillance video images were taken as the input of the autoencoder. Both the encoder and decoder hadN hidden layers, and the network had a symmetrical structure. There was a fully connected weighting structure between each pair of layers, as shown in Figure 2. Taking the qth image in an image dataset with a sample size Q as an example, the operation of the autoencoder is as follows:(1)
Theqth image is converted into a 1×τ0 vector(1)X0q=x1q,x2q,…,xiq,…,xτ0q,whereq∈1,Q. Here, τ0=l×h and l,h are the width (pixels) and height (pixels) of the image, and i is the element number of X0q.(2)
Encoding:N encoding hidden layers were used to perform a nonlinear transformation of X0q to obtain N encoding vectors X1q,X2q,…,Xnq,…,XNq, with the dimensions of 1×ϒ1,1×ϒ2,…,1×ϒn,…,1×ϒN, respectively. The calculation equation is as follows:(2)Xnq=ΓwnqXn−1q+znq,whereΓ is the encoding nonlinear activation function, which was a hyperbolic tangent function tanhx in this study; wnq is the encoding weight matrix of the nth encoding hidden layer in the qth image; XNq represents the encoding vector of the qth image; znq represents the encoding offset matrix of the nth encoding hidden layer in the qth image; n represents the encoding hidden layer number, and n∈1,N.The calculation equation oftanhx is as follows:(3)tanhx=sinhxcoshx=ex−e−xex+e−x,wherex is equal to wnqXn−1q+znq.(3)
Image decoding:N decoding hidden layers were used to perform a nonlinear transformation of XNq, to obtain N decoding vectors, with the dimensions of X˜Nq,…,X˜φq,…,X˜2q,X˜1q, respectively. The decoding vectors were restored to the decoding vector X˜0q with a dimension of 1×τ0. The calculation equation is as follows:(4)X˜φ−1q=Γ˜w˜φqX˜φq+z˜φq,whereΓ˜ is the decoding nonlinear transformation function, which was a hyperbolic tangent function tanh() in this study; w˜φq is the decoding weight matrix of the φth decoding hidden layer in the qth image; z˜φq is the decoding offset matrix of the φth decoding hidden layer in the qth image; φ is the decoding hidden layer number, and φ∈1,N.(4)
Feedback optimization of encoding-decoding training: with the least mean square (LMS) as the objective, through continuous feedback propagation training, the autoencoder minimizes the error between the inputX0q and output X˜0q, to obtain the optimal autoencoder structure and encoding data. The error loss calculation equation is as follows:(5)minLX0q,X˜0q=1τ0∑xiq−x˜iq2s.t.X0q=x1q,x2q,…,xiq,…,xτ0q.Figure 2
Autoencoder structure.


## 3.3. Traffic State Recognition

In this study, the dimension-reduced dataX^N obtained using the optimal autoencoder A∗ was used as the input of the classification and clustering methods for traffic state recognition, including linear classifier in [28], SVM in [29], DNN in [30], and DNN linear combined classifier in [28].(6)X^N=X^N1,X^N2,…,X^Nq,…,X^NQ.Here,X^Nq is the encoding vector of the qth image obtained using A∗, X^Nq=x^1q,x^2q,…,x^iq,…,x^ϒNq, x^iq is the value of the ith element, and Q represents the sample size of the dataset, i.e., the number of images.(1)
The linear classifier, a typical supervised learning model, finds the optimal hyperplane that separates the two classes of samples according to the characteristics of the samples. TakingX^Nq=x^1q,x^2q,…,x^iq,…,x^ϒNq as an example, let the optimal hyperplane be equation (14).(7)κ1x^1q+κ2x^2q+⋯+κix^iq+⋯+κϒNx^ϒNq+κ0=0.The left part of the equation can be defined as a linear discriminant function, as shown in (15).(8)gx^q=∑i=1ϒNκix^iq+κ0.Let the weightsκ=κ1,κ2,…,κτN and κ0 be the offset. Then, the inner product is gx^q=κX^NqT+κ0. The linear classifier gx^q divides the feature space into two regions: gx^q>0 and gx^q<0; in other words, the data points are divided into two categories: σ1and σ2. The optimal hyperplane can be obtained by training the sample data.(9)gx^q=kXNq∧T+κ0=>0,x^q∈σ1,0,beyond recognition,<0,x^q∈σ2.(2)
As a linear classifier, SVM is widely used in image processing, face detection, speech recognition, and so on. The dimension-reduced data that corresponds to each image inX^N was labeled with a category label π, π=−1,1, which indicates that the data are to be divided into two categories. Then, a training set U=X^N1,π1,X^N2,π2,⋅⋅⋅,X^Nq,πq,⋅⋅⋅,X^NQ,πQ was established to train the SVM, thereby obtaining the optimal hyperplane to classify the samples.(10)μTX^N+b=0.Here,μ∈Rn,b∈R represent the normal vector and intercept of the hyperplane, respectively. By introducing a slack variable α, the matrix form of the SVM can be expressed as follows:(11)minμ,b,α12X^NTμ+eb2+νeTαs.t.−X^NTμ++eb≥e−αα>0.whereX^NT is the transpose of the matrix X^N, ν is the penalty parameter, and e is the vector in which each element is 1. By solving the optimization problem of (18), μ and can be solved.(3)
The DNN mainly includes an input layer, a hidden layer, and an output layer. There is a fully connected structure of weights between each pair of layers. This method is widely used in image recognition, natural language processing, speech recognition, and so on. Its structure is shown in Figure3.The input layer hasϒN neurons, which are connected with the element values of ϒN and correspond to X^Nq=x^1q,x^2q,…,x^iq,…,x^ϒNq. After being processed by M hidden layers (the dimension of each layer is 1×Ω1, 1×Ω2,…, 1×ΩM), the probability of the ς category is obtained. The sample category is obtained using the softmax function.(4)
The DNN linear combined classifier consisted of two parts: a linear model and a DNN model. The linear model could receive inputs of continuous features or sparse discrete features and quickly converge to effective feature vectors usingL1 regularization. The principle of the DNN is the same as described above. The DNN Linear combined classifier used the joint training method to simultaneously feedback the error to the linear model and the DNN model to update the parameters. The sigmoid function was used to stack the outputs of the linear model and the DNN model to obtain the final output.Figure 3
DNN network structure.


## 4. Field Experiments

### 4.1. Dataset

The original data consisted of traffic surveillance videos of a main road in a Chinese city, and the image resolution was 704 × 576 pixels. The images were extracted and divided into three traffic states: free traffic flow (free), steady traffic flow (steady), and congested traffic flow (congested). Two datasets were constructed:A1 has a sample size of 1500, which was used to evaluate the influence of the autoencoder structural parameters, and A2 has a sample size of 6000, which was used to optimize the autoencoder and traffic state recognition. The number of images was the same for each traffic state in both datasets. The datasets are shown in Figure 4.Figure 4
Construction of the image datasets.


### 4.2. Optimization of the Autoencoder

#### 4.2.1. Model Training

The training parameters included the basic learning rate, batch training size, and the number of iterations. In the process of model training, the adaptive moment estimation (Adam) iterative optimization algorithm in [31] was used to dynamically adjust the learning rate of each parameter using the first-order moment estimation and second-order moment estimation. The equations are as follows:(12)χt=η×χt−1+1−η×gt,(13)ϑt=ρ×ϑt−1+1−ρ×gt2,(14)χ^t=χt1−ηt,(15)ϑ^t=ϑt1−ρt,(16)υt=υt−1−χ^tϑ^t+ε×β,where gt is the gradient of time step t; χt is the first-order moment estimation of, gt i.e., the exponential moving average of gt; ϑt is the second-order moment estimation of gt, i.e., the exponential moving average of gt; η and ρ are the exponential decay rates; χ^t is the deviation correction of χt; ϑ^t is the deviation correction of ϑt; υt is the parameter vector of time step t; the default learning rate is β=0.001; and ε is the residual term, ε=10−8.The Adam algorithm has a low requirement for memory during the computation. After deviation correction, each iterative learning rate was within a certain range, which results in stable parameters. Therefore, the Adam algorithm was suitable for training and optimization of the large datasets and high-dimensional data that were addressed in this study.


#### 4.2.2. Evaluation of the Autoencoder

According to the loss distribution curve of the autoencoder, two indexes were used to measure its convergence and evaluate the performance of the autoencoder.(1)
A thresholdΘ was set for the loss. When the training loss of the autoencoder became less than Θ after performing iterations to step θ, the average loss L¯Θ between steps θ-D was calculated, as shown in Figure 5. The smaller the value of L¯Θ is, the smaller the difference between the original input data and the restored data of the autoencoder, and the better the performance of the autoencoder.(17)L¯Θ=1D−θ+1∑j=θj=DLj,whereD is the total number of training iterations, Lj is the loss of the j th iteration, j∈θ,D, and Θ is the loss threshold.(2)
The loss distribution curve of the autoencoder was fitted, and the tangent slope at the end of the fitting curve was calculated.(18)kD=y′D,wherey′ is the loss fitting curve function and y′D is the first-order derivative at the Dth iteration, i.e., the slope.As shown in Figure6, when kD was less than 0; the closer it is to 0, the gentler the loss curve, and the better the convergence effect.Under the same parameter settings and training conditions, asL¯Θ decreased, more effective feature data could be extracted and restored by the autoencoder; as kD became less than 0 and approached 0, the convergence effect of the autoencoder loss was better, which indicates high encoding-decoding stability and reliability.(3)
Structure optimization of the autoencoder. To design an autoencoder for video traffic state recognition, 36 sets of crossover tests were conducted based on the dimension of the input data, the dimension of the dimension-reduced data, and the number of hidden layers, as shown in Table1.Figure 5
Calculation diagram ofL¯Θ.Table 1
Design of crossover tests.
Input data dimensionDimension-reduced data dimensionNumber of hidden layersModel number.32 × 323–6, step length: 16–10, step length: 21–1264 × 643–6, step length: 16–10, step length: 213–24128 × 1283–6, step length: 16–10, step length: 225–36Figure 6
Calculation diagram ofkD.The 36 models were trained using the sample datasetA1 with the same training parameters. The basic learning rate was 0.0001. The batch training size was 100, and the number of iterations was 15,000.The distribution of the loss was analyzed, and the average loss whenΘ was 0.01 was calculated, as shown in Figure 7. The average losses of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th groups were at a lower level compared with other groups of tests. In these eight groups of tests, the input data dimension of the first four groups was 32 × 32 and that of the last four groups was 64 × 64. The number of hidden layers in each test was 10. Therefore, when the input data dimension decreased and the number of hidden layers increased, L¯0.01 decreased; the effect of the dimension of the dimension-reduced data on the loss was not significant. However, as the dimension of the dimension-reduced data increased, the model training loss tended to decrease. The eight groups of models were preliminarily selected as the candidate models.Figure 7
L¯0.01 distribution of crossover tests.Then, curve fitting was performed on the loss of the 36 groups of tests. The coefficients of determinationR2 of the fitting functions were all between 0.84 and 0.95. Only 13 groups of tests had R2 lower than 0.9. Therefore, the goodness of fit was high on the whole. The kD of the loss fitting function of each group of tests was calculated, as shown in Figure 8. In general, the tangent slopes of the loss fitting curves of each group of tests were all less than 0 and close to 0. The curve convergence effect was good, and the tangent slopes of the eight candidate models were all small. Therefore, the autoencoder models of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th groups of tests were selected as the candidate models and numbered as model I–model VIII. The structure is shown in Table 2.Figure 8
Tangent slopekD distribution of loss curve.Table 2
Structural parameters of the eight candidate autoencoder models.
Model numberInput data dimensionNumber of autoencoder hidden layersDimension of the dimension-reduced dataAutoencoder structureI32 × 3253(32 × 32,256), (256,128), (128,64), (64,12), (12,3)II32 × 3254(32 × 32,256), (256,128), (128,64), (64,12), (12,4)III32 × 3255(32 × 32,256), (256,128), (128,64), (64,12), (12,5)IV32 × 3256(32 × 32,256), (256,128), (128,64), (64,12), (12,6)V64 × 6453(64 × 64,256), (256,128), (128,64), (64,12), (12,3)VI64 × 6454(64 × 64,256), (256,128), (128,64), (64,12), (12,4)VII64 × 6455(64 × 64,256), (256,128), (128,64), (64,12), (12,5)VIII64 × 6456(64 × 64,256), (256,128), (128,64), (64,12), (12,6)Using the sample datasetA2, the candidate models I–VIII were tested. The basic learning rate was 0.0001. The batch training size was 300, and the number of iterations was 100,000. According to the distribution of the training loss, Θ was set to 0.005. L¯0.005 and kD of each model were calculated. The results are shown in Table 3. Compared with other models, model III had the smallest value of L¯0.005 and the tangent slope kD. Therefore, given the performance of model III, it was selected as the optimal autoencoder model A∗.Table 3
L¯0.005 and slope kD of the candidate models.
ModelIIIIIIIVVVIVIIVIIIL¯0.005 (10–3)3.002.172.112.172.812.782.462.80kD (10–7)1.951.451.381.421.801.781.571.77


## 4.1. Dataset

The original data consisted of traffic surveillance videos of a main road in a Chinese city, and the image resolution was 704 × 576 pixels. The images were extracted and divided into three traffic states: free traffic flow (free), steady traffic flow (steady), and congested traffic flow (congested). Two datasets were constructed:A1 has a sample size of 1500, which was used to evaluate the influence of the autoencoder structural parameters, and A2 has a sample size of 6000, which was used to optimize the autoencoder and traffic state recognition. The number of images was the same for each traffic state in both datasets. The datasets are shown in Figure 4.Figure 4
Construction of the image datasets.


## 4.2. Optimization of the Autoencoder

### 4.2.1. Model Training

The training parameters included the basic learning rate, batch training size, and the number of iterations. In the process of model training, the adaptive moment estimation (Adam) iterative optimization algorithm in [31] was used to dynamically adjust the learning rate of each parameter using the first-order moment estimation and second-order moment estimation. The equations are as follows:(12)χt=η×χt−1+1−η×gt,(13)ϑt=ρ×ϑt−1+1−ρ×gt2,(14)χ^t=χt1−ηt,(15)ϑ^t=ϑt1−ρt,(16)υt=υt−1−χ^tϑ^t+ε×β,where gt is the gradient of time step t; χt is the first-order moment estimation of, gt i.e., the exponential moving average of gt; ϑt is the second-order moment estimation of gt, i.e., the exponential moving average of gt; η and ρ are the exponential decay rates; χ^t is the deviation correction of χt; ϑ^t is the deviation correction of ϑt; υt is the parameter vector of time step t; the default learning rate is β=0.001; and ε is the residual term, ε=10−8.The Adam algorithm has a low requirement for memory during the computation. After deviation correction, each iterative learning rate was within a certain range, which results in stable parameters. Therefore, the Adam algorithm was suitable for training and optimization of the large datasets and high-dimensional data that were addressed in this study.


### 4.2.2. Evaluation of the Autoencoder

According to the loss distribution curve of the autoencoder, two indexes were used to measure its convergence and evaluate the performance of the autoencoder.(1)
A thresholdΘ was set for the loss. When the training loss of the autoencoder became less than Θ after performing iterations to step θ, the average loss L¯Θ between steps θ-D was calculated, as shown in Figure 5. The smaller the value of L¯Θ is, the smaller the difference between the original input data and the restored data of the autoencoder, and the better the performance of the autoencoder.(17)L¯Θ=1D−θ+1∑j=θj=DLj,whereD is the total number of training iterations, Lj is the loss of the j th iteration, j∈θ,D, and Θ is the loss threshold.(2)
The loss distribution curve of the autoencoder was fitted, and the tangent slope at the end of the fitting curve was calculated.(18)kD=y′D,wherey′ is the loss fitting curve function and y′D is the first-order derivative at the Dth iteration, i.e., the slope.As shown in Figure6, when kD was less than 0; the closer it is to 0, the gentler the loss curve, and the better the convergence effect.Under the same parameter settings and training conditions, asL¯Θ decreased, more effective feature data could be extracted and restored by the autoencoder; as kD became less than 0 and approached 0, the convergence effect of the autoencoder loss was better, which indicates high encoding-decoding stability and reliability.(3)
Structure optimization of the autoencoder. To design an autoencoder for video traffic state recognition, 36 sets of crossover tests were conducted based on the dimension of the input data, the dimension of the dimension-reduced data, and the number of hidden layers, as shown in Table1.Figure 5
Calculation diagram ofL¯Θ.Table 1
Design of crossover tests.
Input data dimensionDimension-reduced data dimensionNumber of hidden layersModel number.32 × 323–6, step length: 16–10, step length: 21–1264 × 643–6, step length: 16–10, step length: 213–24128 × 1283–6, step length: 16–10, step length: 225–36Figure 6
Calculation diagram ofkD.The 36 models were trained using the sample datasetA1 with the same training parameters. The basic learning rate was 0.0001. The batch training size was 100, and the number of iterations was 15,000.The distribution of the loss was analyzed, and the average loss whenΘ was 0.01 was calculated, as shown in Figure 7. The average losses of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th groups were at a lower level compared with other groups of tests. In these eight groups of tests, the input data dimension of the first four groups was 32 × 32 and that of the last four groups was 64 × 64. The number of hidden layers in each test was 10. Therefore, when the input data dimension decreased and the number of hidden layers increased, L¯0.01 decreased; the effect of the dimension of the dimension-reduced data on the loss was not significant. However, as the dimension of the dimension-reduced data increased, the model training loss tended to decrease. The eight groups of models were preliminarily selected as the candidate models.Figure 7
L¯0.01 distribution of crossover tests.Then, curve fitting was performed on the loss of the 36 groups of tests. The coefficients of determinationR2 of the fitting functions were all between 0.84 and 0.95. Only 13 groups of tests had R2 lower than 0.9. Therefore, the goodness of fit was high on the whole. The kD of the loss fitting function of each group of tests was calculated, as shown in Figure 8. In general, the tangent slopes of the loss fitting curves of each group of tests were all less than 0 and close to 0. The curve convergence effect was good, and the tangent slopes of the eight candidate models were all small. Therefore, the autoencoder models of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th groups of tests were selected as the candidate models and numbered as model I–model VIII. The structure is shown in Table 2.Figure 8
Tangent slopekD distribution of loss curve.Table 2
Structural parameters of the eight candidate autoencoder models.
Model numberInput data dimensionNumber of autoencoder hidden layersDimension of the dimension-reduced dataAutoencoder structureI32 × 3253(32 × 32,256), (256,128), (128,64), (64,12), (12,3)II32 × 3254(32 × 32,256), (256,128), (128,64), (64,12), (12,4)III32 × 3255(32 × 32,256), (256,128), (128,64), (64,12), (12,5)IV32 × 3256(32 × 32,256), (256,128), (128,64), (64,12), (12,6)V64 × 6453(64 × 64,256), (256,128), (128,64), (64,12), (12,3)VI64 × 6454(64 × 64,256), (256,128), (128,64), (64,12), (12,4)VII64 × 6455(64 × 64,256), (256,128), (128,64), (64,12), (12,5)VIII64 × 6456(64 × 64,256), (256,128), (128,64), (64,12), (12,6)Using the sample datasetA2, the candidate models I–VIII were tested. The basic learning rate was 0.0001. The batch training size was 300, and the number of iterations was 100,000. According to the distribution of the training loss, Θ was set to 0.005. L¯0.005 and kD of each model were calculated. The results are shown in Table 3. Compared with other models, model III had the smallest value of L¯0.005 and the tangent slope kD. Therefore, given the performance of model III, it was selected as the optimal autoencoder model A∗.Table 3
L¯0.005 and slope kD of the candidate models.
ModelIIIIIIIVVVIVIIVIIIL¯0.005 (10–3)3.002.172.112.172.812.782.462.80kD (10–7)1.951.451.381.421.801.781.571.77


## 4.2.1. Model Training

The training parameters included the basic learning rate, batch training size, and the number of iterations. In the process of model training, the adaptive moment estimation (Adam) iterative optimization algorithm in [31] was used to dynamically adjust the learning rate of each parameter using the first-order moment estimation and second-order moment estimation. The equations are as follows:(12)χt=η×χt−1+1−η×gt,(13)ϑt=ρ×ϑt−1+1−ρ×gt2,(14)χ^t=χt1−ηt,(15)ϑ^t=ϑt1−ρt,(16)υt=υt−1−χ^tϑ^t+ε×β,where gt is the gradient of time step t; χt is the first-order moment estimation of, gt i.e., the exponential moving average of gt; ϑt is the second-order moment estimation of gt, i.e., the exponential moving average of gt; η and ρ are the exponential decay rates; χ^t is the deviation correction of χt; ϑ^t is the deviation correction of ϑt; υt is the parameter vector of time step t; the default learning rate is β=0.001; and ε is the residual term, ε=10−8.The Adam algorithm has a low requirement for memory during the computation. After deviation correction, each iterative learning rate was within a certain range, which results in stable parameters. Therefore, the Adam algorithm was suitable for training and optimization of the large datasets and high-dimensional data that were addressed in this study.


## 4.2.2. Evaluation of the Autoencoder

According to the loss distribution curve of the autoencoder, two indexes were used to measure its convergence and evaluate the performance of the autoencoder.(1)
A thresholdΘ was set for the loss. When the training loss of the autoencoder became less than Θ after performing iterations to step θ, the average loss L¯Θ between steps θ-D was calculated, as shown in Figure 5. The smaller the value of L¯Θ is, the smaller the difference between the original input data and the restored data of the autoencoder, and the better the performance of the autoencoder.(17)L¯Θ=1D−θ+1∑j=θj=DLj,whereD is the total number of training iterations, Lj is the loss of the j th iteration, j∈θ,D, and Θ is the loss threshold.(2)
The loss distribution curve of the autoencoder was fitted, and the tangent slope at the end of the fitting curve was calculated.(18)kD=y′D,wherey′ is the loss fitting curve function and y′D is the first-order derivative at the Dth iteration, i.e., the slope.As shown in Figure6, when kD was less than 0; the closer it is to 0, the gentler the loss curve, and the better the convergence effect.Under the same parameter settings and training conditions, asL¯Θ decreased, more effective feature data could be extracted and restored by the autoencoder; as kD became less than 0 and approached 0, the convergence effect of the autoencoder loss was better, which indicates high encoding-decoding stability and reliability.(3)
Structure optimization of the autoencoder. To design an autoencoder for video traffic state recognition, 36 sets of crossover tests were conducted based on the dimension of the input data, the dimension of the dimension-reduced data, and the number of hidden layers, as shown in Table1.Figure 5
Calculation diagram ofL¯Θ.Table 1
Design of crossover tests.
Input data dimensionDimension-reduced data dimensionNumber of hidden layersModel number.32 × 323–6, step length: 16–10, step length: 21–1264 × 643–6, step length: 16–10, step length: 213–24128 × 1283–6, step length: 16–10, step length: 225–36Figure 6
Calculation diagram ofkD.The 36 models were trained using the sample datasetA1 with the same training parameters. The basic learning rate was 0.0001. The batch training size was 100, and the number of iterations was 15,000.The distribution of the loss was analyzed, and the average loss whenΘ was 0.01 was calculated, as shown in Figure 7. The average losses of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th groups were at a lower level compared with other groups of tests. In these eight groups of tests, the input data dimension of the first four groups was 32 × 32 and that of the last four groups was 64 × 64. The number of hidden layers in each test was 10. Therefore, when the input data dimension decreased and the number of hidden layers increased, L¯0.01 decreased; the effect of the dimension of the dimension-reduced data on the loss was not significant. However, as the dimension of the dimension-reduced data increased, the model training loss tended to decrease. The eight groups of models were preliminarily selected as the candidate models.Figure 7
L¯0.01 distribution of crossover tests.Then, curve fitting was performed on the loss of the 36 groups of tests. The coefficients of determinationR2 of the fitting functions were all between 0.84 and 0.95. Only 13 groups of tests had R2 lower than 0.9. Therefore, the goodness of fit was high on the whole. The kD of the loss fitting function of each group of tests was calculated, as shown in Figure 8. In general, the tangent slopes of the loss fitting curves of each group of tests were all less than 0 and close to 0. The curve convergence effect was good, and the tangent slopes of the eight candidate models were all small. Therefore, the autoencoder models of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, and 24th groups of tests were selected as the candidate models and numbered as model I–model VIII. The structure is shown in Table 2.Figure 8
Tangent slopekD distribution of loss curve.Table 2
Structural parameters of the eight candidate autoencoder models.
Model numberInput data dimensionNumber of autoencoder hidden layersDimension of the dimension-reduced dataAutoencoder structureI32 × 3253(32 × 32,256), (256,128), (128,64), (64,12), (12,3)II32 × 3254(32 × 32,256), (256,128), (128,64), (64,12), (12,4)III32 × 3255(32 × 32,256), (256,128), (128,64), (64,12), (12,5)IV32 × 3256(32 × 32,256), (256,128), (128,64), (64,12), (12,6)V64 × 6453(64 × 64,256), (256,128), (128,64), (64,12), (12,3)VI64 × 6454(64 × 64,256), (256,128), (128,64), (64,12), (12,4)VII64 × 6455(64 × 64,256), (256,128), (128,64), (64,12), (12,5)VIII64 × 6456(64 × 64,256), (256,128), (128,64), (64,12), (12,6)Using the sample datasetA2, the candidate models I–VIII were tested. The basic learning rate was 0.0001. The batch training size was 300, and the number of iterations was 100,000. According to the distribution of the training loss, Θ was set to 0.005. L¯0.005 and kD of each model were calculated. The results are shown in Table 3. Compared with other models, model III had the smallest value of L¯0.005 and the tangent slope kD. Therefore, given the performance of model III, it was selected as the optimal autoencoder model A∗.Table 3
L¯0.005 and slope kD of the candidate models.
ModelIIIIIIIVVVIVIIVIIIL¯0.005 (10–3)3.002.172.112.172.812.782.462.80kD (10–7)1.951.451.381.421.801.781.571.77


## 5. Result Analysis and Discussion

### 5.1. Traffic State Recognition and Evaluation Methods

We assume that there wereC types of samples (i.e., traffic states) in the traffic video image dataset. For an arbitrary image sample, its traffic state can be accurately recognized or can be misjudged as other traffic states. Therefore, in the recognition results of certain types of traffic states, there might be errors and omissions. To effectively evaluate the traffic state recognition model, the accuracy rate and recall rate were used to measure the error and omission situations. Furthermore, a comprehensive evaluation index F1 was proposed. The larger the value of F1 is, the better the performance of the model. The principle and calculation process of the three indexes are as follows:(1)
The recognition accuracy rate, recall rate, andF1 value of each traffic state in the image datasets were calculated as follows:(19)Pc=TcSc,Rc=TcQc,F1c=2⋅Pc⋅RcPc+RcHere,C is the total number of traffic states in the traffic image dataset. In this study, C=3 (i.e., free, steady, and congested traffic flow); c is the category of the traffic state, c∈1,C; Pc is the accuracy rate of the model on the cth type of sample in the image dataset; Rc is the recall rate of the model on the cth type of sample in the image dataset; F1c is the F1 value of the model on the cth type of samples in the image dataset; Tc is the number of samples of the cth type correctly recognized using the model; Sc is the number of samples of the cth type obtained using the model; and Qc represents the actual number of samples of the cth type in the image dataset.(2)
The average accuracy rate, recall rate, andF1 value of C-type traffic state all were calculated.(20)P¯=∑c=1CPcC,R¯=∑c=1CRcC,F¯1=2⋅P¯⋅R¯P¯+R¯.where P¯ is the average accuracy rate of the model on the C-type traffic state in the image dataset, R¯ is the in the average recall rate of the model on the C-type traffic state in the image dataset, and F1 is the average F1 value of the model on the C-type traffic state in the image dataset.


### 5.2.A∗-Classifier Recognition Results

The traffic images were encoded using the autoencoderA∗, from which the dimension-reduced data were obtained and taken as input to the SVM, DNN, linear classifier, and DNN Linear classifier for traffic state recognition. In this study, the models that integrated A∗ with the SVM, DNN, linear classifier, and DNN Linear classifier were named A∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear, respectively. Traffic state recognition was conducted on dataset A2. The average accuracy rate P¯, recall rate R¯, and F1 value all were calculated, as shown in Table 4.Table 4
Average values of the evaluation indexes of the A∗ classifier for each of the traffic states in dataset A2 (%).
ModelP¯R¯F¯1A∗-SVM96.295.896.0A∗-DNN96.296.196.2A∗-Linear94.594.594.4A∗-DNN_ Linear97.197.197.1TheP¯ and R¯ values of the four models were all in the range of 94.5%–97.1%, and the average F1 value was 94.4%–97.1%, which indicate that the integration of the A∗ autoencoder with the classification methods produced excellent traffic state recognition results. Thus, A∗ can effectively extract traffic image features to achieve traffic state recognition.To further analyze the recognition results of the model, the accuracy rate, recall rate, andF1 value of each traffic state all were calculated, and the results are shown in Figure 9. The accuracy rate, recall rate, and F1 value varied among the four models. The accuracy rates ranged between 89.0% and 100%. Only the accuracy rate of A∗-SVM for the steady traffic flow and that of A∗-Linear for the congested traffic flow were below 90%; the recall rates were 86.4%–100%, and specifically, the recall rate of A∗-SVM for congested traffic flow and that of A∗-Linear for steady traffic flow were below 90%; the F1 values ranged between 91.1% and 100%, which indicates that the overall performances of the four models were all good.Figure 9
Classification results for each type of traffic state.From the perspective of traffic states, the accuracy rate, recall rate, andF1 value for free traffic flow were the best in all of the models, compared with steady and congested traffic states. The values were close to 100%, which indicates that almost all free traffic flow images were correctly recognized. The accuracy rate and recall rate for steady and congested traffic states were mostly higher than 90%, and the accuracy rate of A∗-SVM for steady traffic flow and the recall rate of A∗-SVM for congested traffic flow, as well as the accuracy rate of A∗-Linear for congested traffic flow and the recall rate of A∗-Linear for steady traffic flow, were below 90%. Figures 10–12 show the recognition results of the four models for the three traffic states in dataset A2. The autoencoder A∗ showed a good effect on the image dimensionality reduction. Integration of the autoencoder with the SVM, DNN, linear classifier, and DNN Linear classifier achieved overall good recognition results for the different traffic states.Figure 10
Recognition ofA∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear for free traffic flow (F1  = 100%, 100%, 99.1%, 100%).
(a)(b)(c)(d)Figure 11
Recognition ofA∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear for steady traffic flow (F1  = 94.1%, 94.3%, 91.1%, 95.6%).
(a)(b)(c)(d)Figure 12
Recognition ofA∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear for congested traffic flow (F1  = 93.4%, 94.5%, 92.6%, 95.7%).
(a)(b)(c)(d)


### 5.3.A∗-K-Means Clustering Results

The autoencoderA∗ was integrated with the k-means clustering method, to make a model that is referred to as A∗-k-means, which was trained using dataset A2. The clustering accuracy rate, recall rate, and F1 value for the different traffic states were calculated, and the results of which are shown in Table 5. The clustering accuracy rate, recall rate, and F1 value of the A∗-k-means model all were between 90.2% and 100%, which suggests good recognition performance for the different traffic states. Moreover, the accuracy rate, recall rate, and F1 value for the steady traffic flow were all 100%; in other words, all of the samples were recognized accurately. The recognition results for the free and congested traffic flow were almost the same, and the F1 values were approximately 93%. Thus, the optimal autoencoder A∗ constructed in this study can achieve a good traffic state recognition effect when combined with common clustering methods, which indicates that A∗ can effectively extract the traffic image features.Table 5
Evaluation indexes of A∗-k-means for each traffic state in dataset A2 (%).
Traffic stateAccuracy rateRecall rateF1 valueFree90.795.693.1Steady100100100Congested95.390.292.7Average95.495.395.3


### 5.4. Comparison between the Proposed Model and CNN Model

The CNN is a typical deep learning method. Common CNN models include AlexNet, VGG16, GoogLeNet, and LeNet in [32], which have been widely applied in the field of image pattern recognition and have achieved remarkable results. In this study, we used dataset A2 to train and test the four CNN models, and we calculated the accuracy rate, recall rate, and F1 value, and the results of which are shown in Table 6.Table 6
Test results of the CNN models and the models proposed in this paper.
ModelModel structureAccuracy rate (%)Recall rate (%)F1 (%)AlexNet5 convolutional layers, 3 pooling layers, 3 fully connected layers, and 1 classification layer94.593.694.0LeNet3 convolutional layers, 2 pooling layers, 1 fully connected layer, and 1 classification layer82.362.471.0GoogLeNet22 network layers, including convolutional layers and pooling layers, and 1 classification layer36.835.236.0VGG1613 convolutional layers, 3 fully connected layers, 5 pooling layers, and 1 classification layer11.133.316.7A∗ Classifier5 encoding hidden layers + common classifiers (linear classification, SVM, DNN, and so on)94.5–97.194.5–97.194.4–97.1A∗-k-means5 encoding hidden layers + k-means clustering95.495.395.3Among the four CNN models, AlexNet had the optimal performance; its accuracy rate, recall rate, andF1 value all reached approximately 94%. The F1 values of the proposed A∗ classifier and A∗-k-means clustering model were 94.4%–97.1%, which was higher than that of AlexNet. The performance of LeNet in traffic state recognition was average, with an accuracy rate, recall rate, and F1 value of 82.3%, 62.4%, and 71.0%, respectively. Moreover, the F1 values of GoogLeNet and VGG16 were all below 40%, which suggests poor performance in traffic state recognition.From the perspective of the model network structure,A∗ used only five encoding hidden layers to obtain the image feature data, whereas AlexNet, LeNet, GoogLeNet, and VGG16 had 12, 9, 23, and 22 network layers, respectively. The complexity of A∗ was much lower than those models. Therefore, by integrating a simple and practicable autoencoder with common classifiers, the models proposed in this study were able to achieve or surpass the traffic state recognition effect of complex CNN networks. Compared with the CNN, the method proposed in this study has the advantages of a lightweight model structure and a low calculation cost, which outperforms CNN models such as AlexNet and GoogLeNet. Therefore, it is suitable for traffic state recognition from videos.


## 5.1. Traffic State Recognition and Evaluation Methods

We assume that there wereC types of samples (i.e., traffic states) in the traffic video image dataset. For an arbitrary image sample, its traffic state can be accurately recognized or can be misjudged as other traffic states. Therefore, in the recognition results of certain types of traffic states, there might be errors and omissions. To effectively evaluate the traffic state recognition model, the accuracy rate and recall rate were used to measure the error and omission situations. Furthermore, a comprehensive evaluation index F1 was proposed. The larger the value of F1 is, the better the performance of the model. The principle and calculation process of the three indexes are as follows:(1)
The recognition accuracy rate, recall rate, andF1 value of each traffic state in the image datasets were calculated as follows:(19)Pc=TcSc,Rc=TcQc,F1c=2⋅Pc⋅RcPc+RcHere,C is the total number of traffic states in the traffic image dataset. In this study, C=3 (i.e., free, steady, and congested traffic flow); c is the category of the traffic state, c∈1,C; Pc is the accuracy rate of the model on the cth type of sample in the image dataset; Rc is the recall rate of the model on the cth type of sample in the image dataset; F1c is the F1 value of the model on the cth type of samples in the image dataset; Tc is the number of samples of the cth type correctly recognized using the model; Sc is the number of samples of the cth type obtained using the model; and Qc represents the actual number of samples of the cth type in the image dataset.(2)
The average accuracy rate, recall rate, andF1 value of C-type traffic state all were calculated.(20)P¯=∑c=1CPcC,R¯=∑c=1CRcC,F¯1=2⋅P¯⋅R¯P¯+R¯.where P¯ is the average accuracy rate of the model on the C-type traffic state in the image dataset, R¯ is the in the average recall rate of the model on the C-type traffic state in the image dataset, and F1 is the average F1 value of the model on the C-type traffic state in the image dataset.


## 5.2.A∗-Classifier Recognition Results

The traffic images were encoded using the autoencoderA∗, from which the dimension-reduced data were obtained and taken as input to the SVM, DNN, linear classifier, and DNN Linear classifier for traffic state recognition. In this study, the models that integrated A∗ with the SVM, DNN, linear classifier, and DNN Linear classifier were named A∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear, respectively. Traffic state recognition was conducted on dataset A2. The average accuracy rate P¯, recall rate R¯, and F1 value all were calculated, as shown in Table 4.Table 4
Average values of the evaluation indexes of the A∗ classifier for each of the traffic states in dataset A2 (%).
ModelP¯R¯F¯1A∗-SVM96.295.896.0A∗-DNN96.296.196.2A∗-Linear94.594.594.4A∗-DNN_ Linear97.197.197.1TheP¯ and R¯ values of the four models were all in the range of 94.5%–97.1%, and the average F1 value was 94.4%–97.1%, which indicate that the integration of the A∗ autoencoder with the classification methods produced excellent traffic state recognition results. Thus, A∗ can effectively extract traffic image features to achieve traffic state recognition.To further analyze the recognition results of the model, the accuracy rate, recall rate, andF1 value of each traffic state all were calculated, and the results are shown in Figure 9. The accuracy rate, recall rate, and F1 value varied among the four models. The accuracy rates ranged between 89.0% and 100%. Only the accuracy rate of A∗-SVM for the steady traffic flow and that of A∗-Linear for the congested traffic flow were below 90%; the recall rates were 86.4%–100%, and specifically, the recall rate of A∗-SVM for congested traffic flow and that of A∗-Linear for steady traffic flow were below 90%; the F1 values ranged between 91.1% and 100%, which indicates that the overall performances of the four models were all good.Figure 9
Classification results for each type of traffic state.From the perspective of traffic states, the accuracy rate, recall rate, andF1 value for free traffic flow were the best in all of the models, compared with steady and congested traffic states. The values were close to 100%, which indicates that almost all free traffic flow images were correctly recognized. The accuracy rate and recall rate for steady and congested traffic states were mostly higher than 90%, and the accuracy rate of A∗-SVM for steady traffic flow and the recall rate of A∗-SVM for congested traffic flow, as well as the accuracy rate of A∗-Linear for congested traffic flow and the recall rate of A∗-Linear for steady traffic flow, were below 90%. Figures 10–12 show the recognition results of the four models for the three traffic states in dataset A2. The autoencoder A∗ showed a good effect on the image dimensionality reduction. Integration of the autoencoder with the SVM, DNN, linear classifier, and DNN Linear classifier achieved overall good recognition results for the different traffic states.Figure 10
Recognition ofA∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear for free traffic flow (F1  = 100%, 100%, 99.1%, 100%).
(a)(b)(c)(d)Figure 11
Recognition ofA∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear for steady traffic flow (F1  = 94.1%, 94.3%, 91.1%, 95.6%).
(a)(b)(c)(d)Figure 12
Recognition ofA∗-SVM, A∗-DNN, A∗-Linear, and A∗-DNN_Linear for congested traffic flow (F1  = 93.4%, 94.5%, 92.6%, 95.7%).
(a)(b)(c)(d)


## 5.3.A∗-K-Means Clustering Results

The autoencoderA∗ was integrated with the k-means clustering method, to make a model that is referred to as A∗-k-means, which was trained using dataset A2. The clustering accuracy rate, recall rate, and F1 value for the different traffic states were calculated, and the results of which are shown in Table 5. The clustering accuracy rate, recall rate, and F1 value of the A∗-k-means model all were between 90.2% and 100%, which suggests good recognition performance for the different traffic states. Moreover, the accuracy rate, recall rate, and F1 value for the steady traffic flow were all 100%; in other words, all of the samples were recognized accurately. The recognition results for the free and congested traffic flow were almost the same, and the F1 values were approximately 93%. Thus, the optimal autoencoder A∗ constructed in this study can achieve a good traffic state recognition effect when combined with common clustering methods, which indicates that A∗ can effectively extract the traffic image features.Table 5
Evaluation indexes of A∗-k-means for each traffic state in dataset A2 (%).
Traffic stateAccuracy rateRecall rateF1 valueFree90.795.693.1Steady100100100Congested95.390.292.7Average95.495.395.3


## 5.4. Comparison between the Proposed Model and CNN Model

The CNN is a typical deep learning method. Common CNN models include AlexNet, VGG16, GoogLeNet, and LeNet in [32], which have been widely applied in the field of image pattern recognition and have achieved remarkable results. In this study, we used dataset A2 to train and test the four CNN models, and we calculated the accuracy rate, recall rate, and F1 value, and the results of which are shown in Table 6.Table 6
Test results of the CNN models and the models proposed in this paper.
ModelModel structureAccuracy rate (%)Recall rate (%)F1 (%)AlexNet5 convolutional layers, 3 pooling layers, 3 fully connected layers, and 1 classification layer94.593.694.0LeNet3 convolutional layers, 2 pooling layers, 1 fully connected layer, and 1 classification layer82.362.471.0GoogLeNet22 network layers, including convolutional layers and pooling layers, and 1 classification layer36.835.236.0VGG1613 convolutional layers, 3 fully connected layers, 5 pooling layers, and 1 classification layer11.133.316.7A∗ Classifier5 encoding hidden layers + common classifiers (linear classification, SVM, DNN, and so on)94.5–97.194.5–97.194.4–97.1A∗-k-means5 encoding hidden layers + k-means clustering95.495.395.3Among the four CNN models, AlexNet had the optimal performance; its accuracy rate, recall rate, andF1 value all reached approximately 94%. The F1 values of the proposed A∗ classifier and A∗-k-means clustering model were 94.4%–97.1%, which was higher than that of AlexNet. The performance of LeNet in traffic state recognition was average, with an accuracy rate, recall rate, and F1 value of 82.3%, 62.4%, and 71.0%, respectively. Moreover, the F1 values of GoogLeNet and VGG16 were all below 40%, which suggests poor performance in traffic state recognition.From the perspective of the model network structure,A∗ used only five encoding hidden layers to obtain the image feature data, whereas AlexNet, LeNet, GoogLeNet, and VGG16 had 12, 9, 23, and 22 network layers, respectively. The complexity of A∗ was much lower than those models. Therefore, by integrating a simple and practicable autoencoder with common classifiers, the models proposed in this study were able to achieve or surpass the traffic state recognition effect of complex CNN networks. Compared with the CNN, the method proposed in this study has the advantages of a lightweight model structure and a low calculation cost, which outperforms CNN models such as AlexNet and GoogLeNet. Therefore, it is suitable for traffic state recognition from videos.


## 6. Conclusions

In this study, an autoencoder model for urban traffic surveillance videos was proposed that can effectively reduce the data dimension by optimizing the input data dimension, number of hidden layers, and dimension of the dimension-reduced data. Taking the low-dimensional image features obtained using the autoencoder as the input, five models were constructed based on common classification methods, including the linear classifier, DNN, SVM, DNN Linear, and k-means clustering method. The performances of the models in traffic state recognition were compared with those of commonly used CNN models. The results show that the averageF1 value of the four models A∗-DNN, A∗-SVM, A∗-Linear, and A∗-DNN Linear was 94.4%–97.1%, and the average F1 value of A∗-k-means was 95.3%. Among the CNN models, AlexNet has the best performance, with an F1 value of 94.0%. Thus, the autoencoder constructed in this study can effectively extract the image features. When integrated with common classification and clustering methods, it can accurately recognize the traffic state, and it achieves better results than common CNN models, such as AlexNet, LeNet, GoogLeNet, and VGG16.The traffic state recognition model was established in two stages to avoid the problems of high algorithm complexity and calculation cost. First, an autoencoder was proposed which effectively reduces video dimension; then, the traffic state recognition model was established by integrating the autoencoder with common classification and clustering methods. The train and test result presents that our method has advantages of a lightweight model structure and a low calculation cost, which outperforms CNN models such as AlexNet and GoogLeNet. The method can also be applied in other fields of video detection, such as image compression, moving target detection, and image classification.Due to the variability and complexity of traffic scenes and the difficulty in building large-scale traffic image datasets, we conducted an exploratory study on the construction and optimization of an autoencoder model using limited traffic scenes and sample size. In the future, we will focus on the train and test dataset construction, autoencoder model architecture and parameter optimization, and classification method selection.



---

*Source: 2850111-2022-03-28.xml*