# Expressway Traffic Flow Missing Data Repair Method Based on Coupled Matrix-Tensor Factorizations

Authors: Hui Jiang; Hongxing Deng
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Traffic flow data is the basis of traffic management, planning, control, and other forms of implementation. Once missing, it will directly affect the monitoring and prediction of expressway traffic status. Regarding this, this paper proposes a repair method for the traffic flow missing data of expressway, combined with the idea of coupled matrix-tensor factorizations (CMTF), to couple the auxiliary traffic flow data into the main traffic flow data and to construct the coupling matrix-tensor expression of traffic flow data, and the alternating direction multiplier algorithm is used to realize the repair of missing traffic flow data. Combined with the measured data of expressway traffic flow, the experimental results show that, under different missing data types and missing rates, the proposed method outperforms the methods lacking auxiliary traffic flow data and achieves a good repair effect, especially for high miss data rates.


---

## Body

## 1. Introduction

The traffic management department requires traffic flow data to be complete to realize accurate assessment and prediction of the traffic status. Thus, the repair of the missing data is crucial for effective management of the traffic flow.In the past few decades, researchers have shown much interest in the problem of repairing the missing traffic flow data. The existing research on this area can be categorized into three types of data composition: vector, matrix, and tensor. Below, we elaborate on the existing studies to provide an overview of the state of the art.First of all, researchers used the traffic flow data in the form of a one-dimensional vector to repair the missing parts based on spatial or temporal correlation. Gold et al. classified the reasons for the missing data based on the characteristics of expressway traffic flow data and repaired the missing data using nuclear regression, polynomial regression, and linear interpolation [1]. Based on the hotspot analysis of the spatial correlation of traffic data, Conklin et al. used the similarity of the adjacent lanes to repair the missing data [2]. Smith et al. utilized the temporal correlation of traffic data and repaired the missing parts by using the average value of the historical data at the corresponding time and the exponential smoothing value of data at adjacent periods from several days ago [3]. Based on the spatial and temporal correlation, Jiang and Gang predicted the value of data in previous periods and adjacent sections to repair the missing data [4]. Jin and Wang proposed a traffic flow parameter correction method based on correlation analysis from the perspective of statistical analysis of traffic flow data [5]. Gheyas and Smith mainly focused on the measured traffic flow data and utilized the methods of historical trend and exponential smoothing to repair the missing parts. By comparison, these two methods were proved to be more effective in dealing with continuous abnormal data [6]. In consideration of the spatial-temporal characteristics of the traffic flow data, Wang et al. proposed a repair method based on the 3D shape function and the spatial-temporal interpolation [7]. Chen et al. proposed a long short-term memory (LSTM) method, which was built on the normal time-series using the calculation error of the Gaussian Bayesian model. Experimental results on three real datasets showed the advantages of this method [8].Then, the data is organized into a matrix to repair the missing data. Kurucz et al. proposed a low-rank matrix completion algorithm using the path norm minimization of the matrix to replace the minimization of the rank function [9]. Qu et al. performed (Bayesian) principal component analysis to reconstruct the traffic flow data into a matrix. The results showed the effectiveness of this method in data repair for low missing rates [10]. Tang et al. made use of fuzzy c-means clustering and genetic algorithm to repair the missing data based on a matrix. The repair effect of data on different time scales was quantified. The results revealed that the fusion model had a significant repair effect [11]. Lu et al. proposed a data repair model structured on an improved multiscale principal component analysis. Based on the spatial-temporal correlation characteristics of the traffic flow data, they calculated the correlation coefficient of the missing data to estimate its true value [12]. Similarly, Jiang proposed a data repair algorithm using the fusion matrix low-rank decomposition [13].The data composition in the form of multidimensional tensor expands the vector and matrix to higher dimensions. In recent years, it has proliferated in machine learning (clustering and dimensionality reduction) and other fields [14]. This method can repair the missing data by organizing the traffic flow data into tensors. Acar et al. proposed an algorithm based on CP decomposition and gradient optimization and then proved that the algorithm had high flexibility [15]. Silva and Herrmann proposed a tensor repair method based on the hierarchical Tucker decomposition to repair the missing seismic data [16]. Tan et al. proposed a traffic data repair method based on tensor for the first time, which achieved good experimental results, especially in the case of high data missing rates. Subsequently, Tan et al. proposed several other tensor-based repair methods, which proved that the tensor-based method was still useful, especially for the cases with high missed data [17]. Liu et al. proposed a low-rank tensor repair algorithm to solve the problem of image repair and proved that their method was better than the tensor model using CP decomposition [18]. Based on the tensor theory, Xiang used the spatial-temporal characteristics of the traffic data for modelling, which significantly improved the accuracy of data repair [19]. Zhang focused on the microwave data of the urban expressway and repaired the data using a CP-ALS algorithm based on tensor decomposition. The results proved that their method could effectively improve data quality [20]. Zhang et al. proposed an iterative tensor decomposition (ITD) method to repair the lost data in the ANPR system. They used the multidimensional intrinsic correlation of the traffic flow data for the detection of missed parts and their repair. The results showed that this method could accurately detect and repair the missing data under various missing rates [21].To summarize, the one-dimensional vector was highly dependent on the historical information, requiring stable traffic flow data within a period or space. Thus, it could not use the information collected after the data missed. Besides, if the data missed within a long period, the error in the repair results turned to be high. On the other hand, although the two-dimensional matrix made full use of the spatial-temporal correlation characteristics of the traffic flow data, it was limited to the 2D data. Furthermore, it could not simultaneously utilize the multimode correlation of the traffic flow data, which resulted in low repair accuracy. In recent years, the high-dimensional tensor model has attracted significant attention in data repair, and some progress has been achieved, especially for the repair of traffic flow missing data. However, most of the existing efforts have focused on the model construction of spatial and temporal correlation of the traffic flow data.In summary, it is necessary to introduce a data repair method, which makes full, reasonable, and effective use of multidimensional information between the traffic flow data. To do so, this work for the first time proposes a traffic flow missing data repair model based on the coupled matrix-tensor factorizations. Compared with the existing work, we make two contributions: (1) we aim to obtain better multidimensional information of traffic flow data, which is divided into main traffic flow data and auxiliary traffic flow data, and coupled the auxiliary traffic flow data into the main traffic flow data to construct the coupling matrix-tensor expression of traffic flow data. (2) In order to get better data repair effect, we used alternating direction multiplier (ADMM) algorithm to decompose large global problems into many smaller and easier local subproblems to solve the optimal solution. By comparison with the prior method, we validate the effectiveness and feasibility of the proposed methodology in data repair.The rest of this paper is organized as follows. Firstly, the tensor theory is explained in Section2. Then, the traffic flow data analysis is provided in Section 3. Section 4 presents the missing data repair method based on CMTF, where the results and discussion on the findings are elaborated in Section 5. Finally, Section 6 concludes the paper.


## 2. Tensor Theory

As an extension of vector and matrix to a high-dimensional space, tensor is a mathematical representation of a multidimensional data. The dimension of a tensor is also called the modulus or the order [22, 23]. Hence, if we consider a vector as a tensor of the first order and a matrix as a tensor of the second order, it refers to an array of three or more dimensions as a tensor of a high order, tensor for short. The third-order tensor χ∈ℝI×J×K with dimensions I, J, and K shown in Figure 1 is an example of this.Figure 1
The third-order tensorχ∈ℝI×J×K.Tensors can be expressed in either a fibrous or a slice form [24, 25]. Any two dimensions in the third-order tensor are kept unchanged. In this case, the tensor can be expressed by a column fiber line x:jk, a fiber xi:k, and a pipe xij:, where the colon represents the dimension all of elements. Alternatively, the tensor can be expressed in the form of a horizontal slice Xi∷, a side slice X:j:, and a front slice X::k when the other two dimensions are changed.


### 2.1. Tensor Matrixing

Tensor matrixing is a process, in which the elements of a tensor are rearranged into matrices. To be specific, tensor operations are simplified by expanding tensorχ into the matrix Xn in the n modulus.The CP decomposition is one of the earliest and most in-depth tensor decomposition methods. It can be interpreted as decomposing a tensor into the form of a sum of rank-1 tensors. Forχ∈ℝI×J×K, the CP decomposition can be expressed in the following matrix form:(1)χ≈A,B,C=∑r=1Rar∘br∘cr=A∘B∘C,where A=a1,a2,…,ar is the vector ar corresponding to rank-1 tensor. Similarly, B and C are the vectors br and cr corresponding to rank-1 tensor. Thus, A, B, and C are the factor matrices decomposed by CP decomposition, where A∈ℝI×R, B∈ℝJ×R, and C∈ℝK×R. The slice of a tensor can be represented as follows:(2)X1≈AC⊙BT,X2≈BC⊙AT,X3≈CB⊙AT,where the symbol “⊙” can be specifically described as the output value of 1 when two input variables are the same, otherwise the output value is.In practice, however, the column vectors in the factor matrix are usually normalized:arF=brF=crF=1. Thus, the CP decomposition can be expressed as(3)χ≈λ;A,B,C=∑r=1Rλrar∘br∘cr,where λ is the regularization parameter.


### 2.2. The Coupled Matrix-Tensor Factorizations

The joint analysis of data is quite useful in understanding the underlying structure of complex datasets. Here, the coupled matrix-tensor factorizations (CMTF) can capture the potential structural characteristics of heterogeneous data by coupling the matrix data into the high-order tensor model.To explain this coupling model in detail, we assume that a third-order tensorχ∈ℝI×J×K and a matrix U∈ℝI×M are derived from different datasets, and U is coupled to a dimension of χ. The schematic diagram of the CMTF is shown in Figure 2.Figure 2
The CMTF model.The underlying structure of the dataset can be extracted by the CMTF, where the R-component of a tensorχ and a matrix U is defined as(4)fA,B,C,V=12χ−A,B,C2+12U−AVT2,where A,B,C are the factor matrices of the tensor χ extracted by CP decomposition, where A∈ℝI×R, B∈ℝJ×R, and C∈ℝK×R.The factor matricesA∈ℝM×Rand V∈ℝM×R are extracted from matrix U, which can be represented as(5)U=AVT,where A is the common mode factor matrix of the tensor χ and the matrix U on a certain dimension (modulus).To ensure that the datasets are codecomposed and share the same potential space, the matrixU should have the same underlying low-rank structure as the tensor χ on at least one dimension.


## 2.1. Tensor Matrixing

Tensor matrixing is a process, in which the elements of a tensor are rearranged into matrices. To be specific, tensor operations are simplified by expanding tensorχ into the matrix Xn in the n modulus.The CP decomposition is one of the earliest and most in-depth tensor decomposition methods. It can be interpreted as decomposing a tensor into the form of a sum of rank-1 tensors. Forχ∈ℝI×J×K, the CP decomposition can be expressed in the following matrix form:(1)χ≈A,B,C=∑r=1Rar∘br∘cr=A∘B∘C,where A=a1,a2,…,ar is the vector ar corresponding to rank-1 tensor. Similarly, B and C are the vectors br and cr corresponding to rank-1 tensor. Thus, A, B, and C are the factor matrices decomposed by CP decomposition, where A∈ℝI×R, B∈ℝJ×R, and C∈ℝK×R. The slice of a tensor can be represented as follows:(2)X1≈AC⊙BT,X2≈BC⊙AT,X3≈CB⊙AT,where the symbol “⊙” can be specifically described as the output value of 1 when two input variables are the same, otherwise the output value is.In practice, however, the column vectors in the factor matrix are usually normalized:arF=brF=crF=1. Thus, the CP decomposition can be expressed as(3)χ≈λ;A,B,C=∑r=1Rλrar∘br∘cr,where λ is the regularization parameter.


## 2.2. The Coupled Matrix-Tensor Factorizations

The joint analysis of data is quite useful in understanding the underlying structure of complex datasets. Here, the coupled matrix-tensor factorizations (CMTF) can capture the potential structural characteristics of heterogeneous data by coupling the matrix data into the high-order tensor model.To explain this coupling model in detail, we assume that a third-order tensorχ∈ℝI×J×K and a matrix U∈ℝI×M are derived from different datasets, and U is coupled to a dimension of χ. The schematic diagram of the CMTF is shown in Figure 2.Figure 2
The CMTF model.The underlying structure of the dataset can be extracted by the CMTF, where the R-component of a tensorχ and a matrix U is defined as(4)fA,B,C,V=12χ−A,B,C2+12U−AVT2,where A,B,C are the factor matrices of the tensor χ extracted by CP decomposition, where A∈ℝI×R, B∈ℝJ×R, and C∈ℝK×R.The factor matricesA∈ℝM×Rand V∈ℝM×R are extracted from matrix U, which can be represented as(5)U=AVT,where A is the common mode factor matrix of the tensor χ and the matrix U on a certain dimension (modulus).To ensure that the datasets are codecomposed and share the same potential space, the matrixU should have the same underlying low-rank structure as the tensor χ on at least one dimension.


## 3. Analysis of the Traffic Flow Data

### 3.1. Traffic Flow Missing Data: Causes and Types

The complete traffic flow data is key for the traffic management department to assess the traffic status. However, due to the influence/restriction of traffic conditions, meteorological environment, and many other factors, the traffic system has formed a complex operating environment. There are four main reasons of the missing data: detection equipment failure, power failure, communication network failure, and environmental factors.The missing data types can be summarized as follows:(1)
Missing Completely at Random (MCAR). It refers to the type where the missing traffic flow data are completely random and independent of each other; that is, the probability of the missing data is not affected by other factors. The missing data location is represented as a random discrete distribution.(2)
Missing at Random (MAR). It refers to the type where the missing traffic flow data is not completely random. The missing data is not correlated with itself but with other adjacent data points. The missing data location is a random continuous distribution.(3)
Missing at Nonrandom (MANR). It refers to the type where there is a certain correlation between the missing data and its characteristics, which also depends on other missing data. This loss is generally caused by the system itself.One of the earlier studies in the field [26, 27] assumed that the MANR data was found and then deleted, where subsequent studies mainly used the MCAR and MAR. To comprehensively reflect the real traffic flow under the condition that the complexity of the data is missing, the follow-up work will be according to the three following case studies:(1)
The missing data type 100% are missing completely at random (MCAR)(2)
Lack of missing at random (MAR) data type 100%(3)
Lack of missing completely at random data type and random loss, 50% each, namely, lack of hybrid (MIX-M)


### 3.2. Analysis of the Traffic Flow Data

In analysis of traffic flow data variation, the traffic flow data is divided into main traffic flow data and auxiliary traffic flow data according to whether it can directly reflect the actual traffic status of the road.


#### 3.2.1. Analysis of the Main Traffic Flow Data

This paper defines the main traffic flow data as the data that can directly reflect the state change of the system. Specifically, it is described as the index value that can represent the state of the traffic flow through the data, such as flow, speed, and occupancy, which plays a leading role in the repair of the missing data.The phenomena of the main traffic flow data changes with time and space are called the spatial and temporal distribution of traffic flow, which has the characteristics of temporal and spatial correlation. The time correlation analysis of the traffic flow data describes the change characteristics in the time domain. The travellers determine the time regularity of the traffic flow. According to the traffic data of different times of a month, week, and day, the similarity of the different degrees is reflected. Considering the different time organization modes of traffic flow data, the time correlation analysis will be performed according to the correlation analysis of different weeks of a month, days of a week, and periods of a day. Some studies have revealed that the traffic flow data has strong daily and weekly correlations [3–5, 7, 16, 18, 28–31].The spatial correlation analysis of the traffic flow data includes two main parts: (i) the data correlation analysis of a lane and its transverse adjacent lanes and (ii) the data correlation analysis of a lane and its longitudinal upstream and downstream sections of the same lane. Some studies have proved that the flow data of an urban expressway has a strong correlation with its horizontal and vertical spaces. In theory, the spatial and temporal correlation of the traffic flow data can be used to effectively repair the missing data.


#### 3.2.2. Analysis of the Auxiliary Traffic Flow Data

The core of data repair is to establish a relationship between the known and missing data. The more information related to missing data is contained in the data repair model, the better the effect of data repair will be. The auxiliary traffic flow data refers to the fact that the traffic flow data itself does not have a direct response state of the quantitative indicators. However, in the process of data repair, it can provide certain information about road conditions and environmental characteristics. Thus, the acquisition and use of auxiliary data will greatly help to restore the missing data.The importance of auxiliary traffic flow data in data repair is explained as follows: Firstly, different weather conditions may affect the operational behaviours of vehicles, for example, in rainy weather, due to the slippery road surface, which will directly affect the speed of vehicles, thus leading to similar running state of vehicles in similar weather conditions. Secondly, the analysis of the running speed of vehicles in different lanes of the same section shows that the vehicles using the middle lane will travel at different speeds compared to the ones in the edge lanes [32, 33]. The difference in the information added for the traffic flow data repair helps to improve the restoration precision of the traffic flow data.


## 3.1. Traffic Flow Missing Data: Causes and Types

The complete traffic flow data is key for the traffic management department to assess the traffic status. However, due to the influence/restriction of traffic conditions, meteorological environment, and many other factors, the traffic system has formed a complex operating environment. There are four main reasons of the missing data: detection equipment failure, power failure, communication network failure, and environmental factors.The missing data types can be summarized as follows:(1)
Missing Completely at Random (MCAR). It refers to the type where the missing traffic flow data are completely random and independent of each other; that is, the probability of the missing data is not affected by other factors. The missing data location is represented as a random discrete distribution.(2)
Missing at Random (MAR). It refers to the type where the missing traffic flow data is not completely random. The missing data is not correlated with itself but with other adjacent data points. The missing data location is a random continuous distribution.(3)
Missing at Nonrandom (MANR). It refers to the type where there is a certain correlation between the missing data and its characteristics, which also depends on other missing data. This loss is generally caused by the system itself.One of the earlier studies in the field [26, 27] assumed that the MANR data was found and then deleted, where subsequent studies mainly used the MCAR and MAR. To comprehensively reflect the real traffic flow under the condition that the complexity of the data is missing, the follow-up work will be according to the three following case studies:(1)
The missing data type 100% are missing completely at random (MCAR)(2)
Lack of missing at random (MAR) data type 100%(3)
Lack of missing completely at random data type and random loss, 50% each, namely, lack of hybrid (MIX-M)


## 3.2. Analysis of the Traffic Flow Data

In analysis of traffic flow data variation, the traffic flow data is divided into main traffic flow data and auxiliary traffic flow data according to whether it can directly reflect the actual traffic status of the road.


### 3.2.1. Analysis of the Main Traffic Flow Data

This paper defines the main traffic flow data as the data that can directly reflect the state change of the system. Specifically, it is described as the index value that can represent the state of the traffic flow through the data, such as flow, speed, and occupancy, which plays a leading role in the repair of the missing data.The phenomena of the main traffic flow data changes with time and space are called the spatial and temporal distribution of traffic flow, which has the characteristics of temporal and spatial correlation. The time correlation analysis of the traffic flow data describes the change characteristics in the time domain. The travellers determine the time regularity of the traffic flow. According to the traffic data of different times of a month, week, and day, the similarity of the different degrees is reflected. Considering the different time organization modes of traffic flow data, the time correlation analysis will be performed according to the correlation analysis of different weeks of a month, days of a week, and periods of a day. Some studies have revealed that the traffic flow data has strong daily and weekly correlations [3–5, 7, 16, 18, 28–31].The spatial correlation analysis of the traffic flow data includes two main parts: (i) the data correlation analysis of a lane and its transverse adjacent lanes and (ii) the data correlation analysis of a lane and its longitudinal upstream and downstream sections of the same lane. Some studies have proved that the flow data of an urban expressway has a strong correlation with its horizontal and vertical spaces. In theory, the spatial and temporal correlation of the traffic flow data can be used to effectively repair the missing data.


### 3.2.2. Analysis of the Auxiliary Traffic Flow Data

The core of data repair is to establish a relationship between the known and missing data. The more information related to missing data is contained in the data repair model, the better the effect of data repair will be. The auxiliary traffic flow data refers to the fact that the traffic flow data itself does not have a direct response state of the quantitative indicators. However, in the process of data repair, it can provide certain information about road conditions and environmental characteristics. Thus, the acquisition and use of auxiliary data will greatly help to restore the missing data.The importance of auxiliary traffic flow data in data repair is explained as follows: Firstly, different weather conditions may affect the operational behaviours of vehicles, for example, in rainy weather, due to the slippery road surface, which will directly affect the speed of vehicles, thus leading to similar running state of vehicles in similar weather conditions. Secondly, the analysis of the running speed of vehicles in different lanes of the same section shows that the vehicles using the middle lane will travel at different speeds compared to the ones in the edge lanes [32, 33]. The difference in the information added for the traffic flow data repair helps to improve the restoration precision of the traffic flow data.


## 3.2.1. Analysis of the Main Traffic Flow Data

This paper defines the main traffic flow data as the data that can directly reflect the state change of the system. Specifically, it is described as the index value that can represent the state of the traffic flow through the data, such as flow, speed, and occupancy, which plays a leading role in the repair of the missing data.The phenomena of the main traffic flow data changes with time and space are called the spatial and temporal distribution of traffic flow, which has the characteristics of temporal and spatial correlation. The time correlation analysis of the traffic flow data describes the change characteristics in the time domain. The travellers determine the time regularity of the traffic flow. According to the traffic data of different times of a month, week, and day, the similarity of the different degrees is reflected. Considering the different time organization modes of traffic flow data, the time correlation analysis will be performed according to the correlation analysis of different weeks of a month, days of a week, and periods of a day. Some studies have revealed that the traffic flow data has strong daily and weekly correlations [3–5, 7, 16, 18, 28–31].The spatial correlation analysis of the traffic flow data includes two main parts: (i) the data correlation analysis of a lane and its transverse adjacent lanes and (ii) the data correlation analysis of a lane and its longitudinal upstream and downstream sections of the same lane. Some studies have proved that the flow data of an urban expressway has a strong correlation with its horizontal and vertical spaces. In theory, the spatial and temporal correlation of the traffic flow data can be used to effectively repair the missing data.


## 3.2.2. Analysis of the Auxiliary Traffic Flow Data

The core of data repair is to establish a relationship between the known and missing data. The more information related to missing data is contained in the data repair model, the better the effect of data repair will be. The auxiliary traffic flow data refers to the fact that the traffic flow data itself does not have a direct response state of the quantitative indicators. However, in the process of data repair, it can provide certain information about road conditions and environmental characteristics. Thus, the acquisition and use of auxiliary data will greatly help to restore the missing data.The importance of auxiliary traffic flow data in data repair is explained as follows: Firstly, different weather conditions may affect the operational behaviours of vehicles, for example, in rainy weather, due to the slippery road surface, which will directly affect the speed of vehicles, thus leading to similar running state of vehicles in similar weather conditions. Secondly, the analysis of the running speed of vehicles in different lanes of the same section shows that the vehicles using the middle lane will travel at different speeds compared to the ones in the edge lanes [32, 33]. The difference in the information added for the traffic flow data repair helps to improve the restoration precision of the traffic flow data.


## 4. The Missing Data Repair Method Based on CMTF

The joint analysis of multisource data improves the understanding of the underlying structure of complex datasets. In the existing research, the repair of traffic flow missing data is mostly based on spatiotemporal information, while the importance of auxiliary information in data repair is neglected. Furthermore, due to the heterogeneity of the latter, there are different quantitative standards in practical application. Therefore, a method is required to integrate the auxiliary information about traffic flow with the spatial-temporal information to repair the missing data.The CMTF model can fit the matrix data into the tensor model through the coupling and capture the underlying structural characteristics of the heterogeneous data, realizing the joint analysis. Thus, combining with the CMTF model, this paper proposes a missing data repair method based on multisource information.


### 4.1. The CMTF Representation of the Traffic Flow Data

The schematic diagram of the CMTF model is shown in Figure3. There are three datasets in the figure: the tensor with missing traffic flow data, the feature matrix of weather feature data, and the lane feature data matrix.Figure 3
The CMTF model of the missing traffic flow data.The third-order tensorχ∈ℝI×J×K is constructed from the perspective of the spatiotemporal correlation of the main traffic flow data, where I represents the longitudinal time-series data of the same day in I consecutive weeks, J represents the J lanes of the same detection section, and K represents the K transverse time-series data of a day.According to the auxiliary traffic flow data, also considering that the third-order tensor contains the same underlying low-rank structure with the construction of the third-order tensor, the two following feature matrices are constructed:(1)
Weather Feature Matrix. Different weather conditions, for example, rain, will affect the operational behaviours of vehicles. The wet roads and low visibility will directly affect the speed of vehicles, resulting in a similar running state for all vehicles. The weather data matrix Q∈ℝI×M is in the composition of feature dimension, where I represents the longitudinal time-series data of the same day in I consecutive weeks, and M represents the M transverse time-series data of a day. Here, the sunny, cloudy (foggy), and rainy/snowy days are assigned the values of 1, 2, and 3, respectively. Thus, the element xim in the matrix is an integer varying from 1 to 3.(2)
Lane Feature Matrix. The running state of the vehicles traveling towards the same direction in the different lanes of the same section will have typical differences. For example, near the separator, the operating characteristics of the vehicle lane edge will be different, which will be considered in the traffic flow data repair. For the lane feature matrix D∈ℝJ×N, where J represents the J lanes of the same detection section and N represents the lane attribute values. Starting from the median, lane of attribute assignment is, respectively, 1, 2, 2, and 3. Thus, the elements in matrix yjn are integers varying between 1 and 3.The tensor containing missing data is constructed for the traffic flow spatiotemporal data, where the lane and weather feature matrices are coupled in lane and date dimensions, respectively. In this way, the data fusion problem is transformed into the CMTF decomposition problem. Thus, the missing data in the original tensor can be repaired by decomposing the tensor and matrix into multiple datasets and then reconstructing the tensor.


### 4.2. Objective Function Construction

The factor matricesA1, B1, and C are obtained by the CP decomposition of tensor χ, where A1∈ℝI×R, B1∈ℝJ×R, and C∈ℝK×R. Decomposition of matrices Q and D, respectively, gives the factor matrices A2, V, B2, and W, where A2∈ℝI×M, V∈ℝI×M, B2∈ℝJ×N, and W∈ℝJ×N. Thus, the objective function of CMTF decomposition can be expressed as(6)fA,B,C,V=12χ−A1,B1,C2+12Q−A2VT2+12D−B2WT2.The missing data in tensorχ can be repaired by coupling it with a known matrix.Next, denote byFx the set of the squared errors between the original tensor and the factor matrices A1, B1, and C, where the reconstruction tensor is obtained by CP decomposition. The smaller Fx is, the better the repair effect of missing data in the original tensor will be. R is the rank of the tensor, and λ is the regularization parameter. A1F, B1F, and CF are the norms corresponding to each factor matrix. Thus, the tensor χ is decomposed and expanded according to one module CP as(7)Fχ=12X1−A1B1⊙CTF2+λ12A1F2+B1F2+CF2,where X1 is the matrix representation of the tensor on one modulus. The sets FQ and FD represent the squared error between the original matrix and the matrix reconstructed from the factor matrix, which are obtained by matrix decomposition as(8)FQ=12Q−A2VTF2+λ22A2F2+VF2,FD=12D−B2WTF2+λ32B2F2+WF2,Ff=Fx+FQ+FD.According to the analysis of the CMTF model, the matrix should have the same underlying low-rank structure as the tensor in at least one dimension. Therefore, the matrix and the tensor should have a common modulus factor matrix. The assumption is thatA1 and A2are equal; and B1 and B2 are equal. Therefore, two global optimal variables A¯ and B¯ are introduced to satisfy the application premise of the CMTF model:(9)A1−A¯=0,A2−A¯=0,B1−B¯=0,B2−B¯=0.Thus, the optimization problem of the objective function is described as(10)minΨFf=Fχ+FQ+FD,s.t.Ai−A¯=0Bi−B¯=0andi=1,2,where Ψ represents the factor matrix set: Ψ=A1,B1,C,A2,V,B2,W. Therefore, the problem of missing data repair is transformed into the problem of solving the optimal value of the objective function Ff.


### 4.3. Optimization Solution

For optimization, the alternating least square (ALS) and gradient descent method are commonly used in the literature [34–40]. Since both the matrix and the tensor decomposition are a nonconvex problem, the commonly used methods cannot achieve the global optimum. To resolve this issue, the alternating direction multiplier algorithm (ADMM) is suggested [41]. In this method, large global problems are decomposed into many smaller and easier local subproblems, obtaining large global optimal solutions.Thus, whenFf is finding the optimal solution, it can be interpreted by solving the local Fχ, FQ, and FD optimal solution problems. The objective function of this problem can be given as(11)LρΨ,ΘA1,ΘA2,ΘB1,ΘB2,A¯,B¯=Ff+∑i=12trΘAiAi−A¯+ρ2Ai−A¯F2+∑i=12trΘBiBi−B¯+ρ2Bi−B¯F2,where ΘA11,ΘA21,ΘB11,ΘB21 are the Lagrangian multipliers [31], ρ is the penalty coefficient >0, and tr is the trace of the matrix, that is, the sum of the diagonal elements in the matrix. The iterative calculation formula of (9) is(12)Ψk+1←argminΨLρΨ,ΘA1k,ΘA2k,ΘB1k,ΘB2k,A¯k,B¯kA¯k+1,B¯k+1←argminA¯B¯LρΨk+1,ΘA1k,ΘA2k,ΘB1k,ΘB2k,A¯,B¯ΘAi=ΘAi+ρAi−A¯i=1,2,ΘBi=ΘBi+ρBi−B¯i=1,2.ConcerningLρ, the partial derivative of each factor matrix can be obtained as(13)∂Lρ∂A1=−X1−A1C⊙B1TC⊙B1+λ1+ρA1+ΘA1−ρA¯.By setting∂Lρ/∂A1=0,(14)A1=X1C⊙B1+ρA¯−ΘA1CTC∗B1TB1+λ1+ρIR−1.Similarly,(15)B1=X2C⊙A1+ρB¯−ΘB1CTC∗A1TA1+λ1+ρIR−1,(16)C=X3B1⊙A1B1TB1∗A1TA+λ1IR−1,(17)A2=QV+ρA¯−ΘA2VTV+λ2+ρIR−1,(18)B2=DW+ρB¯−ΘB2WTW+λ3+ρIR−1,(19)V=QTA2A2TA2+λ2IR−1,(20)W=DTB2B2TB2+λ3IR−1.Similarly, it can be inferred further that(21)A¯=ΘA1+ΘA22ρ+A1+A22,B¯=ΘB1+ΘB22ρ+B1+B22.If setΘAi0=0,i=1,2, then ∑i=12ΘAik=0,k=1,2,…,Imax, and it can be inferred further that(22)A¯=A1+A22,(23)B¯=B1+B22.Based on the above-given analysis, the main steps of the CMTF-based traffic flow missing data repair method are shown in Table1.Table 1
The main steps of CMTF.
Input: the tensor χ∈ℝI×J×K with missing traffic flow data, matrices Q∈ℝI×M and D∈ℝJ×N, tensor rank R, initial iteration number iter, maximum iteration number MaxIterOutput: completed tensor χ∈ℝI×J×K(1) Initializeρ, λ1,λ2,λ3R,A1,B1,CA2,V, B2,WΘA1,ΘA2,ΘB1,ΘB2=0, iter=0(2) CalculateA¯ using (22), and B¯ using (23)(3) Ifiter<MaxIter, repeat Step 4 ∼ Step 8(4) Using (14)–(16), calculate and update A1,B1,C, respectively(5) Using (17)–(20), calculate and update A2,V,B2,W, respectively(6) Using (22), calculate and update A¯. Similarly, calculate and update B¯ using (23)(7) Using (12), calculate and update ΘA11,ΘA21,ΘB11,ΘB21(8) Setiter+1=iter. If the algorithm reaches the convergence standard, the loop is broken. Otherwise, return to Step 3(9) Return toχ=A1∘B1∘C


## 4.1. The CMTF Representation of the Traffic Flow Data

The schematic diagram of the CMTF model is shown in Figure3. There are three datasets in the figure: the tensor with missing traffic flow data, the feature matrix of weather feature data, and the lane feature data matrix.Figure 3
The CMTF model of the missing traffic flow data.The third-order tensorχ∈ℝI×J×K is constructed from the perspective of the spatiotemporal correlation of the main traffic flow data, where I represents the longitudinal time-series data of the same day in I consecutive weeks, J represents the J lanes of the same detection section, and K represents the K transverse time-series data of a day.According to the auxiliary traffic flow data, also considering that the third-order tensor contains the same underlying low-rank structure with the construction of the third-order tensor, the two following feature matrices are constructed:(1)
Weather Feature Matrix. Different weather conditions, for example, rain, will affect the operational behaviours of vehicles. The wet roads and low visibility will directly affect the speed of vehicles, resulting in a similar running state for all vehicles. The weather data matrix Q∈ℝI×M is in the composition of feature dimension, where I represents the longitudinal time-series data of the same day in I consecutive weeks, and M represents the M transverse time-series data of a day. Here, the sunny, cloudy (foggy), and rainy/snowy days are assigned the values of 1, 2, and 3, respectively. Thus, the element xim in the matrix is an integer varying from 1 to 3.(2)
Lane Feature Matrix. The running state of the vehicles traveling towards the same direction in the different lanes of the same section will have typical differences. For example, near the separator, the operating characteristics of the vehicle lane edge will be different, which will be considered in the traffic flow data repair. For the lane feature matrix D∈ℝJ×N, where J represents the J lanes of the same detection section and N represents the lane attribute values. Starting from the median, lane of attribute assignment is, respectively, 1, 2, 2, and 3. Thus, the elements in matrix yjn are integers varying between 1 and 3.The tensor containing missing data is constructed for the traffic flow spatiotemporal data, where the lane and weather feature matrices are coupled in lane and date dimensions, respectively. In this way, the data fusion problem is transformed into the CMTF decomposition problem. Thus, the missing data in the original tensor can be repaired by decomposing the tensor and matrix into multiple datasets and then reconstructing the tensor.


## 4.2. Objective Function Construction

The factor matricesA1, B1, and C are obtained by the CP decomposition of tensor χ, where A1∈ℝI×R, B1∈ℝJ×R, and C∈ℝK×R. Decomposition of matrices Q and D, respectively, gives the factor matrices A2, V, B2, and W, where A2∈ℝI×M, V∈ℝI×M, B2∈ℝJ×N, and W∈ℝJ×N. Thus, the objective function of CMTF decomposition can be expressed as(6)fA,B,C,V=12χ−A1,B1,C2+12Q−A2VT2+12D−B2WT2.The missing data in tensorχ can be repaired by coupling it with a known matrix.Next, denote byFx the set of the squared errors between the original tensor and the factor matrices A1, B1, and C, where the reconstruction tensor is obtained by CP decomposition. The smaller Fx is, the better the repair effect of missing data in the original tensor will be. R is the rank of the tensor, and λ is the regularization parameter. A1F, B1F, and CF are the norms corresponding to each factor matrix. Thus, the tensor χ is decomposed and expanded according to one module CP as(7)Fχ=12X1−A1B1⊙CTF2+λ12A1F2+B1F2+CF2,where X1 is the matrix representation of the tensor on one modulus. The sets FQ and FD represent the squared error between the original matrix and the matrix reconstructed from the factor matrix, which are obtained by matrix decomposition as(8)FQ=12Q−A2VTF2+λ22A2F2+VF2,FD=12D−B2WTF2+λ32B2F2+WF2,Ff=Fx+FQ+FD.According to the analysis of the CMTF model, the matrix should have the same underlying low-rank structure as the tensor in at least one dimension. Therefore, the matrix and the tensor should have a common modulus factor matrix. The assumption is thatA1 and A2are equal; and B1 and B2 are equal. Therefore, two global optimal variables A¯ and B¯ are introduced to satisfy the application premise of the CMTF model:(9)A1−A¯=0,A2−A¯=0,B1−B¯=0,B2−B¯=0.Thus, the optimization problem of the objective function is described as(10)minΨFf=Fχ+FQ+FD,s.t.Ai−A¯=0Bi−B¯=0andi=1,2,where Ψ represents the factor matrix set: Ψ=A1,B1,C,A2,V,B2,W. Therefore, the problem of missing data repair is transformed into the problem of solving the optimal value of the objective function Ff.


## 4.3. Optimization Solution

For optimization, the alternating least square (ALS) and gradient descent method are commonly used in the literature [34–40]. Since both the matrix and the tensor decomposition are a nonconvex problem, the commonly used methods cannot achieve the global optimum. To resolve this issue, the alternating direction multiplier algorithm (ADMM) is suggested [41]. In this method, large global problems are decomposed into many smaller and easier local subproblems, obtaining large global optimal solutions.Thus, whenFf is finding the optimal solution, it can be interpreted by solving the local Fχ, FQ, and FD optimal solution problems. The objective function of this problem can be given as(11)LρΨ,ΘA1,ΘA2,ΘB1,ΘB2,A¯,B¯=Ff+∑i=12trΘAiAi−A¯+ρ2Ai−A¯F2+∑i=12trΘBiBi−B¯+ρ2Bi−B¯F2,where ΘA11,ΘA21,ΘB11,ΘB21 are the Lagrangian multipliers [31], ρ is the penalty coefficient >0, and tr is the trace of the matrix, that is, the sum of the diagonal elements in the matrix. The iterative calculation formula of (9) is(12)Ψk+1←argminΨLρΨ,ΘA1k,ΘA2k,ΘB1k,ΘB2k,A¯k,B¯kA¯k+1,B¯k+1←argminA¯B¯LρΨk+1,ΘA1k,ΘA2k,ΘB1k,ΘB2k,A¯,B¯ΘAi=ΘAi+ρAi−A¯i=1,2,ΘBi=ΘBi+ρBi−B¯i=1,2.ConcerningLρ, the partial derivative of each factor matrix can be obtained as(13)∂Lρ∂A1=−X1−A1C⊙B1TC⊙B1+λ1+ρA1+ΘA1−ρA¯.By setting∂Lρ/∂A1=0,(14)A1=X1C⊙B1+ρA¯−ΘA1CTC∗B1TB1+λ1+ρIR−1.Similarly,(15)B1=X2C⊙A1+ρB¯−ΘB1CTC∗A1TA1+λ1+ρIR−1,(16)C=X3B1⊙A1B1TB1∗A1TA+λ1IR−1,(17)A2=QV+ρA¯−ΘA2VTV+λ2+ρIR−1,(18)B2=DW+ρB¯−ΘB2WTW+λ3+ρIR−1,(19)V=QTA2A2TA2+λ2IR−1,(20)W=DTB2B2TB2+λ3IR−1.Similarly, it can be inferred further that(21)A¯=ΘA1+ΘA22ρ+A1+A22,B¯=ΘB1+ΘB22ρ+B1+B22.If setΘAi0=0,i=1,2, then ∑i=12ΘAik=0,k=1,2,…,Imax, and it can be inferred further that(22)A¯=A1+A22,(23)B¯=B1+B22.Based on the above-given analysis, the main steps of the CMTF-based traffic flow missing data repair method are shown in Table1.Table 1
The main steps of CMTF.
Input: the tensor χ∈ℝI×J×K with missing traffic flow data, matrices Q∈ℝI×M and D∈ℝJ×N, tensor rank R, initial iteration number iter, maximum iteration number MaxIterOutput: completed tensor χ∈ℝI×J×K(1) Initializeρ, λ1,λ2,λ3R,A1,B1,CA2,V, B2,WΘA1,ΘA2,ΘB1,ΘB2=0, iter=0(2) CalculateA¯ using (22), and B¯ using (23)(3) Ifiter<MaxIter, repeat Step 4 ∼ Step 8(4) Using (14)–(16), calculate and update A1,B1,C, respectively(5) Using (17)–(20), calculate and update A2,V,B2,W, respectively(6) Using (22), calculate and update A¯. Similarly, calculate and update B¯ using (23)(7) Using (12), calculate and update ΘA11,ΘA21,ΘB11,ΘB21(8) Setiter+1=iter. If the algorithm reaches the convergence standard, the loop is broken. Otherwise, return to Step 3(9) Return toχ=A1∘B1∘C


## 5. Results and Discussion

### 5.1. Data Source and Analysis

#### 5.1.1. Data Sources

The main section of an urban expressway is selected as the area to be analysed. The section includes five detection sections (in the east and west) with four lanes in each section and a total of 40 coil detectors. The data that will be collected by the detectors is shown in Table2, where “TY” indicates the code of the detection section, “D” refers to the east, and “10 (2)” means Lane 2 of the detection section #10. The data is sampled from August 27, 2018, to September 29, 2018, with a sampling interval of 5 min. There are 161280 groups of data.Table 2
Data types.
Detector numberAttributeDetection timeFlowSpeedOccupancyTYD10(2)Valid2018-8-27 0 : 053204TYX10(1)Invalid2018-8-27 0 : 050013TYD12(1)Lost2018-8-27 0 : 10000


#### 5.1.2. Judgment Criteria for Data Repair

The existence and reparability of the missing flow data have to be judged before starting a repair process. To determine whether anything is missing, the traffic flow data can be used. For example, if the flow data is displayed as “0,” it might be due to no vehicle passing through the detector or might be due to some missed data.Under normal traffic conditions, the missing traffic flow data can be repaired according to the changing trend in the traffic flow. However, under abnormal conditions, such as traffic accidents conditions, once the data is “0,” it is impossible to judge the existence and (if exists) the reparability of the data.Following the continuous progress in detection technology, the detectors have become more capable of realizing the preliminary discrimination of data anomalies. For example, the location detector can classify the data as valid, invalid, and/or lost. For repair judgment, this paper only considers the invalid data with the value of “0,” which is denoted as the missed data that will be repaired.


### 5.2. Experimental Scheme Design

(1)
Data Representation. Let us construct the tensor χ∈ℝ5×4×288 for the target data, where “5” represents “Day-mode,” which means the longitudinal time-series data of five consecutive Mondays, “4” represents “Lane-mode,” which means the transverse spatial sequence data of four lanes in the same detection section, and “288” represents “Time-mode,” which means the 288 transverse time-series data obtained in one day. The weather feature matrix is Q∈ℝ5×288, where “5” refers to five longitudinal time-series data of Monday, and “288” represents the 288 transverse time-series data obtained in one day. The lane feature matrix is D∈ℝ4×3, where “4” refers to the data of four lanes of the same detection section, and “3” represents the lane attribute values.(2)
The evaluation index of the missing data repair effect. The root mean square error (RMSE) and mean absolute percentage error (MAPE) are used as evaluation indexes [14]. It should be noted that the smaller the difference between the repaired and actual data is, the smaller the RMSE and MAPE will be. These errors are calculated as follows: (24)RMSE=∑i=1ny^i−yi2n,MAPE=∑i=1ny^i−yi/yin×100%,whereyi is the actual data, y^i is the repaired data, and n is the number of repaired pieces of data.(3)
Missing Data Type Design. To illustrate the data repair performance of the CMTF, three types of missing data were manually set up: MCAR, MAR, and MIX-M. Then, the data missing rates of each type are set to 10%, 20%, …, 90%. In order to avoid the influence of the randomly set data missing position on the repair result, ten settings were made for different data missing types, and the average value of the 10 repair effects (RMSE and MAPE) corresponding to each method was selected as the final repair result.


### 5.3. Analysis of the Experimental Results

The experiment environment is designed using Matlab-R2014a software, where the data repair based on the CMTF algorithm is realized by the CMTF_Toolbox_v1_1toolbox [15]. The relevant parameters of the model are obtained by cross-validation.The TYD12(2) detector with relatively complete flow data on August 27, 2018, was selected for tests, which were loaded with three missing data types (MCAR, MAR, and MIX-M) and nine missing rates (10% to 90%).Three methods, namely, matrix repair method [13] (Method 1), tensor repair algorithm (Method 2) [20], and the optimization algorithm of CMTF based on alternating least square (Method 3), are compared with the algorithm proposed in this paper (Method 4). The specific analysis of the data repair effect is as follows.


#### 5.3.1. Repair Effect Evaluation of the MACR

For the MACR, the repair results are shown in Tables3 and 4.Table 3
The RMSE of the methods under the MACR.
MACRMissing rates (%)RMSEMethod 1Method 2Method 3Method 41103.472.321.5661.3362205.392.981.9911.7913308.046.155.895.17444010.717.997.877.33755012.1111.4610.7679.76766013.3512.7511.25210.90277015.03314.23313.35412.65488017.6117.6213.9613.23699020.9518.4715.1914.422Table 4
The MAPE of the methods under the MACR.
MACRMissing rate (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1105.102.902.501.802207.283.983.702.983309.638.978.206.2544014.1813.7812.8010.8055019.4517.4414.1112.3166021.6119.1215.4713.3377025.7019.9815.9514.7588027.8021.2616.2115.1099032.4325.6717.4515.97The results show that the RMSE and MAPE of all methods are relatively small and close to each other for low data missing rates. However, with the gradual increase of the missing rate, the RMSE and MAPE for all methods also increase gradually. For high missing rates, the RMSE of Method 3 and Method 4 remains around 15, which is significantly lower than that of the other two methods. This indicates that the multidimensional flow data has a significant effect on the repair of the missing data. The RMSE and MAPE of Method 4 are generally lower than the corresponding values of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS.


#### 5.3.2. Repair Effect Evaluation of the MAR

For the MAR, the repair results are shown in Tables5 and 6.Table 5
The RMSE of the methods under the MAR.
MARMissing rates (%)RMSEMethod 1Method 2Method 3Method 41104.713.22.732.3312207.094.123.783.05433010.429.437.976.2844011.411.7210.219.63155015.2714.9814.0511.53666017.0716.2714.2813.41777020.3318.5615.5914.8988023.6119.3716.0115.73399030.8621.316.4616.193Table 6
The MAPE of the methods under the MAR.
MARMissing rates (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1106.883.773.262.262208.774.814.834.1833010.979.099.557.9944015.8814.2613.7711.8955021.9919.8717.2714.5666024.8520.3318.6315.7177028.3722.3119.3716.4588030.5524.6919.9317.9799034.6930.7120.1119.54The results show that the RMSE and MAPE of all methods are relatively small for low missing rates similar to MACR. However, with the gradual increase of the miss rate, the RMSE and MAPE values also increase gradually, where Method 4 has the slowest growth rate. For high missing rates, the RMSE of Method 3 and Method 4 is still lower than 16.5%, which is significantly lower than the other two methods. This indicates that the multisource flow data has a more significant effect on the repair.The repair effect of Method 4 is lower than that of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS. Compared to the MACR, the RMSE and MAPE of all methods under the MAR are relatively larger. This is because the missing data is continuous, which will affect the repair effect to some extent.


#### 5.3.3. Repair Effect Evaluation of the MIX-M

For the MIX-M, the repair results are shown in Tables7 and 8.Table 7
The RMSE of the methods under the MIX-M.
MIX-MMissing rates (%)RMSEMethod 1Method 2Method 3Method 41104.052.632.051.4122206.053.412.922.1413309.558.737.885.97644011.3810.59.518.355014.614.31310.32466016.0415.7813.9212.87577019.416.4614.3613.68488022.518.7515.5114.299027.4220.3115.9814.996Table 8
The MAPE of the methods under the MIX-M.
MIX-MMissing rates (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1106.903.422.911.942208.234.763.893.283309.978.888.657.4744013.8814.1113.1411.7655021.0018.5616.7212.9966023.5519.7617.4314.8177026.5720.1218.9715.9588029.6522.2519.0016.7799035.0727.8119.9917.04The overall trend is similar to that of MACR and MAR, but the RMSE and MAPE of all methods are in the middle. This is because the missing data in MIX-M includes both the discrete missing data of MACR and the continuous missing data of MAR. Thus, the existence of continuous missing data will affect the repair process.The MIX-M can reflect the lack of traffic flow data. From Figure4, for the missing rate of 50%, the MAPE of all methods is lower than 21%. When the missing rate exceeds 60%, the MAPE difference between the methods becomes larger. Here, Method 4 still has the best repair effect for high data missing rates.Figure 4
The MAPE versus missing rate for all methods under the MIX-M.Under the three types of data missing, all methods can repair the missing data at certain levels, where the repair effect of the MACR is the best, and the MAR is the worst.In comparison, the repair effect of Method 2 is better than that of Method 1. This indicates that the tensor algorithm based on multidimensional data can improve the repair effect.The repair effect of Method 3 and Method 4 is better than the other two, indicating that the addition of auxiliary data is useful in data repair.The repair effect of Method 4 is better than that of Method 3; that is, the optimal solution of ADMM is better than the traditional solution that ALS offers.Our proposed method has a best repair effect, which has the lowest RMSE and MAPE among all methods. Even for the MAR with a miss rate of more than 80%, the RMSE and MAPE of our method are still around 16.5 and 20%, respectively. This shows that the accuracy of traffic flow missing data repair can be effectively improved by adding auxiliary data based on sufficient and reasonable use of the spatial and temporal correlation between the traffic flow data.


## 5.1. Data Source and Analysis

### 5.1.1. Data Sources

The main section of an urban expressway is selected as the area to be analysed. The section includes five detection sections (in the east and west) with four lanes in each section and a total of 40 coil detectors. The data that will be collected by the detectors is shown in Table2, where “TY” indicates the code of the detection section, “D” refers to the east, and “10 (2)” means Lane 2 of the detection section #10. The data is sampled from August 27, 2018, to September 29, 2018, with a sampling interval of 5 min. There are 161280 groups of data.Table 2
Data types.
Detector numberAttributeDetection timeFlowSpeedOccupancyTYD10(2)Valid2018-8-27 0 : 053204TYX10(1)Invalid2018-8-27 0 : 050013TYD12(1)Lost2018-8-27 0 : 10000


### 5.1.2. Judgment Criteria for Data Repair

The existence and reparability of the missing flow data have to be judged before starting a repair process. To determine whether anything is missing, the traffic flow data can be used. For example, if the flow data is displayed as “0,” it might be due to no vehicle passing through the detector or might be due to some missed data.Under normal traffic conditions, the missing traffic flow data can be repaired according to the changing trend in the traffic flow. However, under abnormal conditions, such as traffic accidents conditions, once the data is “0,” it is impossible to judge the existence and (if exists) the reparability of the data.Following the continuous progress in detection technology, the detectors have become more capable of realizing the preliminary discrimination of data anomalies. For example, the location detector can classify the data as valid, invalid, and/or lost. For repair judgment, this paper only considers the invalid data with the value of “0,” which is denoted as the missed data that will be repaired.


## 5.1.1. Data Sources

The main section of an urban expressway is selected as the area to be analysed. The section includes five detection sections (in the east and west) with four lanes in each section and a total of 40 coil detectors. The data that will be collected by the detectors is shown in Table2, where “TY” indicates the code of the detection section, “D” refers to the east, and “10 (2)” means Lane 2 of the detection section #10. The data is sampled from August 27, 2018, to September 29, 2018, with a sampling interval of 5 min. There are 161280 groups of data.Table 2
Data types.
Detector numberAttributeDetection timeFlowSpeedOccupancyTYD10(2)Valid2018-8-27 0 : 053204TYX10(1)Invalid2018-8-27 0 : 050013TYD12(1)Lost2018-8-27 0 : 10000


## 5.1.2. Judgment Criteria for Data Repair

The existence and reparability of the missing flow data have to be judged before starting a repair process. To determine whether anything is missing, the traffic flow data can be used. For example, if the flow data is displayed as “0,” it might be due to no vehicle passing through the detector or might be due to some missed data.Under normal traffic conditions, the missing traffic flow data can be repaired according to the changing trend in the traffic flow. However, under abnormal conditions, such as traffic accidents conditions, once the data is “0,” it is impossible to judge the existence and (if exists) the reparability of the data.Following the continuous progress in detection technology, the detectors have become more capable of realizing the preliminary discrimination of data anomalies. For example, the location detector can classify the data as valid, invalid, and/or lost. For repair judgment, this paper only considers the invalid data with the value of “0,” which is denoted as the missed data that will be repaired.


## 5.2. Experimental Scheme Design

(1)
Data Representation. Let us construct the tensor χ∈ℝ5×4×288 for the target data, where “5” represents “Day-mode,” which means the longitudinal time-series data of five consecutive Mondays, “4” represents “Lane-mode,” which means the transverse spatial sequence data of four lanes in the same detection section, and “288” represents “Time-mode,” which means the 288 transverse time-series data obtained in one day. The weather feature matrix is Q∈ℝ5×288, where “5” refers to five longitudinal time-series data of Monday, and “288” represents the 288 transverse time-series data obtained in one day. The lane feature matrix is D∈ℝ4×3, where “4” refers to the data of four lanes of the same detection section, and “3” represents the lane attribute values.(2)
The evaluation index of the missing data repair effect. The root mean square error (RMSE) and mean absolute percentage error (MAPE) are used as evaluation indexes [14]. It should be noted that the smaller the difference between the repaired and actual data is, the smaller the RMSE and MAPE will be. These errors are calculated as follows: (24)RMSE=∑i=1ny^i−yi2n,MAPE=∑i=1ny^i−yi/yin×100%,whereyi is the actual data, y^i is the repaired data, and n is the number of repaired pieces of data.(3)
Missing Data Type Design. To illustrate the data repair performance of the CMTF, three types of missing data were manually set up: MCAR, MAR, and MIX-M. Then, the data missing rates of each type are set to 10%, 20%, …, 90%. In order to avoid the influence of the randomly set data missing position on the repair result, ten settings were made for different data missing types, and the average value of the 10 repair effects (RMSE and MAPE) corresponding to each method was selected as the final repair result.


## 5.3. Analysis of the Experimental Results

The experiment environment is designed using Matlab-R2014a software, where the data repair based on the CMTF algorithm is realized by the CMTF_Toolbox_v1_1toolbox [15]. The relevant parameters of the model are obtained by cross-validation.The TYD12(2) detector with relatively complete flow data on August 27, 2018, was selected for tests, which were loaded with three missing data types (MCAR, MAR, and MIX-M) and nine missing rates (10% to 90%).Three methods, namely, matrix repair method [13] (Method 1), tensor repair algorithm (Method 2) [20], and the optimization algorithm of CMTF based on alternating least square (Method 3), are compared with the algorithm proposed in this paper (Method 4). The specific analysis of the data repair effect is as follows.


### 5.3.1. Repair Effect Evaluation of the MACR

For the MACR, the repair results are shown in Tables3 and 4.Table 3
The RMSE of the methods under the MACR.
MACRMissing rates (%)RMSEMethod 1Method 2Method 3Method 41103.472.321.5661.3362205.392.981.9911.7913308.046.155.895.17444010.717.997.877.33755012.1111.4610.7679.76766013.3512.7511.25210.90277015.03314.23313.35412.65488017.6117.6213.9613.23699020.9518.4715.1914.422Table 4
The MAPE of the methods under the MACR.
MACRMissing rate (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1105.102.902.501.802207.283.983.702.983309.638.978.206.2544014.1813.7812.8010.8055019.4517.4414.1112.3166021.6119.1215.4713.3377025.7019.9815.9514.7588027.8021.2616.2115.1099032.4325.6717.4515.97The results show that the RMSE and MAPE of all methods are relatively small and close to each other for low data missing rates. However, with the gradual increase of the missing rate, the RMSE and MAPE for all methods also increase gradually. For high missing rates, the RMSE of Method 3 and Method 4 remains around 15, which is significantly lower than that of the other two methods. This indicates that the multidimensional flow data has a significant effect on the repair of the missing data. The RMSE and MAPE of Method 4 are generally lower than the corresponding values of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS.


### 5.3.2. Repair Effect Evaluation of the MAR

For the MAR, the repair results are shown in Tables5 and 6.Table 5
The RMSE of the methods under the MAR.
MARMissing rates (%)RMSEMethod 1Method 2Method 3Method 41104.713.22.732.3312207.094.123.783.05433010.429.437.976.2844011.411.7210.219.63155015.2714.9814.0511.53666017.0716.2714.2813.41777020.3318.5615.5914.8988023.6119.3716.0115.73399030.8621.316.4616.193Table 6
The MAPE of the methods under the MAR.
MARMissing rates (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1106.883.773.262.262208.774.814.834.1833010.979.099.557.9944015.8814.2613.7711.8955021.9919.8717.2714.5666024.8520.3318.6315.7177028.3722.3119.3716.4588030.5524.6919.9317.9799034.6930.7120.1119.54The results show that the RMSE and MAPE of all methods are relatively small for low missing rates similar to MACR. However, with the gradual increase of the miss rate, the RMSE and MAPE values also increase gradually, where Method 4 has the slowest growth rate. For high missing rates, the RMSE of Method 3 and Method 4 is still lower than 16.5%, which is significantly lower than the other two methods. This indicates that the multisource flow data has a more significant effect on the repair.The repair effect of Method 4 is lower than that of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS. Compared to the MACR, the RMSE and MAPE of all methods under the MAR are relatively larger. This is because the missing data is continuous, which will affect the repair effect to some extent.


### 5.3.3. Repair Effect Evaluation of the MIX-M

For the MIX-M, the repair results are shown in Tables7 and 8.Table 7
The RMSE of the methods under the MIX-M.
MIX-MMissing rates (%)RMSEMethod 1Method 2Method 3Method 41104.052.632.051.4122206.053.412.922.1413309.558.737.885.97644011.3810.59.518.355014.614.31310.32466016.0415.7813.9212.87577019.416.4614.3613.68488022.518.7515.5114.299027.4220.3115.9814.996Table 8
The MAPE of the methods under the MIX-M.
MIX-MMissing rates (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1106.903.422.911.942208.234.763.893.283309.978.888.657.4744013.8814.1113.1411.7655021.0018.5616.7212.9966023.5519.7617.4314.8177026.5720.1218.9715.9588029.6522.2519.0016.7799035.0727.8119.9917.04The overall trend is similar to that of MACR and MAR, but the RMSE and MAPE of all methods are in the middle. This is because the missing data in MIX-M includes both the discrete missing data of MACR and the continuous missing data of MAR. Thus, the existence of continuous missing data will affect the repair process.The MIX-M can reflect the lack of traffic flow data. From Figure4, for the missing rate of 50%, the MAPE of all methods is lower than 21%. When the missing rate exceeds 60%, the MAPE difference between the methods becomes larger. Here, Method 4 still has the best repair effect for high data missing rates.Figure 4
The MAPE versus missing rate for all methods under the MIX-M.Under the three types of data missing, all methods can repair the missing data at certain levels, where the repair effect of the MACR is the best, and the MAR is the worst.In comparison, the repair effect of Method 2 is better than that of Method 1. This indicates that the tensor algorithm based on multidimensional data can improve the repair effect.The repair effect of Method 3 and Method 4 is better than the other two, indicating that the addition of auxiliary data is useful in data repair.The repair effect of Method 4 is better than that of Method 3; that is, the optimal solution of ADMM is better than the traditional solution that ALS offers.Our proposed method has a best repair effect, which has the lowest RMSE and MAPE among all methods. Even for the MAR with a miss rate of more than 80%, the RMSE and MAPE of our method are still around 16.5 and 20%, respectively. This shows that the accuracy of traffic flow missing data repair can be effectively improved by adding auxiliary data based on sufficient and reasonable use of the spatial and temporal correlation between the traffic flow data.


## 5.3.1. Repair Effect Evaluation of the MACR

For the MACR, the repair results are shown in Tables3 and 4.Table 3
The RMSE of the methods under the MACR.
MACRMissing rates (%)RMSEMethod 1Method 2Method 3Method 41103.472.321.5661.3362205.392.981.9911.7913308.046.155.895.17444010.717.997.877.33755012.1111.4610.7679.76766013.3512.7511.25210.90277015.03314.23313.35412.65488017.6117.6213.9613.23699020.9518.4715.1914.422Table 4
The MAPE of the methods under the MACR.
MACRMissing rate (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1105.102.902.501.802207.283.983.702.983309.638.978.206.2544014.1813.7812.8010.8055019.4517.4414.1112.3166021.6119.1215.4713.3377025.7019.9815.9514.7588027.8021.2616.2115.1099032.4325.6717.4515.97The results show that the RMSE and MAPE of all methods are relatively small and close to each other for low data missing rates. However, with the gradual increase of the missing rate, the RMSE and MAPE for all methods also increase gradually. For high missing rates, the RMSE of Method 3 and Method 4 remains around 15, which is significantly lower than that of the other two methods. This indicates that the multidimensional flow data has a significant effect on the repair of the missing data. The RMSE and MAPE of Method 4 are generally lower than the corresponding values of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS.


## 5.3.2. Repair Effect Evaluation of the MAR

For the MAR, the repair results are shown in Tables5 and 6.Table 5
The RMSE of the methods under the MAR.
MARMissing rates (%)RMSEMethod 1Method 2Method 3Method 41104.713.22.732.3312207.094.123.783.05433010.429.437.976.2844011.411.7210.219.63155015.2714.9814.0511.53666017.0716.2714.2813.41777020.3318.5615.5914.8988023.6119.3716.0115.73399030.8621.316.4616.193Table 6
The MAPE of the methods under the MAR.
MARMissing rates (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1106.883.773.262.262208.774.814.834.1833010.979.099.557.9944015.8814.2613.7711.8955021.9919.8717.2714.5666024.8520.3318.6315.7177028.3722.3119.3716.4588030.5524.6919.9317.9799034.6930.7120.1119.54The results show that the RMSE and MAPE of all methods are relatively small for low missing rates similar to MACR. However, with the gradual increase of the miss rate, the RMSE and MAPE values also increase gradually, where Method 4 has the slowest growth rate. For high missing rates, the RMSE of Method 3 and Method 4 is still lower than 16.5%, which is significantly lower than the other two methods. This indicates that the multisource flow data has a more significant effect on the repair.The repair effect of Method 4 is lower than that of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS. Compared to the MACR, the RMSE and MAPE of all methods under the MAR are relatively larger. This is because the missing data is continuous, which will affect the repair effect to some extent.


## 5.3.3. Repair Effect Evaluation of the MIX-M

For the MIX-M, the repair results are shown in Tables7 and 8.Table 7
The RMSE of the methods under the MIX-M.
MIX-MMissing rates (%)RMSEMethod 1Method 2Method 3Method 41104.052.632.051.4122206.053.412.922.1413309.558.737.885.97644011.3810.59.518.355014.614.31310.32466016.0415.7813.9212.87577019.416.4614.3613.68488022.518.7515.5114.299027.4220.3115.9814.996Table 8
The MAPE of the methods under the MIX-M.
MIX-MMissing rates (%)MAPEMethod 1 (%)Method 2 (%)Method 3 (%)Method 4 (%)1106.903.422.911.942208.234.763.893.283309.978.888.657.4744013.8814.1113.1411.7655021.0018.5616.7212.9966023.5519.7617.4314.8177026.5720.1218.9715.9588029.6522.2519.0016.7799035.0727.8119.9917.04The overall trend is similar to that of MACR and MAR, but the RMSE and MAPE of all methods are in the middle. This is because the missing data in MIX-M includes both the discrete missing data of MACR and the continuous missing data of MAR. Thus, the existence of continuous missing data will affect the repair process.The MIX-M can reflect the lack of traffic flow data. From Figure4, for the missing rate of 50%, the MAPE of all methods is lower than 21%. When the missing rate exceeds 60%, the MAPE difference between the methods becomes larger. Here, Method 4 still has the best repair effect for high data missing rates.Figure 4
The MAPE versus missing rate for all methods under the MIX-M.Under the three types of data missing, all methods can repair the missing data at certain levels, where the repair effect of the MACR is the best, and the MAR is the worst.In comparison, the repair effect of Method 2 is better than that of Method 1. This indicates that the tensor algorithm based on multidimensional data can improve the repair effect.The repair effect of Method 3 and Method 4 is better than the other two, indicating that the addition of auxiliary data is useful in data repair.The repair effect of Method 4 is better than that of Method 3; that is, the optimal solution of ADMM is better than the traditional solution that ALS offers.Our proposed method has a best repair effect, which has the lowest RMSE and MAPE among all methods. Even for the MAR with a miss rate of more than 80%, the RMSE and MAPE of our method are still around 16.5 and 20%, respectively. This shows that the accuracy of traffic flow missing data repair can be effectively improved by adding auxiliary data based on sufficient and reasonable use of the spatial and temporal correlation between the traffic flow data.


## 6. Conclusions

To improve the repair effect of traffic flow missing data, this paper proposes a CMTF-based repair method. Different from the existing methods, the proposed method can make full and reasonable use of the spatial and temporal data of traffic flow to further increase the auxiliary traffic flow data and use the ADMM algorithm for optimization solution. The results showed that the proposed algorithm outperformed the other three comparison methods; that is, it has a better repair effect under three data deletion modes and nine data deletion rates. It can also achieve the best repair accuracy under high data deletion rates. The acquisition of complete traffic flow data realized by the proposed method will be highly beneficial in traffic management and control.Since this paper only considered the coupling of a 3D tensor and a 2D matrix, future research will focus on the coupling of multiple matrices based on the multidimensional tensor. While doing that, we will also consider the total time cost of the running model.



---

*Source: 2919073-2021-02-11.xml*