A deep learning algorithm for joint direct tracking and classification of manoeuvring sources

IET Radar, Sonar & NavigationVolume 16, Issue 7 p. 1198-1211 ORIGINAL RESEARCHOpen Access A deep learning algorithm for joint direct tracking and classification of manoeuvring sources Wanting Yu, Wanting Yu orcid.org/0000-0002-1515-3179 School of Information System Engineering, Information Engineering University, Zhengzhou, ChinaSearch for more papers by this authorHongyi Yu, Corresponding Author Hongyi Yu [email protected] School of Information System Engineering, Information Engineering University, Zhengzhou, China Correspondence Hongyi Yu, School of Information System Engineering, Information Engineering University, Zhengzhou, Henan, China. Email: [email protected]Search for more papers by this authorJianping Du, Jianping Du School of Information System Engineering, Information Engineering University, Zhengzhou, ChinaSearch for more papers by this authorMengli Zhang, Mengli Zhang orcid.org/0000-0002-7411-2561 State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, ChinaSearch for more papers by this authorDing Wang, Ding Wang orcid.org/0000-0001-6533-9206 School of Information System Engineering, Information Engineering University, Zhengzhou, ChinaSearch for more papers by this author Wanting Yu, Wanting Yu orcid.org/0000-0002-1515-3179 School of Information System Engineering, Information Engineering University, Zhengzhou, ChinaSearch for more papers by this authorHongyi Yu, Corresponding Author Hongyi Yu [email protected] School of Information System Engineering, Information Engineering University, Zhengzhou, China Correspondence Hongyi Yu, School of Information System Engineering, Information Engineering University, Zhengzhou, Henan, China. Email: [email protected]Search for more papers by this authorJianping Du, Jianping Du School of Information System Engineering, Information Engineering University, Zhengzhou, ChinaSearch for more papers by this authorMengli Zhang, Mengli Zhang orcid.org/0000-0002-7411-2561 State Key Laboratory of Mathematical Engineering and Advanced Computing, Zhengzhou, ChinaSearch for more papers by this authorDing Wang, Ding Wang orcid.org/0000-0001-6533-9206 School of Information System Engineering, Information Engineering University, Zhengzhou, ChinaSearch for more papers by this author First published: 07 April 2022 https://doi.org/10.1049/rsn2.12254AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The remarkable success of deep learning technologies has provided new ideas for solving complex tracking problems. It is difficult for traditional algorithms to directly estimate the trajectory vector and target class from the received signal due to the limitation of modelling ability, which causes inevitable information loss. Moreover, existing algorithms suffer severe performance degradation when dealing with problems that are difficult to mathematically model in advance, such as highly nonlinear observations and manoeuvring scenarios. To address these issues, we propose a deep learning algorithm for joint direct tracking and classification (DeepDTC), which is a novel direct tracking framework. Specifically, we construct a convolutional neural network (CNN)-based signal processing component to capture observation features, and a Transformer-based trajectory tracking component to capture the features of target state and identity. Meanwhile, in signal processing component, we design an attribute network to learn auxiliary knowledge features. Finally, we construct a multi-task learning network to connect these components and estimate trajectory vectors and target classes simultaneously. Our algorithm takes the transformer with attention mechanism as the core network, which is highly scalable and suitable for introducing various auxiliary knowledge. The comparison experiments with traditional methods demonstrate the effectiveness and advancement of the proposed algorithm. The comparative experiments with the single-task model show that DeepDTC can further improve the tracking accuracy by utilising the learnt classification information. 1 INTRODUCTION Direct tracking of manoeuvering source is a vital topic in military and civilian fields, such as navigation, aerospace, and radio monitoring. Complex observations and high manoeuverability of moving targets make the traditional tracking technology seriously challenged [1]. Research on stable and accurate trajectory tracking algorithms is crucial for the safety and effectiveness of the tracking technology. There have been many traditional tracking methods based on positioning parameters, such as the extended Kalman filter (EKF) [2], the unscented KF (UKF) [3], and the particle filter (PF) [4] etc. To deal with more complex discrete nonlinear systems, accurate continuous-discrete EKF (ADC-EKF) [5] and ADC-extended UKF [6] are also proposed. These are typical improved Kalman adaptive algorithms. The traditional tracking algorithm is a two-step method, and usually requires additional position measurements, such as the time difference of arrivals (TDOAs) and frequency difference of arrivals (FDOAs). Two-step methods can easily lead to information loss and performance degradation. For higher accuracy, direct tracking has been proposed as the one-step scheme. It directly determines the position and velocity from observed signals without estimating the positioning parameters. For example, Wang et al. [7] proposed a Newton-type iterative algorithm based on matrix eigenvalue perturbation theory. Ma et al. [1] proposed a new multiple PF-based direct position determination (DPD) method to estimate the locations and velocities. They all deduced the Cramér-Rao bound (CRB) for the dynamic source, and verified the convergence of the method in theory and actual values. The premise of these derivations is a uniform linear motion target. Nan et al. [8] linked the cyclic cross-spectrum with the PF algorithm and proposed a direct tracking method for multiple co-channel transmitters, which considers the motion modelled by the constant turning rate acceleration (CTRA) [9]. Other methods also use optimization algorithms to achieve DPD of moving targets by establishing a maximum likelihood model [10]. The direct tracking method can effectively avoid severe information loss and has better performance. However, the existing algorithms still face two challenges. First, the direct tracking algorithm needs to estimate multiple parameters in a highly nonlinear model jointly. Even using the optimization method such as PF to improve the accuracy in estimating motion states, the calculation is still too cumbersome to achieve real-time tracking. Second, model-based tracking algorithms require a priori known dynamic model, which is challenging to adapt to manoeuvering targets with very uncertain motion states. In this paper, we propose a direct tracking algorithm based on deep learning to address the difficulty of high-dimensional modelling and real-time tracking. In fact, the performance of traditional Bayesian-based tracking algorithms largely depends on the predefined dynamic model [11]. The major challenge for accurate tracking lies in the complexity and uncertainty of the target movement. Multi-model (MM) algorithms, such as interactive MM (IMM) [12], fixed structure MM (FSMM) [13] and interactive multi-model algorithms fusing input estimation and best linear unbiased estimation filter (MIE-BLUE-IMM) [14], have been proposed to provide better manoeuvering model approximations. Further, the target dynamic law is closely related to the target identity class. The joint tracking and classification (JTC) algorithm is proven to improve trajectory tracking performance [15]. JTC technology carries out two tasks of tracking and classification simultaneously. The interaction between the tracker and classifier can effectively improve the accuracy of both tracking and classification. For instance, trajectory tracking can affect target classification by offering different kinematic information, and classification can also affect tracking utilising category-related dynamics. Therefore, it is essential to handle target tracking and classification under a unified interaction framework [16]. Specifically, JTC technology can be divided into rigid body-based methods and particle-based methods. In rigid body-based forms, attribute information is mainly used for classification, such as the target shape provided by sensors [15] and electronic support measure (ESM) [17] etc. In particle-based forms, kinematic information is mainly used for classification, such as the motion attribute (including velocity and acceleration). Ristic et al. [18] systematically discussed the JTC technology based on the optimal Bayesian estimation and interactive multi-model filters for the first time. Angelova et al. [19] implemented the PF-based JTC algorithm using the movement information obtained by the tracking radar to classify and track the target simultaneously. Yang et al. [20] defined a set of motion models conditioned on the target class, which improves the performance by demonstrating the relationship between motion states and classes. However, traditional JTC algorithms usually require a set of predefined models, which suffer from model mismatch. Due to limited modelling capabilities and high computational cost, these methods are not always applicable. Especially when the observation noise and nonlinearity are severe, the delay of model estimation will cause the tracking real-time performance and accuracy to decrease significantly. To the best of our current knowledge, there is no joint algorithm for direct tracking and classification so far. Recently, several control methods based on neural network architecture are used in trajectory tracking applications [21], such as model reference adaptive control (MRAC) methods [22], neural dynamic programing (NDP) algorithms [23], and the value iterative reinforcement learning (VIRL) [24, 25]. With considerable advancement of artificial intelligence technology, the deep learning algorithm also plays a significant role in target tracking [26]. The deep learning technology makes it possible to utilise auxiliary knowledge that is difficult for traditional algorithms to model. Typical deep neural network structures include convolutional neural networks (CNN) [27], recurrent neural networks (RNN) [28], and long short-term memory (LSTM) [29] etc. Deep neural networks have high nonlinearity and powerful information fusion capabilities [30] to achieve manoeuvering tracking. For example, Jingxian et al. [31] constructed a BiLSTM-based model, which remarkably improves the accuracy and efficiency of manoeuvering target tracking. Yu et al. [32] constructed an LSTM-based model by learning dynamic laws, and realised real-time and accurate tracking of various manoeuvering motions. The interpretability experiments in [32] further indicate that dynamic laws of the manoeuvering target are captured by the network during tracking. Nevertheless, existing tracking networks only consider the two-step tracking, and mainly utilise RNNs, such as LSTM and gated recurrent unit (GRU) [33]. These networks often require a specific data structure, which makes them inflexible when dealing with more complex signal data from manoeuvering target. For the first time, we try to focus the attention of neural networks on learning the nonlinear mapping between observation information (include observed signal and auxiliary information) and target conditions (include identity and motion state). This paper mainly considers the particle-based JTC that uses the dynamic characteristics of the target to classify the target. We conduct a single-step scheme for tracking manoeuvering sources in complex scenes. Different from traditional methods based on mathematical modelling, our algorithm can fully utilise all observations, which avoids the information loss of two-step localization methods and is not limited by the Markov assumption in conventional tracking algorithms. We construct a deep learning model with a Transformer [34] as the core tracking network. As a fundamental difference from CNNs and RNNs, the Transformer does not make any specific assumption about the data structure. This effectively makes the Transformer a very versatile network architecture, with the potential to capture different ranges of dependencies and various multi-modal features [35, 36]. Specifically, we also design a deep learning algorithm for joint direct tracking and classification (DeepDTC) of manoeuvering sources. The algorithm improves both the performance of tracking and classification by learning the dependencies of received signals and trajectories. To summarise, the contributions of this paper mainly include: A novel deep learning model is constructed for direct trajectory tracking (DeepDT), which mainly consists of two main parts: CNN-based signal processing component, Transformer-based trajectory generating component. A multi-task learning model is designed for joint direct classification and tracking (DeepDTC), which enables the network to improve further tracking performance by learning the dynamic laws of different types of targets. Extensive experiments are implemented to prove the effectiveness and advancement of the proposed model. Experimental results show that the DeepDTC outperforms traditional methods and single-task deep learning algorithms. 2 DATA COLLECTION The radiation source location system studied in this paper belongs to a kind of passive radar. The radio signals radiated by the target can be intercepted by multiple stations. We use the received signals to achieve location and tracking. The moving radiation source radiates radio signals intermittently according to the communication rules. Limited by the communication system, obtaining observations at non-uniform time intervals is a more practical scenario. Trajectory model Consider different types of motion transmitters and L base stations receiving the signals. The tracking scenario is illustrated in Figure 1. Define the lth base station's position as the vector of two-dimensional (2D) Cartesian coordinate q l = q x , q y l T . We construct two types of aircraft signal sources manoeuvering in the observation area. The goal is to simultaneously track moving targets and identify types through multi-station observation signals. Define the motion state vector at the observation index n in the coordinate system as s n = x , v x , y , v y n T , p n = [ x , y ] n T represents the target position, v n = v x , v y n T is the velocity. Dynamic models describe the process of the movement state changing over time [37]. According to the dynamic model, we can get s n = f s n − 1 + A w n , (1)where f (⋅) represents the state transition function, wn represents the transition noise. A is the coefficient matrix that converts velocity noise to position noise. FIGURE 1Open in figure viewerPowerPoint The tracking scenario. The red lines represent trajectories of class (a), the green lines represent trajectories of class (b), and the blue dots correspond to the observation points of manoeuvering sources As shown in Figure 1, we assume two types of aircraft, A and B. Although the aircraft follow similar dynamic laws, different types of aircraft, such as helicopters and transport aeroplanes, are always diverse in their speed range, state noise, and manoeuvering characteristics. Typical dynamic models can be found in [32]. Specifically, the Equation (1) in CV model is s n = 1 T 0 0 0 1 0 0 0 0 1 T 0 0 0 1 s n − 1 + T 2 / 2 0 T 0 0 T 2 / 2 0 T w n , (2)where s n = x , v x , y , v y n T is the discrete-time state of the trajectory, w n = w x , w y n T is the transition noise sequence and is usually modelled as Gaussian noise. T is the sampling interval. Similarly, Equation (1) in CT model is given as: s n = 1 sin ( ω T ) / ω 0 [ cos ( ω T ) − 1 ] / ω 0 cos ( ω T ) 0 − sin ( ω T ) 0 [ 1 − cos ( ω T ) ] / ω 1 sin ( ω T ) / ω 0 sin ( ω T ) 0 cos ( ω T ) s n − 1 + T 2 / 2 0 T 0 0 T 2 / 2 0 T w n , (3)where ω is the angular velocity. And Equation (1) in CTRV model is given as: s n = s n − 1 + v sin ( ω T + θ ) / ω − v sin ( θ ) / ω − v cos ( ω T + θ ) / ω + v sin ( θ ) / ω 0 ω T 0 + T 2 cos ( θ ) / 2 0 T 2 sin ( θ ) / 2 0 T 0 0 T 2 / 2 0 T w n , (4)where s n = [ x , y , v , θ , ω ] n T is the trajectory state, v represents the velocity magnitude. θ is the yaw angle, which indicates the direction; w n = w a , w ω n T is the transition noise. The noise in the CTRV model mainly comes from two sources: linear acceleration noise wa and yaw angular noise wω. To unify the trajectory data, we set the state vectors as s ′ n = x , y , v x , v y n T , where vx = v cos θ, vy = v sin θ. Define the observation length as K, the sampling interval of step k as Tk, then the observation sequence of trajectory i can be described as: S 1 : K ( i ) = s n 1 , s n 2 , ⋯ , s n K ( i ) . In addition, we consider the non-uniform time interval. Assuming that the observation process obeys the Poisson process of parameter λ, the distribution of arrival time interval T k o in K observations can be described as F T k o ( t ) = P T k o ≤ t = 1 − e − λ t , t ≥ 0 , 0 , t < 0 . (5) Then the observation time can be defined as the start time of each time slot t k = ∑ i = 1 k T k o . The time series corresponding to the observation trajectory i is T ( i ) = t 1 , t 2 , ⋯ , t K ( i ) . Observation signal model The base station samples the signal in K time slots. It is assumed that the time of each observation slot is very short, and the state of the target in the observation slot is maintained. To achieve direct tracking, we assume that the sampling clocks of each station are synchronous. Define the sampling time as Ts at each time, then the complex envelope of the signal observed by the lth base station in the kth time slot is [7]. r l , k ( t ) = α l , k s ˜ k t − τ l , k ⋅ e j 2 π f l , k t + n l , k ( t ) 1 ≤ l ≤ L ; 1 ≤ k ≤ K ; 1 ≤ t ≤ T s , (6)where s ˜ k ( t ) represents the complex envelope of the source signal, which is consistent for each observing station. αl,k is the propagation coefficient of the lth observing station in the kth slot segment, and τl,k is the intercepted signal delay. The delay refers to the signal propagation delay from the radiation source to the observation station. Although all the receivers observe signals synchronously, the signal arrival times are different due to the different distances to the radiation source. According to the Doppler theory, the intercepted signal frequency fl,k is given by f l , k = f c c ∼ v k T q l − p k q l − p k 2 , (7)where fc represents the signal carrier frequency, c ˜ is the speed of the electromagnetic waves. Then the observation signal's matrix corresponding to the ith trajectory is R 1 : K ( i ) = r 1 , 1 r 1 , 2 ⋯ r 1 , K r 2 , 1 r 2 , 2 ⋯ r 2 , K ⋮ ⋮ ⋱ ⋮ r L , 1 r L , 2 ⋯ r L , K ( i ) . (8) Finally, we integrate the observation signal R1:K, observation time T, trajectory state vector S1:K, and the corresponding target classification c to form our training data set. 3 THE PROPOSED DeepDTC MODEL Model architecture The architecture of DeepDTC is shown in Figure 2. The model input I = {R, Z} represents the multi-station observation signals and corresponding auxiliary information (observation time). The final output E = S ^ , c ^ is the estimation of trajectory state and classification. The DeepDTC model mainly consists of three components: signal processing component, trajectory tracking component, and multi-task learning component. The signal processing component captures and integrates the observation feature, and its output is fed to subsequent networks as part of the input. The trajectory tracking component is the core component that transforms the observation characteristics into the target state features. Finally, we utilise a multi-task learning component to combine these networks to achieve joint direct tracking and classification. In summary, the model is to learn a mapping F DTC from I to E which satisfies min F DTC L ( I , E ) = min α N t ∑ i S ^ 1 : K ( i ) − S 1 : K ( i ) 2 + 1 − α N t ∑ i D KL c c ^ = min α N t ∑ i F DTC ( 1 ) I ( i ) − S 1 : K ( i ) 2 + 1 − α N t ∑ i ∑ c = 1 M y i , c l o g 1 F DTC ( 2 ) I ( i ) , (9)where DKL (⋅) represents the Kullback-Leibler (KL) divergence, Nt and M are the number of trajectory samples and target classes, respectively, yi,c is the sign function (takes 1 when the actual class is equal to class c, otherwise take 0), and α represents the weight coefficient. FIGURE 2Open in figure viewerPowerPoint The structure of DeepDTC model Signal processing component The signal processing component consists of two modules. The first module is the Signal-Conv network. It converts the received signal sequence into a series of feature maps. The second module is the Attribute network. It incorporates auxiliary information of the observation signal into our model. CNN is a standard structure of neural networks which is widely used in image and video processing [38]. Due to the design of convolution kernels, CNN is good at processing numerical matrix data such as images. As shown in Figure 3, the CNN-based Signal-Conv network captures the feature of received signals as a series of vectors. The detailed configuration of the Signal-Conv network is shown in Table 1. In addition to the pooling process and linear connection layer, the Signal-Conv mainly comprises an initial convolutional layer and two residual blocks. Each residual block contains two convolutional layers and basic batch normalisation. We implement a local CNN method by controlling the kernel and stride size in convolution layers, and obtain K sets of signal feature vectors corresponding to the observation points. Define the input of Signal-Conv as Y i ( 0 ) = { R } i = 1 N t , where R ∈ R ( L × K ) × N × 2 is a three-dimensional numerical matrix. Specifically, the parameter dimension of the first convolution layer is (width = L × K, length = N, channel = 2), where N denotes the number of sampling points in each time slot segment, channel = 2 represents the real and imaginary parts of the complex signal. Specifically, the transformation at layer j is defined as Y i , k ( j ) = σ CNN W conv ( j ) ∗ Y i , k ( j − 1 ) + b ( j ) , (10)where Wconv and b are the learnable weight parameters, σCNN represents the corresponding activation function, and ∗ is the convolutional operation. Defining the feature dimension as dsig, we finally transform the signals to Y i ( J ) ∈ R K × d sig as the output feature of trajectory i. FIGURE 3Open in figure viewerPowerPoint Structure of signal processing component TABLE 1. Detailed configuration of the Signal-Conv network Name Configuration Output size Initial layer Conv (filter size: L × 3, feature maps: C1, stride size: L × 1) (K, length1, C1) Residual block Batch normalisation and leaky relu × 2 times (K, length2, C2) Conv (filter size: 1 × 3, feature maps: C2, stride size: 1 × 1) Batch normalisation and leaky relu Conv (filter size: 1×3, feature maps: C2, stride size: 1 × 1) Residual addition Pooling layer Batch normalisation and leaky relu and average pooling (K, length3, C2) Reshape layer Linear (K, 1, dsig) The Attribute network implements embedding layers to incorporate auxiliary information into our model. Taking the observation time as an example, the embedding layers convert each timestamp φ ∈ [Φ] to a real number space H ∈ R d attr × 1 (embedded space) by multiplying the learnable parameter matrix Wemb, as shown in the following formula, H i , k = σ emb W emb ⋅ φ i , k + b emb , (11)where dattr represents the dimension of the embedded space. Then, we can obtain the observation vector Y i , k ′ by concatenating Y i , k ( J ) and Hi,k, that is, Y i , k ′ = Concat Y i , k ( J ) , H i , k . (12) The output of the signal processing component can be expressed as Y i o = Y ′ i , 1 , Y ′ i , 2 , ⋯ , Y ′ i , K . The feature dimension of the DeepDTC model is dmodel = dsig + dattr. Trajectory tracking component Transformer [34] is originally designed as a sequence-to-sequence model [39] for machine translation but has been widely adopted in various fields including computer vision (CV) [40], audio processing [41] and even other disciplines [42]. Transformer provides new ideas for intelligent trajectory processing in complex actual scenes. In this section, we introduce the trajectory tracking component. This component models the sequence relations between trajectories and observations. The neural networks have been shown to perceive the movement laws of targets [32]. We propose to utilise the Transformer network in our trajectory tracking component. As shown in Figure 4, this component consists of two parts (encoder and decoder), which contain Lt stacks of the repeated layer structure. A multi-head self-attention mechanism and a fully connected feed-forward network (FFN) is included in each stack of the encoder and decoder. To facilitate these residual connections, all sub-layers in the model produce outputs of dimension dmodel. Define Q, K, V as the query, key, and value vectors, respectively. The self-attention can be calculated as the dot products of Q with all K in a Softmax function: Attention ( Q , K , V ) = Softmax Q K T d k V , (13)where dk denotes the dimension of K. The Softmax function is used to output a dk dimensional vector, which represents the probability of xi assigned under dk different categories: softmax θ x i = p y i = 0 | x i ; θ p y i = 1 | x i ; θ ⋮ p y i = d k − 1 | x i ; θ = 1 ∑ j = 0 d k − 1 exp θ j T x i exp θ 0 T x i exp θ 1 T x i ⋮ exp θ d k − 1 T x i (14)where θ represents the learnable parameters of the model. Moreover, multi-head attention makes the network to jointly focus on information from multiple representation subspace, that is, MultiHead ( Q , K , V ) = Concat h e a d 1 , h e a d 2 , ⋯ , h e a d h W O , (15)where headi = Attention (QW1,i, KW2,i, VW3,i), the projections W are learnable parameter matrices and dk = dv = dmodel/h. FIGURE 4Open in figure viewerPowerPoint Structure of trajectory tracking component Compared with the encoder, the decoder inserts a sub-layer called Cross-Attention, which performs multi-head attention on each stack output of the encoder. The Cross-Attention layer allows the decoder to use the classification features captured by the encoder when extracting trajectory features. At the same time, trajectory features are also used for classification in joint training. A position-wise fully connected FFN is included in every layer of the encoder and decoder, which consists of two linear transformations (W1,2 and b1,2): FFN ( x ) = max 0 , x W 1 + b 1 W 2 + b 2 (16) Since the Transformer-based trajectory tracking component is constructed without recurrent or convolution, it is necessary to introduce relative or absolute sequence position-coding to make the model use sequence order information. We use sin and cos functions of different frequencies to represent the position-code [34]: P E p o s , 2 i d = sin p o s / 100 0 2 i d / d model P E p o s , 2 i d + 1 = cos p o s / 100 0 2 i d / d model (17)where pos represents the position, and id indicates the dimension. The complete trajectory is input simultaneously into the Transformer during parallel training. To prevent subsequent information leakage, the decoder input is shifted right by one position, and the corresponding weight of self-attention sub-layers is masked. This ensures that the estimation of position i can only depend on the known output whose position is less than i. At the same time, the network can use the observation information of all historical moments when estimating the trajectory. Finally, the classification feature vector Ci is output by the encoder, and the