As the growing of the consumer market of personal drones, technologies are needed to monitor drones in the airspace. Radar has been proven to be a suitable tool for the detection and surveillance of the drone. One of the main obstacles in drone detection is the interference of the birds; since two targets share the same flight level, ground speed, and size. In this Final Y ear Project research, the impact of principal component analysis(PCA) on the classification of birds and drones is investigated and demonstrated. The whole project was built and tested on the L band staring radar dataset. Several different algorithm of PCA is implemented first and a classifier is trained based on support vector machine(SVM). The SVM classifier is used to demonstrate the impact of PCA preprocessing on classification. The impact of preprocessing is evaluated by the results such as training time cost, confusion matrix and general accuracy of classification. Finally, the different performance of PCA preprocessing algorithm is demonstrated and the reason of these result is concluded. It shows that PCA preprocessing can be an efficient way to increase the performance of the classifier when facing the radar spectrogram data. 1 Introduction In recent years, the development of drones has progressed rapidly and continues to evolve. While drones have brought many bene- fits to industries such as agriculture and environmental monitoring, concerns have been raised about their misuse. The use of drones with cameras has raised concerns over personal privacy as they can capture images or videos without consent. This becomes especially concerning when drones fly over private property. Drones can also be dangerous to the airspace, Operating drones within restricted airspace, such as near airports or government installations, can inter- fere with manned aircraft operations and create hazardous situations. In 2016, the FAA raised the alarm about drones in the airspace and received over 100 reports of unmanned aircraft flying near other manned aircraft or airports per month.[1] In a word, efficient and economical ways of surveillance the airspace and protecting key areas from drone intruding such as airports are needed. Radar is one of the most efficient technologies to monitor drones, as well as the only capable device that can offer all-weather, continuous surveillance of a certain area of airspace. Birds as a radar target, are always the main obstacle in drone detection. Birds and drones share the same flying altitude and ground speed. It is hard to tell the difference by simply looking at the 4-D data(Azimuth, Elevation, Range of time) of radar feedback. Birds unlike drones will not cause privacy or environmental issues and birds can be controlled by bird-repellent devices. Therefore one of the main challenges in designing a drone surveillance radar is the classification of birds and drones. Machine learning and deep learning techniques have been instru- mental in solving the problems of classification. The trend of researching the birds and drones classification problem is adopting machine learning or deep learning algorithms with different features of radar data. There are a few features of radar that can be utilized to classify birds and drones. In[2], researchers utilized physical fea- tures such as height and position and trained a decision tree to solve the classification problem. In 2015, a group of researchers showed the potential of using polarimetric features to distinguish birds and drones.[3] In 2020, a group of researchers provided another approach of using the trajectory differences between birds and drones for classification.[4] They evaluate features of trajectories such as cur- vature and turning angle, and then train SVM to distinguish these features. Another class of related work use visual methods for clas- sification, for instance, in [5], the researchers classified the birds anddrones by training a convolutional neural network (CNN) based on VGG-16 architecture. Despite all of the methods with different features above, a trend- ing, as well as popular way, is training the classifier based on the micro-Doppler(u-Doppler) signature. Micro-Doppler is a phe- nomenon where small, rapid movements of objects within a larger, primary movement generate additional frequency shifts in the radar signal. In the case of drones, the fast rotating blades produce the u-Doppler effect and will produce the Helicopter Rotor Modula- tion(HERM) lines on the spectrogram of drone target. As for the birds, the flapping of wings can also produce the u-Doppler effect however the frequency of flapping is nothing compared to the rota- tion of the drone’s blade.[6] HERM lines can be discovered by perform long window short time Fourier transform(STFT) to the radar data.[7] The STFT basically slice the time domain into small slots and then perform Fourier transform on these time slots. Figure 1[8] shows the example spectrogram of a bird and a drone from the same radar as this project use. The HERM line can be seen where there is shifting in Doppler frequency. As for the bird’s spectro- gram, the signal is more concentrative. Besides the spectrogram, there are a few more ways to demonstrate the u-Doppler signa- ture. The cadence velocity diagram(CVD) can be calculated by performing a fast Fourier transform(FFT) on the spectrogram, and the Cepstrogram[9] can be found by performing an inverse Fourier transform(IFT) along the Doppler axis of the spectrogram. One of the representative works using these features to train the classifier is [10], where the researcher performs dictionary learning on the CVD they extracted. In [11] the author summarizes the u-Doppler feature recognition and in a word, the state-of-the-art technique for the clas- sification of u-Doppler signatures is using spectrograms as dataset to train a deep learning model. At the University of Birmingham(UoB), research on bird vs. drone classification is conducted based on a testbed of two quantum- enabled L-band staring radars (Gamekeeper Radar). The radar can work in monostatic and multistatic modes. The dataset of this project use is from the same L-band staring radar testbed. The specific status and overview of the testbed can be found in [12]. In [13], researchers conducted several measurements with control bird and drone targets and verified the possibility of using the Gamekeeper radar as the testbed to solve the classification problem. In [14], researchers conducted measurements with drone targets and proved that the Gamekeeper radar is suitable for producing the features of drones that can be used for machine learning. Then, researchers of IET Research Journals, pp. 1–14 © The Institution of Engineering and Technology 2015 1 (a) drone spectrogram (b) bird spectrum Fig. 1 : Example Spectrogram of Bird and Drone via STFT[8] the UoB developed a method of conducting the experiment of gather- ing ground truth for supervised machine learning. In [15], the paper explains the process of how the dataset of this project is created. Based on this dataset, several machine learning and deep learning model is developed. A robust two-stage decision tree is trained in [16]. Both of u-Doppler and kinematic features from Gamekeeper radar are used in training the model. The overall precision of 2- stage trees has a true positive rate of around 0.85. A CNN is used in [17]. The CNN is trained based on the concept of transfer learn- ing. A preliminary classifier is used with the Alex-net model and then researchers retrain the classifier. The F1 score of the retrained classifier is around 0.94. In general, an F1 score of over 90% seems to be great. However, in a real scenario of radar detection of drones, a surveillance radar system like the Gamekeeper may encounter hun- dreds of thousands of targets at some moment, and most of the targets are birds. Because birds vastly exist in the environment yet drones are not. In this circumstance, a 0.05 false alarm rate shown in the article[17] will bring 50 false alarms if 10,000 birds are detected. Therefore a false alarm rate like this is not tolerable. More- over, researchers also verified the performance of CNN in low SNR scenario[18][19]. Researchers added Gaussian noise artificially and it was been proved that the performance of CNN classifier will be affected by low SNR: The accuracy of Alex-net will drop by 15% as the SNR falls from 40dB to 20dB. In general, as for the UoB radar dataset classification problem, there are improvements to be made.PCA preprocessing is a technique used to reduce the dimension of data while retaining essential information. As for the classifier, when the dimension of the example increases, the computation may arise exponentially. PCA preprocessing is a possible way to reduce some computation. The PCA works by projecting the data to a lower dimension which will have the max variance, and in many cases, a high-variance direction means meaningful information, and a low- variance direction means noise. Therefore, PCA may have the ability to reduce the noise. There are some previous research using PCA to pre-process the dataset and then train the classifier. The PCA pre- processing before a classifier such as CNN or SVM will increase the accuracy of classification. Representative work is [20]. In [21], researchers use PCA on the spectrogram and then train the SVM classifier. Although the signal they classified is some biochemical signal, this project is inspired by the idea of using PCA prepro- cessing on the spectrogram. In [22], researchers use t-SNE on the spectrogram of different types of drones and then train a SVM with low-dimension data. t-SNE is a similar algorithm which can reduce the dimension of the original dataset. Although this is a similar work to my project, there are differences. This paper used STFT spec- trogram with a short window, and t-SNE is used less often in data visualization than preprocessing. The lower dimension of t-SNE has a maximum value of 3. Therefore t-SNE is not a suitable algorithm to test the dimensional reduction impact on the classifier. In this project, several PCA algorithms are used on the Game- keeper radar dataset. Then a SVM is trained based on the low dimension data. The accuracy as well as the time cost will be measured at last as the judgement of the performance of the PCA algorithm. For each PCA algorithm, performance will be measured against different coefficient of PCA, such as the subspace dimen- sion. Different sizes of spectrograms from the dataset will be tested. The original data has a Doppler range of 1200Hz, then a reduction Doppler range to 600Hz and 300Hz will be tested. The key thing of this project is not to train a high-performance classifier model, but the verify if the PCA preprocessing can be an efficient way to increase the classifier performance on UoB staring radar dataset. SVM classifier should be initialized with same kernel function and have identical parameters for a fair comparison. Section 2 shows that how the dataset is compared and initialized. Section 3 shows algorithm and implementation of the PCA, Two Dimensional PCA(2D-PCA) and PCA based on L1-Norm(L1-PCA). Section 4 shows the performance of each algorithm and theoretical analysis. 2 Dataset Preparation This project starts with the raw time-series dataset. Each example of the raw dataset is a time series of the target which has an uncertain length between 20 to 300 time steps. Each time step corresponds to 0.28 seconds. The real dataset consists 123 time series of birds and drones. Each sample has a data format of ’mat’ file, and each mat file contains a ‘tgtData’ Matlab class. Name of the class is ’TgtGamekeeperRaw’.An example data structure of the class is: 1TypeVersion: 1 2truth:[110x22 double] 3sourceRunFolderName: 4rawData:[110x4096 double] 5frameTimes:[110x1 double] 6tgtLabel:’OpportuneBird’ Where the ’truth’ is 22 double variable containing the physical fea- tures of the target in 110 time steps, the ’rawData’ is time series. The ’tgtLabel’ indicates the type of the target(Bird or Drone). The dataset we need is spectrograms as samples, therefore the first task of this project is to generate them. The script for generating dataset is ’Gen_Dataset.m’ which can be found in the appendix. By calling the ’simplefft’ function we can transform the time series raw data to the spectrogram, and by setting the ’binsToKeep’ input of this function, we can select the Doppler IET Research Journals, pp. 1–14 2 © The Institution of Engineering and Technology 2015 shift range to keep of the spectrogram. This project use 300bins, 600bins and 1200bins as Doppler shift range separately. On spec- trogram, the body return of the radar target is concentrated around 0Hz, and the u-Doppler signature is spread around a few hundred Doppler bins. Hence this comparison is to verify the performance of the classification model against the lack of u-Doppler signatures. If the model don’t lost too much accuracy while the spectrogram width dropping, then reduce of the width can be considered as a method of optimize the time performance of the model. Figure 2 shows two examples of a bird and a drone in the dataset. Figure 2 100 200 300 400 500 600 700 time/steps200 400 600 800 1000 1200Doppler Bin (zero doppler shift at 600) -120-100-80-60-40-200 (a) drone spectrogram in the Gamekeeper dataset 20 40 60 80 100 time/steps200 400 600 800 1000 1200Doppler Bin (zero doppler shift at 600) -80-60-40-2002040 (b) bird spectrum in the Gamekeeper dataset Fig. 2 : Example Spectrogram of Bird and Drone of Gamekeeper Dataset tells clearly the difference between a bird spectrogram without the u-Doppler effect and a drone spectrogram with HERM lines in the dataset. Notice that these examples have a Doppler range of 1200Hz. Notice that the time length for the drone spectrogram is 800 time steps and 110 time steps for the bird spectrogram. The PCA algorithm requires a uniform length for each sample, therefore the next step of the script is trimming these spectrograms. The origi- nal target spectrogram was sliced into pieces with identical lengths. The output of the ’Gen_Dataset.m’ script is another mat file with two variables: ’label’ and ’fftfrac’. The ’label’ is simply a string which indicates the type of data and the ’fftfrac’ is a spectrogram corresponding to the label with size of ’Doppler width(300, 600, 1200)’ ×’20 time steps’. The dataset with 3006 samples is gen- erated. A Python module ’scipy.io’ is installed and the ’loadmat’ function can be used to read the mat file and convert the ’fftfrac’ into a numpy array. The specific code can be seen in the ’dataset.py’ in the appendix. The ’train_test_split’ function from the ’sklearn.model_selection’ package is used to split the 3006 samples into test set and training set. The ratio used in this project is 80% of the training set, therefore atraining set of 2404 spectrograms and a test set of 602 spectrograms is generated. 3 PCA Algorithms and Implement 3.1 Conventional PCA Implementation PCA is originally brought up my Karl Pearson in 1901 and is always a important technique when facing large data. In this project, conventional PCA means original way of selecting the principal components of high dimensional data by maximize the variance of projected data. The word ’conventional’ is used to distinguish this method with its extension algorithms in the next sections of the project. All PCA algorithms have the same basic idea and can be deduced in two ways: 1. The subspace has the ’maximize re-factorize’, which means that the distance from original samples to the new hyperplane of the subspace should be the closest; 2. The subspace has the ’max- imize separable’, which means the projected data have maximum of variance. We deduced the conventional PCA via ’maximize re-factorize’:[23] Assume the training set of our data is: Xi:i={1,2,···, n}, where nis the number of data in the test set. The data has the dimen- sion of X∈ Rd×nwhere dis the dimension of original data. Each column of the Xrepresent a reshaped 1-D spectrogram. The pro- jection matrix of PCA is W={w1,w2···,w′ d}, where d′is the dimension of the subspace. The projection matrix has a dimension of W∈ Rd×d′Projected data is Z,where Z=WT·X. In this case, the data point in original high dimension xihas a projected point zi= (zi1;zi2;. . .;zid′), where zij=wT jxiis the jth dimension coordinate of the sample point in the subspace. Therefore, the recon- structed Xcan be expressed by: ˆxi=Pd′ j=1zijwj, and then we set the objective function of conventional PCA as Jobj. The distance between reconstructed data points to the original data point is: mX i=1 d′X j=1zijwj−xi 2=mX i=1∥Wzi−xi∥2 2 =mX i=1(Wzi−xi)T(Wzi−xi) =−mX i=1zT izi+const =−mX i=1tr zizT i +const ∝ −tr WT XXT W(1) where const is non-irrelevant constant value and trrepresents the trace of matrix. According to the basic idea of PCA, minimum value of the distance in equation(1) needs to be found, hence the objective function is: Jobj= max Wtr WTXXTW s.t.WTW=I(2) Problem (2) is a constrained optimization can be solved by the Lagrangian method. The Lagrangian function of the problem is: L(W,Θ) = tr WTXXTW −D Θ,WTW−IE = tr WTXXTW −tr ΘT WTW−I (3) Where Θ∈Rd′×d′is the Lagrangian multiplier matrix and the ele- ments of which are the unknown Lagrangian multiplier. Consider IET Research Journals, pp. 1–14 © The Institution of Engineering and Technology 2015 3 the constrain of object function WTW=I, the Lagrangian matrix should be a diagonal matrix which can be written as: Λ = diag ( λ1, λ2, . . . , λ d′)∈Rd′×d′(4) where the λis the eigenvalue of the covariance matrix of data. Substitute the equation(4) into (3), we have: L(W,Λ) =−tr WTXXTW + tr ΛT WTW−I (5) An optimal(maximum) value of the trace can be found when the derivative of the Lagrangian function against to θis zero, let ∂L(W,Λ)/∂W=0, and thus we have: XXTwi=λiwi, i= 1,2, . . . , d′(6) Substitute equation(6) into the objective function(2) thus we have: Jobj= argmaxd′X i=1λi (7) Therefore, the projection matrix of optimal convention PCA can be derived by combining the largest d′eigenvalues of the covariance matrix XTX. When implementing the conventional PCA algorithm, the steps are followed: 1. Centralized test set and training set by ’Stan- dardScaler’ function from the ’sklearn preprocessing’ package; 2. Calculate the covariance matrix of the test set and perform singu- lar value decomposition(SVD) on it to calculate the eigenvalues, and then generate the eigen matrix. The Python class ’PCA’ from ’sklearn.decomposition’ package can be used here. The class ’PCA’ has two methods that are important in the project, the ’PCA.fit’ method can calculate the projection matrix of a certain test set and the ’PCA.transform’ method can project the original data on the sub- space; 3. Project the data onto the subspace; Low dimension data by conventional PCA is so far produced. The test is repeated with changing d′varies from 5 to 60. The rela- tion of classification accuracy with the dimension of subspace should be found out. The whole test is conducted on 3 groups of different Doppler shift width datasets. The specific code of conventional PCA can be found in the appendix. 3.2 2D-PCA Implementation In the previous section, a conventional PCA is adopted, where each spectrogram is reshaped to 1-D and is considered as a column in X. Reshaping data matrices to 1-D vectors may cause some issues on losing the spatial information. 2DPCA is developed by [24] for the application of facial recognition. In their work, conventional 1-D PCA do not work well with extracting the spatial feature of human face images such as human expressions. It is proved to be an effec- tive way of using 2DPCA on the human face dataset. 2DPCA takes into account the spatial arrangement of pixels in an image or the structure of a matrix. This means that it considers how nearby pix- els interact and captures spatial patterns, edges, textures, and other local features. However, it’s important to mention that 2DPCA is not always the more optimal solution than traditional PCA. Its advan- tages become prominent in scenarios where the spatial arrangement of data matters, such as image analysis on facial features. In this section, the performance impact of 2DPCA on our Game- keeper dataset is investigated. In 2DPCA, rather than proceeding original data matrix of X∈ Rd×n, the ith spectrogram in the dataset Xiis directly projected by the projection matrix W. Let the spectrogram has the dimension of: Xi∈ Rhw, where the hand wis the height and width of each spectrogram(image). Similar to thededuction from previous section, the object function of 2DPCA can be written as: J(W) = tr ( Sx) (8) Where the S(x)is the covariance matrix of the projected low dimensional matrix. Considering the relationship of projection, the covariance matrix of projected data can be expressed as: Sx=E(Z−EZ)(Z−EZ)T =E[WTX−E(WTX)][WTX−E(WTX)]T =E[(W−EW)X][(W−EW)X]T(9) Where Erepresent the centralization of data and Z, W is the projected and projection matrix. Combining the equation (9) with objective function (8), we have: J(W) = tr ( Sx) =WTh E(X−EX)T(X−EX)i W =WT1 nnX j=1 Xj−XT Xj−X W(10) Consider the1 nPn j=1 Xj−XT Xj−X as the image scat- ter matrix G. It’s straightforward that the Jhas a maximum value when the Wis the listing of largest eigenvalue of G. In a word, the 2DPCA substitute the covariance matrix of original data in con- ventional PCA to the covariance matrix of the average value of all spectrograms. In the implementation of 2DPCA, no ready-made algorithm pack- age is used and the code is designed based on the deduction from previous content. The pseudo-code of 2DPCA for this project is shown in algorithm 1: Data: Spectrogram Dataset, dimension of subspace d′ Result: Projected matrix for 2DPCA Z initialization(Do not centralized the data); while i<ndo G=1 nPn j=1 Xj−XT Xj−X ; Eigenvalue Decomposition G; Get largest d′Eigenvalue; W= (w1, w2,···, wd′); Z=WTX; i++; end Algorithm 1: 2D-PCA The eigenvalue decomposition step can be achieved by the ’linalg.eig’ method in the numpy package. The rest of the steps in the pseudo-code are relativity simple. The test on 2DPCA repeated from d′= 5 tod′= 60 . Each projected matrix is used to train an SVM classifier. The accuracy is then calculated to evaluate the performance of 2D-PCA. 3.3 PCA with L1-Norm Implementation L1-Norm Based PCA is first brought up in [25]. Generally speaking, the implementation of L1 distance in the optimization of object func- tion in machine learning can increase the robustness of the model. There are a few reasons for the robustness: firstly, the L2 norm directly uses the Euclidean Distance, which is the square value of data points’ deviations from the central point. This means that the impact of large positive and negative deviations, which is consid- ered abnormal samples in the dataset, are amplified. This lack of sensitivity to outliers is why the L2 norm is not considered robust in the presence of extreme values. However, the L1 norm treated the distance equally, which is more robust to abnormal points. Secondly, when minimizing the L1 norm, the solutions tend to be sparse, which IET Research Journals, pp. 1–14 4 © The Institution of Engineering and Technology 2015 means that many components of the solution vector are exactly zero. Outliers can have a reduced effect on the solution because their influence on the objective function is limited by the absolute value penalty. The outlier in this project could mean the data points that significantly deviate from the overall pattern or distribution of data in an image or a dataset. The reason of the outlier could be noise, mislabelled data or a unique target. Moreover, the L1-norm in PCA brings sparse to the subspace, more zeros in the feature space and could make the main components contain more valuable information than conventional PCA. The L1-Norm PCA problem is similar to the conventional L2- Norm PCA. Only the distance is measured in Manhattan distance rather than Euclidean distance. The distance is expressed by: mX i=1 d′X j=1zijwj−xi 1=mX i=1∥Wzi−xi∥1(11) Similar to the proof in section 3.1, the object function of L1-PCA can be deduced by maximize the dispersion of reconstruct images. J= arg max W WTX 1= arg max WnX i=1d′X k=1 dX j=1wjkxji 1 s.t.WTW=Im. (12) Notice that in the conventional PCA and 2DPCA, the optimal value of projection matrix Wis not independent against d′. In other word, in the conventional scenario, the subspace with lower dimension is always the subset of the larger ones, because the rank of eigenvalue is fixed. However, in L1PCA, the optimal ith projection vector wi will vary as d′changes. Therefore it is not easy to find a global maximum value of the objective function. In [25], a sub-optimal solution is given based on the similarity. The optimal projection matrix when d′>1can be approximate by the optimal projection matrix when d′= 1. A greedy search algorithm in [25] is proposed based on the fixed point iteration method. More efficient algorithm has been developed in [26]. The [26] provides a non-greedy search algorithm to solve the problem (12) with iterative alternating optimization and proved to have a better accuracy on classifier. The method used in this project is a optimal algorithm using iter- ation via bit-flapping based on the work of [27]. The problem of L1PCA showed in equation (12) can be expressed by: [28] Bopt= argmax B∈{±1}N×K∥XB∥∗,(13) and W=U(XBopt) (14) Where U(X)represent the Umatrix in the SVD of X=UΛV. B∈ {± 1}N×Krefers to a binary matrix where all elements is either -1 or 1. [27] provide a proof that the equation(13) can be written in a more simple form of: Bopt= argmax B∈{±1}N×K∥YB∥∗, (15) where Y= ΛVhas a dimension less than Xand will same some computation. Then the optimal value of projection matrix can be calculated by iteration of finding a optimal value of B. Before the iteration, a array of Lwith the length of nd′indicates all the element that haven’t been flipped. In each iteration, the algorithm go through all elements of binary matrix Bby flipping them. Changing in L1 norm will be measured after each flipping and the maximum value of increasing will be record. The optimal flipping element corresponding to the maximum L1 norm increase will beupdate to next iteration. The principal idea of L1 norm maximiza- tion via bit flipping is shown in the pseudo code of algorithm 2. Data: Spectrogram Dataset Xdimension of subspace d′ Result: Projection matrix for L1PCA W initialization(Centralized the data); (U,Λd×d,V)←svd(X); Y←ΣV⊤,v←[V]:,1; Initialize B,B= sgn v1⊤ K ; while not convergence, i<nd’ do while j<nd’ do Bit Flapping; j++; end i++; end ˆUd×d′,ˆΣd′×d′,ˆVd′×d′ ←svd (XBbf); W←ˆUˆV⊤ Algorithm 2: L1-PCA Convergence proof and computation cost are further discussed in [27]. The source code of L1PCA is shown in the appendix. Algorithm 2 implements modification on a function ’l1pca_sbfk’, which is from an open-source program. The test on L1PCA is repeated from 5 to 60 main component extractions. The trial is per- formed with the spectrogram dataset of 600Hz Doppler range due to the limit of hardware(The 1200Hz spectrograms are too large for the RAM). The performance of L1PCA can be analyzed to tell the differences between these algorithms. SVM classifier is trained by the space data with the same kernel function as the previous section. 3.4 Testbed of SVM Classifier The main goal of the SVM algorithm is to find a hyperplane that can correctly separate data points of different classes with maximum margin. In our binary classification problem of birds and drones, the SVM produce a boundary between two clusters of data point in the subspace after PCA. The SVM Classifier testbed is implement by ’sklearn.svm’ package. A typical SVM classifier can be set with the hyper-parameters shows below: 1"C": loguniform(1e3, 1e5), 2"gamma": loguniform(1e-4, 1e-1), 3kernel: "rbf" 4class_weight: "balanced" where the "C" controls the regularization strength, and "gamma" is a parameter for the radial basis function (RBF) kernel. The Sup- port Vector Classification (SVC) class is instanced with a radial basis function (RBF) kernel and "balanced" class weights. The RBF kernel is often used for SVMs when dealing with non-linearly sepa- rable data, which could be suitable for image classification, and the "balanced" class weights help address class imbalance issues in the training data. It needs to be noted that the selection of parameters in SVM is not intended to create an "optimal" classifier that can offer greater accuracy. The SVM model is just a shallow machine learning model when compared to a deep learning algorithm, and hence is only used as a testbed for the PCA algorithms. The SVM model can be trained quickly and efficiently. What needs to be assured is that the parameters remain the same for each trial of PCA. 4 Classification Result and Analysis 4.1 Principals of Trials The performance of the PCA is measured by these several met- rics: accuracy, precision, recall, F1-score and confusion matrix. The macro accuracy is the simple average of the accuracy calculated for each class individually. It treats all classes equally and doesn’t take IET Research Journals, pp. 1–14 © The Institution of Engineering and Technology 2015 5 class imbalances into account. It’s calculated by summing up the accuracies for each class and then dividing by the total number of classes; The precision of each class measures the ratio of correctly predicted positive data points to the total data points predicted as positive; Recall is the sensitivity and the true positive rate, it’s the ratio of number of correctly prediction to the total actual positive predictions. The calculation of the precision and recall is shown in equation (16). Precision =Correctly Predict Positive All Positive Recall =Correctly Predict Positive All Correct Prediction(16) The F1 score is a metric that combines both precision and recall into a single value, providing a balanced view of a model’s perfor- mance in a binary or multi-class classification scenario. F1 score is particularly useful when the precision and recall are both need to be considered. The calculation of the F1 score is shown in the equation (17). F1 =2×Precision ×Recall Precision +Recall(17) Confusion matrix of a model is a more intuitive way of demonstrat- ing the performance of the algorithm. A typical confusion matrix is a square matrix with rows and columns corresponding to the classes in your dataset. The main components of each cell in the confusion matrix are true positive(Correctly predicted in the class), true nega- tive(correctly predicted outside the class), false positive(incorrectly predicted in the class) and false negative(incorrectly predicted out the class). Time cost as another metric is considered as well. The time for dimension reduction and training an SVM classifier is mea- sured. Ideally, PCA dimension reduction will save the computation of SVM. In the project, 10 instantiations(trails) of the SVM model are trained and three PCA preprocessing algorithms are used on them: •without-PCA300: The comparison group of the SVM model trained by original data without PCA. Original data are spectrograms which have a Doppler width of 300Hz. •without-PCA600: The comparison group of the SVM model trained by original data without PCA. Original data are spectrograms which have a Doppler width of 600Hz. •without-PCA1200: The comparison group of the SVM model trained by original data without PCA. Original data are spectrograms which have a Doppler width of 1200Hz. •conv-PCA300: The test group of the SVM model trained by orig- inal data with conventional PCA. Original data are spectrograms which have a Doppler width of 300Hz. PCA use 5 to 60 components.•conv-PCA600: The test group of the SVM model trained by orig- inal data with conventional PCA. Original data are spectrograms which have a Doppler width of 600Hz. PCA use 5 to 60 components. •conv-PCA1200: The test group of the SVM model trained by original data with conventional PCA. Original data are spectro- grams which have a Doppler width of 1200Hz. PCA use 5 to 60 components. •2D-PCA300: The second test group of the SVM model trained by original data with 2DPCA. Original data are spectrograms which have a Doppler width of 300Hz. PCA use 5 to 60 components. •2D-PCA600: The second test group of the SVM model trained by original data with 2DPCA. Original data are spectrograms which have a Doppler width of 600Hz. PCA use 5 to 60 components. •2D-PCA1200: The second group of the SVM model trained by original data with 2DPCA. Original data are spectrograms which have a Doppler width of 1200Hz. PCA use 5 to 60 components. •L1-PCA600: The third group of the SVM model trained by orig- inal data with L1PCA. Original data are spectrograms which have a Doppler width of 600Hz. PCA use 5 to 60 components. 4.2 Demonstration and Analysis of Result Table 1 shows metrics measured in the trials of this project. The con- ventional PCA, 2DPCA and L1PCA data are gathered from the trial where the 60 largest principal components are used as subspace. The trial is conducted two times. For the first time, CPU computation is implemented with traditional Numpy and Scipy packages. The sec- ond time, GPU is involved by cupy packages in solving the matrices during 2DPCA and L1PCA. In Table 1, brackets in the column of time metric refer to time consumption when the projection matrices are solved by GPU. The metrics are tested with a hardware platform of "Intel 6-cores i7" with a main frequency at 2.6GHZ plus "Nvidia GTX 1650" with a CUDA computation ability of 7.5. 4.2.1 Accuracy Analysis: Comparing the metrics of the SVM classifier without PCA preprocessing, it shows a trend of increasing macro accuracy when the width of Doppler frequency increases. The metric of average accuracies shows that, for an SVM classifier, most of the features that can distinguish the birds and drones lie within the u-Doppler range of -300Hz to 300Hz, since the accuracies are the same for the last two trials. The confusion matrix of SVM without PCA of 1200Hz range is shown in Figure 3. The figure shows that the classifier has a higher chance of predicting a bird target as a drone. The classification accuracy of SVM towards the original data may be a mediocre result and less competitive to advanced classifiers. However, it is only a benchmark of algorithm performance. As for the conventional PCA algorithm. The metrics show that the performance of SVM with PCA dimension reduction is close to the Table 1 Metrics Measured by 12 Trials, PCA reconstruction with 60 main components Metric Time for PCA Time for SVM Precision Precision Recall Recall F1 score F1 score Avg Accuracy Class of Target: (s) (s) birds drones birds drones birds drones without-PCA300 N/A 521.71 0.73 0.83 0.77 0.79 0.75 0.81 0.78 without-PCA600 N/A 1209.85 0.90 0.87 0.80 0.93 0.85 0.90 0.88 without-PCA1200 N/A 2185.63 0.83 0.92 0.89 0.87 0.86 0.89 0.88 conv-PCA300 0.54 22.54 0.57 0.89 0.91 0.52 0.70 0.65 0.68 conv-PCA600 0.90 11.83 0.76 0.95 0.94 0.79 0.84 0.86 0.85 conv-PCA1200 1.74 15.87 0.76 0.94 0.93 0.79 0.84 0.86 0.85 2D-PCA300 (0.94) 164.40 0.66 0.84 0.82 0.69 0.73 0.76 0.75 2D-PCA600 (1.40) 153.90 0.67 0.81 0.75 0.73 0.71 0.77 0.74 2D-PCA1200 (3.44) 145.84 0.66 0.84 0.82 0.70 0.73 0.76 0.75 L1-PCA600 19352 24.02 0.88 0.91 0.87 0.91 0.88 0.91 0.90 IET Research Journals, pp. 1–14 6 © The Institution of Engineering and Technology 2015 Fig. 3 : Confusion Matrix for Trial ’without-PCA1200’ SVM trained by the original data, with only a few percent drop in the average accuracy. However the trial with 300Hz wide u-Doppler spectrograms is different, the performance of it is extremely poor. The reason for this may be the effective principal components lie in the 150 to 300Hz Doppler shift. Figure 4 is the confusion matrix of the trial ’conv-PCA1200’. Generally speaking, the conventional Fig. 4 : Confusion Matrix for Trial ’conv-PCA1200’ PCA has a better Recall on bird classification than without PCA, yet the F1 scores and macro accuracy are nearly the same. Moreover, the conventional PCA preprocessing in this case will make the SVM more sensitive to the ’edge-spectrogram’ data(part of a spectrogram with Doppler shifting larger than 150Hz). Conventional PCA with the Gamekeeper dataset will bring misclassified drones as birds, as shown in Figure 4. The performance of 2DPCA is not ideal enough compared to its performance on other datasets[24]. The accuracy of SVM trained with 2DPCA preprocessing is 10% lower than conventional PCA implementation. The F1 score of each class is poor in the 2DPCA as well. Now from the result of this project, we can say that the 2DPCA is not suitable for preprocessing the Gamekeeper dataset, however, the reason for this could be varied. One possible reason could be that the 2DPCA is focusing on the two-dimensional feature of the images however, the spectrograms of radar targets contain more important features in only one direction. For instance, 1-D PCA treats the spec- trogram as an array and important features for classification such as HERM lines are parallel to the time-step axis. Therefore using 2DPCA on the radar dataset might just not correspond to the nature of the dataset. Another possible reason is different preprocessing techniques have varying sensitivities to noise and outliers. If the radar data contains noise or outlier points that are not handled wellby 2DPCA, it could negatively impact its performance. In a word, the effectiveness of preprocessing techniques can be highly depen- dent on the characteristics of the data. The structure, noise level, distribution, and other features of spectrogram data might be bet- ter suited for conventional PCA rather than 2DPCA. The reason for this result could be complicated and sophisticated, yet a valuable outcome is concluded, which is, that the 2DPCA is not suitable in this scenario and may not be suitable for any model that is trained with the Gamekeeper dataset. Figure 5 shows the confusion matrix from trial ’2D-PCA1200’. It can be seen from it that the precision is Fig. 5 : Confusion Matrix for Trial ’2D-PCA1200’ poor. Moreover, the 2DPCA algorithm has performance nearly the same when the spectrograms in the dataset have different widths, which indicates that the 2DPCA will make SVM less sensitive to the ’edge-spectrogram’. As for L1PCA performance. Comparing the ’L1-PCA600Hz’ trial with other trials with a width of 600Hz shows that the L1-PCA has the best performance among the three algorithms. Not only does the L1-PCA have the highest F1 scores and macro accuracy, but it also shows the balance of precision and recall. The precision and recall of the two classes are both nearly 0.9. Figure 6 shows the confusion matrix of this trial. The effectiveness of the L1-PCA preprocess- Fig. 6 : Confusion Matrix for Trial ’L1-PCA600’ ing could be because noise and outlier exist in the original dataset, and the robustness of L1-PCA boost the performance of the model. Moreover, L1-PCA encourages sparse solutions, meaning that some components of the resulting representation can be exactly zero. This property can be advantageous when dealing with high-dimensional data where not all features are relevant. Sparse solutions might lead IET Research Journals, pp. 1–14 © The Institution of Engineering and Technology 2015 7 (a) Conventional PCA (b) 2D-PCA (c) L1-PCA Fig. 7 : Accuracy vs. Number of Components of difference PCA algorithm preprocessing to better feature selection, making the data more informative for classification. Overall, the success of L1-PCA could be attributed to a combination of its ability to handle outliers, its sparsity-inducing nature, and its capacity to extract informative features from the radar spectrogram data. The figure 7 shows the accuracy changing against the number of main components in the algorithm. In figure 7, it shows that the performance of L1-PCA is better than other algorithms whatever number of d′is chosen. The conventional PCA shows a higher accu- racy as the number of components rises, and 2D-PCA and L1-PCA show a relatively uniform value of accuracy against the number of components. It means that in 2D-PCA and L1-PCA, the information about classification is concentrated in the first several components. As for the conventional PCA which has a larger ’slope’, the rest of the components contain some of the information and adding them into the SVM will boost the performance. Therefore the efficiency of 2D-PCA and L1-PCA is higher. 4.2.2 Time Consumption Analysis: Noticing that the time consumption metric in Table 1 is not a precise reflection on the com- putation of the algorithm, because the hardware performance is not always stable. Yet comparisons and some conclusions can still be made. The Training time for SVM without a PCA preprocessing cost 500 to 2000 second. In the rest of the trials, this step will only cost less than 200 seconds. Especially for L1-PCA with 600Hz width, it boosts the time cost for training SVM over 10 times faster compared to no-PCA with 600Hz width. For the conventional and 2D PCA algorithms, the cost of time is very limited. However, the L1-PCA algorithm has a heavy computation of over 19000s. Using L1-PCA that costs 19000s to trade for 10 times faster SVM training seems to increase the computation, however, the SVM model is only a benchmark and the classifier model in the real appli- cation in radar result processing could be much more complicated than an SVM model and larger dataset. Comparing the L1-PCA and conventional PCA, there is a trade-off between computation and performance. 5 Conclusion This project compares the impact of three PCA algorithms on the performance of the classifier of the UoB Gamekeeper radar dataset. The L1-PCA algorithm is proved to be the optimal algorithm for preprocessing the spectrogram dataset due to its robustness. The Conventional and L1-PCA algorithms not only reduce the complex- ity of the dataset used to train machine learning models but also increase the accuracy of models. 2DPCA is not suitable for this par- ticular spectrogram dataset when facing a classification problem of birds and drones. PCA is well known for its anti-noise ability, and future works could explore the impact of the PCA algorithm when the dataset is corrupted by noise. More PCA algorithms are still waiting to be implemented with the Gamekeeper dataset.6 Acknowledgments This paper is a report on the University of Birmingham MSc Pro- gram Final Year Project. The project was came up with and under the supervision of Dr Michail Antoniou. The author wants to express his thanks to Dr Mohammed Jahangir and Me Daniel White, who provided instruction on this work. IET Research Journals, pp. 1–14 8 © The Institution of Engineering and Technology 2015 7