PCA, LDA, and SVD: Model Tuning Through Feature Reduction for Transportation POI Classification
By: Pranav Thaenraj
Tags: Feature Engineering, Classification, Feature Reduction, Data Science, Machine Learning

Feature Reduction Using PCA

PCA is a dimension reduction method that takes datasets with a large number of features and reduces them to a few underlying features. PCA finds the underlying features in a given dataset by performing the following steps:

1. Calculate the covariance of the matrix of features

2. Calculate the eigenvectors and eigenvalues of the covariance matrix

3. Sort the eigenvectors in descending order by the eigenvalues

4. Choose the first k eigenvectors. This k value will be the k new dimensions

5. Transform the original data into k dimensions

This article provides a deep dive into the math behind PCA: PCA Article

The sklearn PCA package performs this process for us.

In the snippet of code below we are reducing the 75 features that the initial dataset has into 8 features.

from sklearn.decomposition import PCA pca = PCA(n_components=8) Principal_components=pca.fit_transform(transportation_df.drop([‘location_name’, ‘Class’], axis=1)) pca_df = pd.DataFrame(data = Principal_components, columns = [‘PC 1’, ‘PC 2’, ‘PC 3’, ‘PC 4’, ‘PC 5’, ‘PC 6’, ‘PC 7’, ‘PC 8’]) pca_df.head(3)

This snippet serves to show the optimal number of features for the feature reduction algorithm to fit into. The scree plot shows that the ideal value is n = 2 , however, we are going to continue to use n = 8 features for the remaining analysis since it actually provides slightly better accuracy for the classifiers below than n = 2 features

PC_values = np.arange(pca.n_components_) + 1 plt.plot(PC_values, pca.explained_variance_ratio_, ‘ro-’, linewidth=2) plt.title(‘Scree Plot’) plt.xlabel(‘Principal Component’) plt.ylabel(‘Proportion of Variance Explained’) plt.show()

Adding class column back to the PCA dataframe

PCA_df = pd.concat([pca_df, transportation_df[‘Class’]], axis = 1) PCA_df.head()

Plotting PC1 and PC2 as a scatter plot with the different classes classified

PC_0 = PCA_df.where(PCA_df[‘Class’] == 0).dropna()[[‘PC 1’,’PC 2']] PC_1 = PCA_df.where(PCA_df[‘Class’] == 1).dropna()[[‘PC 1’,’PC 2']] PC_2 = PCA_df.where(PCA_df[‘Class’] == 2).dropna()[[‘PC 1’,’PC 2']] plt.scatter(PC_0[‘PC 1’],PC_0[‘PC 2’]) plt.scatter(PC_1[‘PC 1’],PC_1[‘PC 2’]) plt.scatter(PC_2[‘PC 1’],PC_2[‘PC 2’])

Classification Algorithms Using PCA

The below snippets will show how to use the Gaussian Naive Bayes, Decision Tree, and the K-Nearest Neighbors Classifiers with the reduced features.

splitting data into training and testing data

x_cols = [] for item in list(PCA_df.columns): if(item != 'Class'): x_cols.append(item) X = PCA_df[x_cols] y = PCA_df['Class'] X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

Gaussian Naive Bayes Classifier:

gnb = GaussianNB().fit(X_train, y_train) gnb_predictions = gnb.predict(X_test)

The feature reduction to n = 8 features seems to help the overall model accuracy of the Gaussian Naive Bayes model. The prediction rate is slightly better than the accuracy of the original dataset ~26%. The model predicts actual Airports as a predicted Train at a rate that is slightly higher for the PCA dataframe than the initial dataframe (.770 vs .802). The model correctly predicts train stations very well but performs slightly worse than the initial data (.940 vs. .968). This model does slightly worse in correctly predicting the bus stations (.009 vs. 0.00), but this decrease in accuracy leads to the model incorrectly predicting all records that are bus stations. The model predicts Airports at an accuracy that is double that of the original dataset (.59 vs .120). We can see from the above heatmap that this particular that most of the records have been classified as Train stations.

Decision Tree Classifier:

dtree_model = DecisionTreeClassifier(max_depth = 3).fit(X_train, y_train) dtree_predictions = dtree_model.predict(X_test)

The Decision Tree model performs slightly worse overall with the PCA dataframe than it does with the original dataframe (75% vs 73.93%). The model performs very well in correctly predicting Airports and in fact, does better than the original data did with this particular model(.958 vs .912). The model does not do well with predicting bus stations similar to the way the model performs with the original dataset. The model predicts Train stations correctly at a rate that is lower than that of the original dataset (.387 vs. .553). This model tends to predict most records as airports.

K-Nearest Neighbor :

knn = KNeighborsClassifier(n_neighbors = 22).fit(X_train, y_train)

The K-Nearest Neighbors algorithm has a good overall accuracy and performs as well as the algorithm did with the original dataset. The model predicts the airport records accurately at a rate that is similar to that of the original dataset(.931 vs. .937). The model accurately predicts the bus records and the train records at the same rate as it did with the original data (.027 vs. .027, .180 vs. .180). Most of the data is classified as it was before the PCA algorithm was applied and thus it seems that for this particular model it may not be the best choice to reduce the dimensions of the features using PCA.

From these three models and their results, we can see that the PCA dataframe doesn't increase the accuracy of models such as the Decision Tree and the K-Nearest Neighbors. The only model that sees an increase is the Gaussian Naive Bayes model. This could be attributed to the model not considering the correlations between the features, this issue is resolved with the principal components- since the principal components take into account the correlations between the original features. Thus this accommodation through the principal components leads to a slightly better accuracy.

Photo by Anthony Intraversato on Unsplash

Feature Reduction using LDA

Linear Discriminant Analysis is yet another dimension reduction algorithm. Here is a deep dive into the LDA algorithm: LDA Article

LDA works through the following steps:

1) Calculate the distance between the mean of different features — this is known as the between feature variance

2) Calculate the distance between the mean and sample of each feature — this is known as the within feature variance

3) Construct the lower-dimensional space to maximizes the between feature variance and minimize the within feature variance

The sklearn LinearDiscriminantAnalysis package performs this analysis for us very conveniently

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis lda = LinearDiscriminantAnalysis(n_components=2, solver=’svd’) X_lda = lda.fit_transform(transportation_df.drop([‘location_name’, ‘Class’], axis=1), transportation_df[‘Class’]) lda_df = pd.DataFrame(data = X_lda, columns = [‘PC 1’, ‘PC 2’]) lda_df.head()

Adding the Class column to the LDA dataframe:

lda_df = pd.concat([lda_df, transportation_df[‘Class’]], axis = 1) lda_df.head()

The snippet below plots the different classes in the dataset as a function of PC 1 and PC 2. From the graph that is displayed, we can see that the dimension reduction that is resulted from the LDA analysis is much different from that of the PCA analysis. Let’s see the effectiveness of this particular dimension reduction algorithm

PC_0 = lda_df.where(lda_df[‘Class’] == 0).dropna()[[‘PC 1’,’PC 2']] PC_1 = lda_df.where(lda_df[‘Class’] == 1).dropna()[[‘PC 1’,’PC 2']] PC_2 = lda_df.where(lda_df[‘Class’] == 2).dropna()[[‘PC 1’,’PC 2']] plt.scatter(PC_0[‘PC 1’],PC_0[‘PC 2’]) plt.scatter(PC_1[‘PC 1’],PC_1[‘PC 2’]) plt.scatter(PC_2[‘PC 1’],PC_2[‘PC 2’])

Classification Algorithms Using LDA

The below snippets will show how to use the Gaussian Naive Bayes, Decision Tree, and the K-Nearest Neighbors Classifiers with the reduced features.

splitting data into training and testing data

x_cols = [] for item in list(lda_df.columns): if(item != 'Class'): x_cols.append(item) X = lda_df[x_cols] y = lda_df['Class'] X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

Gaussian Naive Bayes Classifier:

gnb = GaussianNB().fit(X_train, y_train) gnb_predictions = gnb.predict(X_test)

Despite the stigma behind the Gaussian Naive Bayes model being inefficient because of its inability to consider the features of a dataset as dependent of one another and instead considering them and separate and unrelated features (hence the naive nature of the model), the model seems to perform very well after on the dimension reduced data. the model performs very well in predicting the true Airport records (.973) and does better at predicting the true bus stop records(.064 vs. .009). The model does much worse with correctly classifying the true train records (.968 vs. .035). Unlike the model’s performance with the original data, the model now predicts most values as Airports. The reason for this model’s predictions could be attributed to the imbalance in the data. Since the data mostly consists of airport records, the model could tend to predict more records as airports

Decision Tree Model:

dtree_model = DecisionTreeClassifier(max_depth = 3).fit(X_train, y_train) dtree_predictions = dtree_model.predict(X_test)

The Decision Tree model using the LDA dataframe performs slightly worse than the model when using the original dataframe (.703 vs. .75). The model is slightly worse at accurately predicting the true airport records (.91 vs .9). The model is unable to predict any of the true bus stop records with either dataframe and performs significantly worse with correctly classifying the true train station records (.292 vs. .553). This model shows a tendency of predicting most records as Airports, which could be attributed to the unbalanced nature of the data.

K-Nearest Neighbors:

knn = KNeighborsClassifier(n_neighbors = 22).fit(X_train, y_train)

The K-Nearest Neighbors classifier performs better with the dimension reduced dataframe than the original dataframe (.703 vs .673). The model is slightly worse at predicting the true airport records with the LDA dataframe in comparison to the original dataframe (.927 vs. .937), but it does equally well with correctly predicting the correct bus stop records. The model performs slightly worse with predicting the true Train station records (.173 vs. .180). While the model using the LDA dataframe performs equally or slightly worse than the original dataframe, the accuracy increases since the rate of incorrectly predicted records are better distributed with the LDA dataframe than in the case of the original dataframe.

From the analysis performed on the LDA dataframe, we start to see a pattern. The Classification algorithms that don’t revolve around the interconnected nature of the data see a reduction in overall accuracy with the advent of feature reduction. This can be seen with the decision tree algorithm, which works in a straightforward manner that revolves around the notions of choosing one particular class based on the variations in the features. When the number of features is reduced from 75 features to 2 features, the accuracy of the model drops. The feature reduction, however, helps the accuracy of models that are based entirely on probabilities or unsupervised models such as the Gaussian Naive Bayes model and the K-Nearest Neighbors algorithms. Let us see if this trend continues into the last feature reduction method.

Photo by Marek Piwnicki on Unsplash

Feature reduction using SVD

Singular Value Decomposition is yet another dimension reduction algorithm. Here is a deep dive into the SVD algorithm: SVD Article

The SVD algorithm works by factorizing the matrix of features into three matrices that come together using this formula:

A = U𝚺V^T

where A is the matrix of features with dimensions (m x n), U is the orthogonal matrix with dimensions (m x m), V is the orthogonal matrix with dimensions (n x n), and 𝚺 is the nonnegative rectangular diagonal matrix of size (m x n)

The TruncatedSVD function in the sklearn package performs this transformation very well

from sklearn.decomposition import TruncatedSVD svd = TruncatedSVD(n_components=2, algorithm=’randomized’, random_state=0) X_svd = svd.fit_transform(transportation_df.drop([‘location_name’, ‘Class’], axis=1)) svd_df = pd.DataFrame(data = X_svd, columns = [‘PC 1’, ‘PC 2’]) svd_df.head(3)

Adding the class column to the SVD dataframe

svd_df = pd.concat([svd_df, transportation_df[‘Class’]], axis = 1) svd_df.head(3)

Graphing the various classes as a function of PC1 and PC2. The graph below seems similar to the graph generated through the PCA analysis. This shows a similarity between the outputs of the two algorithms. Let’s see if the classification algorithms show similarities in accuracy to the PCA analysis

PC_0 = svd_df.where(svd_df[‘Class’] == 0).dropna()[[‘PC 1’,’PC 2']] PC_1 = svd_df.where(svd_df[‘Class’] == 1).dropna()[[‘PC 1’,’PC 2']] PC_2 = svd_df.where(svd_df[‘Class’] == 2).dropna()[[‘PC 1’,’PC 2']] plt.scatter(PC_0[‘PC 1’],PC_0[‘PC 2’]) plt.scatter(PC_1[‘PC 1’],PC_1[‘PC 2’]) plt.scatter(PC_2[‘PC 1’],PC_2[‘PC 2’])

Classification Algorithms Using SVD

The below snippets will show how to use the Gaussian Naive Bayes, Decision Tree, and the K-Nearest Neighbors Classifiers with the reduced features.

splitting data into training and testing data

x_cols = [] for item in list(svd_df.columns): if(item != ‘Class’): x_cols.append(item) X = svd_df[x_cols] y = svd_df[‘Class’] X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)

Gaussian Naive Bayes Classifier:

gnb = GaussianNB().fit(X_train, y_train) gnb_predictions = gnb.predict(X_test)

The heatmap for the Gaussian Naive Bayes shows that the overall accuracy of the SVD dataframe is slightly better than the overall accuracy of the original dataframe (.265 vs. .319). The model overall does not perform very well in predicting the actual airport records and bus stop records with either dataframe (.140 vs. .059) and (0 vs. .009). In both cases, most of the records are classified as Train stations and in both cases, the models predict the actual train station records very well (.968 vs. .968). As with the case with all of the uses of the Gaussian Naive Bayes model, regardless of the use of feature reduction, the model tends to mostly predict most records as Train stations. This is again because of the nature of the model and how it considers each of the features as individual features rather than features that are correlated with each other.

Decision Tree Classifier:

dtree_model = DecisionTreeClassifier(max_depth = 3).fit(X_train, y_train) dtree_predictions = dtree_model.predict(X_test)

The decision tree model using the SVD dataframe is slightly less accurate than the model using the original dataframe (67.8 vs. 75). the model predicts the real airport records at a rate that is slightly worse than the model when using the original data (.91 vs. .9). The model doesn't change in its lack of prediction of true bus station records but does much worse in predicting the true train records than the original datasets (.533 vs. .292). The model shows a tendency to classify most records as Airports because of the imbalance in the data and the majority of the records having an initial ground truth of being an airport

K-Nearest Neighbors Classifier

knn = KNeighborsClassifier(n_neighbors = 22).fit(X_train, y_train)

The K-nearest Neighbors model using the SVD dataframe performs slightly worse than when the model uses the original data (.67 vs. .679). Like the original model, the SVD dataframe model does a good job at predicting the actual airport records (.927 vs. .93). The accuracy of the prediction of actual bus and train station records is also similar to that of the original data (0.027 vs. .027)and (.180 vs. .173). Like previous models, the imbalance in the dataset and the larger number of airport records leads to this model predicting most records as an Airport.

The above analysis corroborates our theory from before. The algorithms that are based on probabilistic decisions and unsupervised models like the Gaussian Naive Bayes and the K-Nearest Neighbors models perform better with the feature-reduced data than the models such as the decision tree model that require a larger number of features for better performance.

Conclusion:

From the analysis that we performed with the three different forms of feature reduction and the application of these feature-reduced dataframes on 3 different multiclass classifiers, we see one particular form of feature reduction that works much better than the other two and that is the LDA transformation. The LDA algorithm helps to improve the accuracy of most of the classifiers that were used and in fact, makes the Gaussian Naive Bayes model almost 30% more accurate. Thus this model is most effective in reducing SafeGraph data’s features and providing the best results.

This notebook gave us an introduction to model tuning using feature reduction and classification of POI categories based on SafeGraph Patterns visit data. Our algorithms saw some success in classifying airports vs train stations vs bus stops, all based on things like dwell time, visitor distance from home, popularity by hour of the day, and popularity by day of the week.

The next notebook in the series (link coming soon!) will take a deep dive into the use of deep learning classifiers to perform the same analysis and see if neural networks provide a boost in prediction accuracy

Questions?

I invite you to ask them in the #safegraphdata channel of the SafeGraph Community, a free Slack community for data enthusiasts. Receive support, share your work, or connect with others in the GIS community. Through the SafeGraph Community, academics have free access to data on over 7 million businesses in the USA, UK, and Canada.