A Gentle Introduction to IoT/GPS Trajectory Clustering and Geospatial Clustering | by William Yee | Towards Data Science

Internet of things, connected devices, and geospatial data are all hot topics when it comes to developing value-added propositions. In this article, I will provide a gentle introduction to unsupervised learning techniques applied to GPS Trajectories.
The data comes from Microsoft Research Asia, containing data from 182 users and 17,621 trips between April 2007 and August 2012. The data is mostly densely populated with a longitude and latitude recording every 1 to 5 seconds.
Now let’s take a look at the data files themselves. Hmm ‘.plt’ format, a format I have never seen before, but let’s open it in text edit.
Okay great, looks like pandas read_csv function will work well with the .plt file format, and the accompanying documentation about the data structure looks accurate.
import pandas as pdcolnames = ['lat','long', 'null', 'alt' ,'DateTime','Date','Time']df = pd.read_csv('20090724005608.plt', skiprows=6, names = colnames)
Having the time series in a data frame is great, but we need to convert each data frame into a single observation with multiple features. Doing so will format the data for loading into clustering algorithm. Let’s aggregate the 10th 25th 50th 75th 90th and average values for speed, acceleration, and jerk . To do so, we are going to write a few python function to apply to each file.
def add_feat(df_input):    """    This function takes in raw lat long time series from the microsoft geolife data.Preprocessing notes: skip the first six lines before doing pandas read csv , expecting columns in ['lat','long', 'null', 'alt' ,'DateTime','Date','Time']Requres: pandas imported as pd             from vincenty import vincenty             Adds:        speed        acceleration        jerk        bearing rate        distance travelled    """    df = df_input    # add some initial shifts    df['lat_shift'] = df.lat.shift(-1)    df['long_shift'] = df.long.shift(-1)    df['time_shift'] = df.DateTime.shift(-1)    # add speed    def speed(x):        try:            s = vincenty((x[-3],x[-2]),(x[0],x[1]), miles = True) / ((x[-1]-x[4]) * 24)        except:            s= np.nan        return s    df['speed_mph'] = df.apply(speed,axis =1)    df['speed_shift'] = df.speed_mph.shift(-1)    # add acceleration    def accel(x):        try:            a = (x[-1] - x[-2]) / ((x[9] - x[4]) *24*60*60)        except:            a = np.nan        return a    df['acceleration'] = df.apply(accel, axis =1)    df['acceleration_shift'] = df.acceleration.shift(-1)    # add jerk    def jerk(x):        try:            j = (x[-1] - x[-2]) / ((x[9] - x[4]) *24*60*60)        except:            j = np.nan        return a    df['jerk'] = df.apply(accel, axis =1)    df['jerk_shift'] = df.jerk.shift(-1)    # add y for bearing calculator    def y(x):        try:            yy = np.sin((x[8] - x[0]) * np.pi/180) *np.cos( x[7]* np.pi/180)        except:            yy= np.nan        return yy    df['y'] = df.apply(y, axis =1)    # add x for bearing calculator     def x(x):        try:            xx = np.cos(x[0] * np.pi/180) *np.sin(x[7]* np.pi/180) - np.sin(x[0]* np.pi/180) * np.cos(x[7]* np.pi/180)*np.cos((x[8]-x[1])* np.pi/180)        except:            xx = np.nan        return xx    df['x'] = df.apply(x,axis =1)    # calculate bearing    def bearing(x):        try:            b = np.arctan2(x[-2],x[-1])*180/np.pi        except:            b = np.nan        return b    df['bearing'] = df.apply(bearing,axis=1)df['brearing_shift'] = df.bearing.shift(-1)    # calculate bearing rate (rate of change of direction)    def bearing_rate(x):        try:            br = abs(x[-1]-x[-2])        except:            br = np.nan        return br    df['bearing_rate'] = df.apply(bearing_rate,axis=1)    # calculate distance travelled    def distance(x):        try:            dist = vincenty((x[7],x[8]),(x[0],x[1]), miles = True)        except:            dist= np.nan        return dist    df['distance'] = df.apply(distance,axis = 1)        df.drop(df.tail(4).index,inplace=True)    return dfdef list_df_summary(input_df):    '''    Converts output from add_feat function into an observation for machine learning    '''    names = ['speed_10','speed_25','speed_50','speed_75','speed_90','speed_ave',            'accel_10','accel_25','accel_50','accel_75','accel_90','accel_ave',            'jerk_10','jerk_25','jerk_50','jerk_75','jerk_90','jerk_ave',            'bearingRate_10','bearingRate_25','bearingRate_50','bearingRate_75','bearingRate_90','bearingRate_ave',            'distance_traveled','time_traveled' ,'last_lat','last_long','start_lat','start_long'                        ]    values = list()    #speed    values.append(input_df.quantile(.10)[8])    values.append(input_df.quantile(.25)[8])    values.append(input_df.quantile(.50)[8])    values.append(input_df.quantile(.75)[8])    values.append(input_df.quantile(.90)[8])    values.append(input_df.mean()[8])    #accel    values.append(input_df.quantile(.10)[10])    values.append(input_df.quantile(.25)[10])    values.append(input_df.quantile(.50)[10])    values.append(input_df.quantile(.75)[10])    values.append(input_df.quantile(.90)[10])    values.append(input_df.mean()[10])    #jerk    values.append(input_df.quantile(.10)[12])    values.append(input_df.quantile(.25)[12])    values.append(input_df.quantile(.50)[12])    values.append(input_df.quantile(.75)[12])    values.append(input_df.quantile(.90)[12])    values.append(input_df.mean()[12])    #bearing    values.append(input_df.quantile(.10)[18])    values.append(input_df.quantile(.25)[18])    values.append(input_df.quantile(.50)[18])    values.append(input_df.quantile(.75)[18])    values.append(input_df.quantile(.90)[18])    values.append(input_df.mean()[18])    #distance travelled    values.append(input_df.distance.sum())    #time travelled    values.append((input_df.iloc[-1,4] - input_df.iloc[0,4])*24*60)    #lat long    values.append(input_df.iloc[-1,0])    values.append(input_df.iloc[-1,1])    values.append(input_df.iloc[0,0])    values.append(input_df.iloc[0,1])            return pd.DataFrame([values],columns=names)
To loop through the file directory, I used a package called glob. Below is the code that accomplishes creating a list of pathnames for every .plt file in the GeoLife directory. Then runs a for loop through the .plt list and applies the previous function to get us ready for machine learning!
plts = []for folder in glob('Geolife Trajectories 1.3/Data/*/Trajectory'):    for file in glob(folder +'/*.plt'):        plts.append(file)large_df = []for i, file in enumerate(plts):    print( int(i*100 / len(plts)))    try:        large_df.append(list_df_summary(add_feat(pd.read_csv(file, skiprows=6,names=colnames))))    except:        print('error at: ' + file)df = pd.concat(large_df)
For my example, I am going to take a subset of the modeling_df features to create clusters. Not every feature has useful information(read more on the curse of dimensionality). It’s a bit of an iterative process, you can try different subsets to develop different clusters. Initially, I looked at the distributions of each feature to guess if it would be valuable inclusion to the unsupervised learning algorithm.
from sklearn.cluster import KMeansfrom sklearn.preprocessing import StandardScalerimport seaborn as snsfrom collections import Counter## Remove longer tripsdf_local_all_feat=df[(df.time_traveled < 300) & (df.distance_traveled < 100)& (df.speed_90<75)]## Select Features with unique distributionsdf_local = df_local_all_feat[['speed_90','speed_ave','accel_75','speed_50','accel_ave','distance_traveled','time_traveled']]## Scale Data for KMeansX = StandardScaler()X = X.fit_transform(df_local)#################################### PERFORM KMeans CLUSTERING ####################################Sum_of_squared_distances = []scores = []cluster_count = []K= range(2,15,1)for i in K:    km = KMeans(n_clusters=i)    km.fit(X)    pred =km.predict(X)    Sum_of_squared_distances.append(km.inertia_)    scores.append(silhouette_score(X, pred))    cluster_count.append(i)    print(Counter(pred))    print(str(silhouette_score(X, pred)) + ' clusters:' +str(i))    print('------')########################################### Elbow plot with silhouette score ###########################################sns.set_context('talk')sns.lineplot(cluster_count,scores)plt.xlabel('Cluster Count')plt.ylabel('Silhouette Score')plt.title('Elbow Method For Optimal Cluster Count')plt.show()
df_local_all_feat['cluster'] = KMeans(n_clusters=7).fit_predict(X)
Clusters 4,5 contain very few number of observations therefore, are not included in this analysis.
Let’s take a look at the final destination and the Trajectory clusters.
Looking at Tsinghua University we can see a significant amount of clusters 0 and 1.
Looking at Tecent offices we can start to see significantly more subway rides, clusters 5 and 6.
Looking at a local shopping district a less subway/train trips, but train trips typically travel longer distances (size is total distance travelled).
The larger “zoom-out” map of the above section, can be found below. We have identified 3 general points of interest, it’s time to cluster them.
Looks like we need algorithm that can combine distance with proximity, while giving the capability to exclude noise. It sounds like DB Scan would be algorithm for this application.
from sklearn.cluster import DBSCAN# pull out the only the location metricsdf_cluster = df_local_all_feat[['Last Lat', 'Last Long']]X = StandardScaler()X = X.fit_transform(df_cluster)scores = []EP =[]number =[]eps= [.08,.10,.15,.20,.25,.30,.40,.50,.60]size= [200,250,300]for ep in eps:    for siz in size:        db = DBSCAN(eps=ep, min_samples=siz)           pred = db.fit_predict(X)                print('ep:'+ str(ep) + ' size:'+ str(siz))        print(Counter(pred))        try:            scores.append(silhouette_score(X, pred))        except:            scores.append(np.nan)        EP.append(ep)        number.append(siz)
The variable eps and size will allow you to grid search the appropriate hyperparameters for your application. In the end I, landed on the following values.
db = DBSCAN(eps= 0.37, min_samples = 250)
By combing both clustering algorithms, we can filter specific trips based on point of interest to determine where people are coming from and the method of transportation used to arrive at the said point of interest.
For Tsinghua University, we can see people are generally taking shorts trip from nearby.
For the shopping district, we can see people are traveling a bit further and are mostly using transportation mode 0.
For the Tencent office, people are traveling from the furtherest away. However, people commuting from nearby are typically taking transportation mode 0, while people commuting from further away are taking transportation mode 6.
Using various unsupervised learning algorithms, we are able to determine how people travel to specific locations, how to bound points of interest, and where people originated.
With adjustments, in the data collection architecture, it would be interesting to determine how people dwell stay at a location, based on the factors of distance travelled and method of transportation.