Clustering City Nightlife using Machine Learning
By: Ajit Rajput
Tags: K Means, Party, Nightlife, Clustering, Machine Learning

KMeans Clustering using Foursquare API

Introduction

Everyone knows how Covid-19 pandemic devastated the nightlife industry with social distancing, lockdowns, mask-wearing and early curfews. These nightlife spaces were shuttered because they had been deemed non-essential services and places of easy transmission for the coronavirus. Now that central and state governments in India have eased the restrictions people can finally enjoy a breather, commemorating a special occasion or just spending time with friends over food and drinks.

In a city like Pune, which boasts a happening nightlife scene, there’s always a party happening somewhere or the other. Widely known as the “IT hub of India”, “Automobile and Manufacturing hub of India” and “Oxford of the East”, Pune is known for its lifestyle, pleasant weather and just… everything good. With escalating number of pubs, bars, lounges and similar hangout joints, night life in Pune has become ever more exciting. There are many bars, clubs, pubs in Pune for couples, groups of friends, as well as night owls.

For any new visitor in Pune, it can be difficult to figure out where they should hang out for party and drinks. With this article we will aim to cluster the city into areas or suburbs having high concentration of bars and pubs

Neighborhood Exploration

For clustering we will require list of neighborhoods of Pune and their latitude and longitude.

Get the Neighbors

Wikipedia is great source of information. The list of neighborhoods of Pune was scraped using Python’s BeautifulSoup library from the given Wikipedia page.

data = requests.get(" https://en.wikipedia.org/wiki/Category:Neighbourhoods_in_Pune ").text soup = BeautifulSoup(data, 'html.parser') neighbors = []

for row in soup.find_all("div", {"class": "mw-category"})[0].findAll("li"):

neighbors.append(row.text)



#Create a new DataFrame from the list

df_neighbors = pd.DataFrame({"neighbor": neighbors}) df_neighbors = df_neighbors.drop(labels=[26,32,39, 40], axis=0)

df_neighbors.reset_index(drop=True)

Fetch the Neighbor Co-ordinates

Next, using geocoder python library, we will fetch the latitude and longitude of each neighborhood.

## Function to fetch the geo coordinates for the locations

def getNeighborCoords(neighbor):

lat_lng_coords = None



while(lat_lng_coords is None):

geo = geocoder.arcgis('{}, Pune, India'.format(neighbor))

lat_lng_coords = geo.latlng



return lat_lng_coords

Using the above function we will fetch the co-ordinates for each of the neighborhood areas. The resultant output from the function is mapped as new columns i.e. ‘latitude’ and ‘longitude

coords = [ getNeighborCoords(neighbor) for neighbor in df_neighbors["neighbor"].tolist()] ## Map the latitude and longitude retuned from the function

df_neighbors['latitude'], df_neighbors['longitude'] = zip(*coords)

Our final dataframe now has the neighbor and its corresponding latitude and longitude co-ordinates.

Let’s see how our neighbors look on a map. I have used the geopy and folium library to create a map of Pune with our neighbor coordinates superimposed on top.

## center of pune

pune_center = [18.5204303, 73.8567437] ## plot the neighbors on the map using Folium library map_pune = folium.Map(location=pune_center, zoom_start=11) for neighbor, lat, lon in zip(df_neighbors['neighbor'],df_neighbors['latitude'],df_neighbors['longitude']):

label = '{}'.format(neighbor)

label = folium.Popup(label, parse_html=True)

folium.CircleMarker([lat, lon], radius=5, color='black', fill_color='blue', fill_opacity=0.25, popup=label, fill=True).add_to(map_pune)



map_pune

Venues Exploration

Now that we have the neighborhoods of Pune, with its respective latitude and longitude information, let us fetch the surrounding venues within specified radius (km)

Fetch the Venues

Foursquare API, a social location service allows users to discover about businesses and attractions. As we are trying to find bars venues, I added the category ID ‘4bf58dd8d48988d116941735’ as part of the API request URL. I have defined the limit of venues returned to be 100 within radius of 1 km.

With the Foursquare API call, we fetched around 168 venues

Data Engineering

With our extracted information of nearby bar venues, we can merge this into a new data frame containing our neighbor information. Also, from this new dataframe, we will drop neighbors that do not have any venues returned i.e. drop NA records.

df_Venues = getNearbyVenues(df_neighbors['neighbor'],df_neighbors['latitude'],df_neighbors['longitude']) df_Venues.dropna(how='any', inplace=True)

df_Venues.reset_index(drop=True, inplace=True)

df_Venues.head()

Next, within the defined radius of 1km, there are high chances of Foursquare API returning same venues for neighbors which are in close proximity. Let’s find and drop the duplicate venues. In this process we will keep the “first” occurrence of the duplicate records and delete the rest.

df_Venues.drop_duplicates(subset=['name','categories','lat','lng'], keep='first', inplace=True)

This resulted into deleting duplicate venues and we are left with 113 venues now. So far, great ! Now lets explore the unique venues returned by the Foursquare API

df_Venues['categories'].unique()

Unique list of venues

You see that the unique list includes all sorts of venues including Juice Bars, Falafel Restaurants, Jazz Club, Cafe etc. Though not sure if all of these venues serve alcohol, but for sake of simplicity and better clustering let’s just focus on few known venue categories.

## filter out only Bars from the fetched venues df_hotels = df_Venues[df_Venues['categories'].isin(['Bar', 'Hotel Bar','Pub', 'Cocktail Bar', Lounge','Sports Bar', 'Gastropub','Brewery','Distillery']) ]

df_hotels.reset_index(drop=True, inplace=True) ## rearrange and rename the columns

df_hotels.columns=['neighbor','latitude','longitude','venue','category','venue_lat','venue_lon']

df_hotels.head()

Clustering Neighbors and Venues

With the neighbors and venues data in place with all data engineering tasks completed, lets now get onto clustering.

We will first apply one-hot encoding technique on the ‘category’ column; take mean frequency for each venue category and finally perform the K-Means clustering, an unsupervised learning algorithm used to create K clusters of data points based on feature similarity.

## one-hot encode the category

df_hotels_OHE = pd.get_dummies(df_hotels['category'], prefix='', prefix_sep='')

df_hotels_OHE['neighbor'] = df_hotels['neighbor'] ## rearrange the columns with 'neighbor' as first column. fixed_columns = [df_hotels_OHE.columns[-1]] + list(df_hotels_OHE.columns[:-1])

df_hotels_OHE = df_hotels_OHE[fixed_columns] df_hotels_grouped = df_hotels_OHE.groupby('neighbor').mean().reset_index() df_hotels_grouped.head()



Using this grouped data frame, we will drop the neighbor column and apply K-means clustering technique to arrive at optimal ‘k’ value. Looking at the elbow method and silhouette score, it appears that the optimal number of clusters to use is K = 8.

Now, lets perform the actual K-Means Clustering using k=8 on the dataset and map cluster labels to the each neighbor.

kclusters = 8 x = df_hotels_grouped.drop(['neighbor'], 1)​ # run k-means clustering kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(x)​ df_hotels_merged["cluster"] = kmeans.labels_ df_clustered = pd.merge(df_hotels, df_hotels_merged[['neighbor','cluster']] ,on=["neighbor"])

To visualize how the clusters are formed, lets again plot map of neighbors and their cluster-labels with unique color-coding.

occurences = folium.map.FeatureGroup()

map_clusters = folium.Map(location=pune_center, zoom_start=11) # Setting color scheme for the clusters

x = np.arange(kclusters)

ys = [i+x+(i*x)**2 for i in range(kclusters)] colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))

rainbow = [colors.rgb2hex(i) for i in colors_array] for lat, lon, poi, cluster in zip(df_clustered['latitude'], df_clustered['longitude'], df_clustered['neighbor'], df_clustered['cluster']):



label = folium.Popup(str(poi) + ' - Cluster ' + str(cluster), parse_html=True)



folium.CircleMarker([lat,lon],

radius=5,

popup=label,

color=rainbow[cluster-1],

fill=True,

fill_color=rainbow[cluster-1],

fill_opacity=1,

icon = folium.Icon(icon='beer', prefix='fa')).add_to(map_clusters) map_clusters

Below is the map displaying the clusters for different neighborhoods. The data points in different colors are the clustered areas.

To better understand how the neighborhood has been clustered, lets visualize it in a bar chart plot. As you see, there are 8 clusters and the plot shows clustered neighbors according to their count of venues.

To further explore and an additional insight into clustering, here is the heat-map of the clustered neighbors. If you see the area — Vimannagar is showing strong density whereas central Pune areas with medium density.

df_neighbor_HM = df_clustered.groupby(['neighbor','latitude','longitude'])['category'].count().reset_index() pune_latlons = df_neighbor_HM[['latitude','longitude','category']] pune_map = folium.Map(location=pune_center, zoom_start=10)

folium.TileLayer('cartodbpositron').add_to(pune_map) #cartodbpositron cartodbdark_matter

HeatMap(pune_latlons, blur=10).add_to(pune_map) folium.Marker(pune_center).add_to(pune_map) pune_map

Lets also plot heatmap for the different venues as well to see how these are placed on the geographical map.

pune_latlons = df_clustered[['venue_lat','venue_lon']] pune_map = folium.Map(location=pune_center, zoom_start=13)

folium.TileLayer('cartodbpositron').add_to(pune_map) #cartodbpositron cartodbdark_matter

HeatMap(pune_latlons, blur=20).add_to(pune_map) folium.Circle(pune_center, radius=2000, fill=False, color='white').add_to(pune_map)

folium.Circle(pune_center, radius=4000, fill=False, color='white').add_to(pune_map) pune_map

Conclusion

To conclude, we used K-means clustering which created 8 different clusters for Pune neighbors which are populated with nightlife venues around them. A good number of night-life venues are densely populated (within 1km radius) around Vimannagar, KalyaniNagar. These neighborhoods are in close proximity to IT hubs and hence these venues are good revenue earning sources.

Additionally, if you see the central Pune, Camp areas also have good number of nightlife party venues. Surprisingly, the Pimpri-Chinchwad, Baner, Wakad areas which are close to Hinjewadi Park (major IT Hub) appear not to have many venues. This could be probably because of the neighborhood data/locations OR not many places/venues registered on Foursquare.

So, if you are visiting Pune for first time OR planning a party, you can use this map to get a good idea of where to binge.

Happy Partying ! Drink Responsibly 🍻 🍸 🍷

I hope you found the article insightful. I would love to hear feedback to improvise it and come back with better content.