Neighbourhood Segmentation and Clustering using Foursquare API | by Mathangpeddi | Towards Data Science

Whoever said money can’t buy happiness didn’t know where to go shopping.
For many shoppers, visiting shopping malls is a great way to relax and enjoy themselves during weekends and holidays. Property developers are also taking advantage of this trend to build more shopping malls to cater to the demand. As a result, there are many shopping malls in the city of Hyderabad and many more are being built. Opening shopping malls allow property developers to earn consistent rental income. As with any business decision, opening a new shopping mall requires serious consideration and is a lot more complicated than it seems. Particularly, the location of the shopping mall is one of the most important decisions that will determine whether the mall will be a success or a failure.
The objective of this project is to analyze and select the best locations in the city of Hyderabad, India, to open a new shopping mall. This project is mainly focused on geospatial analysis of the Hyderabad City to understand which would be the best place to open a new mall. Using data science methodology and machine learning techniques like clustering, this project aims to provide solutions to answer the business question: In the city of Hyderabad, if a property developer is looking to open a new shopping mall, where would you recommend that they open it?
To solve the problem, we will need the following data: • List of neighbourhoods in Hyderabad. This defines the scope of this project which is confined to the city of Hyderabad, the capital city of Telangana, which is in South India• Latitude and longitude coordinates of those neighbourhoods. This is required in order to plot the map and also to get the venue data • Venue data, particularly data related to shopping malls. We will use this data to perform clustering on the neighbourhoods.
This Wikipedia page is a list of neighbourhoods in Hyderabad, with 200 neighbourhoods. I have used web scraping techniques to extract the data from the Wikipedia page, with the help of Python requests and beautifulsoup packages. Then we can get the latitude and longitude coordinates of the neighbourhoods using Python Geocoder package. After that, I have used the Foursquare API to get the venue data for those neighbourhoods.
Foursquare API will provide many categories of the venue data, and we are particularly interested in the Shopping Mall category in order to help us solve the business problem. This is a project that will make use of many data science skills, from web scraping (Wikipedia), working with API (Foursquare), data cleaning, data wrangling, to machine learning (K-means clustering) and map visualization (Folium).
The Foursquare API allows application developers to interact with the Foursquare platform. The API itself is a RESTful set of addresses to which you can send requests, so there’s really nothing to download onto your server.
Search for Cafes in Hyderabad
Cafes returned by Foursquare
On the left, you see all the cafes and their name, category, address of every venue in Hyderabad. On the right, you see the map of the venues on the left.
Exploring the cafes
If you click on the first one which is The Coffee Cup, then you are redirected to this page where you see all the information in the Foursquare dataset about The Coffee Cup. This includes its name, full address, working hours, tips and images that users have posted about the cafe. So similarly you can explore the shopping malls in the city of Hyderabad.
To explore the Foursquare click here
Web Scraping
Perform scraping using Python requests and beautifulsoup packages to extract the list of neighbourhoods data.
# Send the GET requestdata = requests.get("https://en.wikipedia.org/wiki/Category:Neighbourhoods_in_Hyderabad,_India").text# Parse data from the html into a beautifulsoup objectsoup = BeautifulSoup(data, 'html.parser')# Create a list to store neighbourhood dataneighborhoodList = []# Append the data into the listfor row in soup.find_all("div", class_="mw-category")[0].findAll("li"):  neighborhoodList.append(row.text)# Create a new DataFrame from the listkl_df = pd.DataFrame({"Neighborhood": neighborhoodList})kl_df.head()
This is the data frame created after scraping the data. We need to get the geographical coordinates in the form of latitude and longitude in order to be able to use Foursquare API. To do so, we will use the Geocoder package that will allow us to convert the address into geographical coordinates in the form of latitude and longitude.
# Defining a function to get coordinatesdef get_latlng(neighborhood):    # initialize your variable to None    lat_lng_coords = None    # loop until you get the coordinates    while(lat_lng_coords is None):        g = geocoder.arcgis('{}, Hyderabad, India'.format(neighborhood))        lat_lng_coords = g.latlng    return lat_lng_coords# Call the function to get the coordinates, store in a new list using list comprehensioncoords = [ get_latlng(neighborhood) for neighborhood in kl_df["Neighborhood"].tolist()]
We have obtained the latitude and longitude coordinates for all the places, so we need to merge the coordinates into the original data frame.
# Create temporary dataframe to populate the coordinates into Latitude and Longitudedf_coords = pd.DataFrame(coords, columns=['Latitude', 'Longitude'])# Merge the coordinates into the original dataframekl_df['Latitude'] = df_coords['Latitude']kl_df['Longitude'] = df_coords['Longitude']print(kl_df.shape)kl_df(200,3)
This is the combined data frame which contains all the neighbourhoods along with the geographical coordinates.
# Getting the coordinates of Hyderabadaddress = 'Hyderabad, India'geolocator = Nominatim(user_agent="my-application")location = geolocator.geocode(address)latitude = location.latitudelongitude = location.longitudeprint('The geograpical coordinate of Hyderabad, India {}, {}.'.format(latitude, longitude))
After gathering the data, we have to populate the data into a pandas DataFrame and then visualize the neighbourhoods in a map using Folium package.
map_kl = folium.Map(location=[latitude, longitude], zoom_start=11)# Adding markers to mapfor lat, lng, neighborhood in zip(kl_df['Latitude'],  kl_df['Longitude'], kl_df['Neighborhood']): label = '{}'.format(neighborhood) label = folium.Popup(label, parse_html=True) folium.CircleMarker([lat, lng],radius=5,popup=label,color='blue',fill=True,fill_color='#3186cc',fill_opacity=0.7).add_to(map_kl)map_kl
Use the foursquare API to explore the neighbourhoods
CLIENT_ID = '' # your Foursquare IDCLIENT_SECRET = '' # your Foursquare SecretVERSION = '20180604'radius = 2000LIMIT = 100venues = []for lat, long, neighborhood in zip(kl_df['Latitude'], kl_df['Longitude'], kl_df['Neighborhood']):# Create the API request URLurl = "https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}".format(CLIENT_ID,CLIENT_SECRET,VERSION,lat,long,radius,LIMIT)# Make the GET requestresults = requests.get(url).json()["response"]['groups'][0]['items']# Return only relevant information for each nearby venuefor venue in results:    venues.append((neighborhood,lat,long,venue['venue']['name'],    venue['venue']['location']['lat'],venue['venue']['location']    ['lng'],venue['venue']['categories'][0]['name']))
After extracting all the venues, we have to convert the venues list into a new DataFrame.
To know more about Foursquare documentation click here
venues_df = pd.DataFrame(venues)# Defining the column namesvenues_df.columns = ['Neighborhood', 'Latitude', 'Longitude', 'VenueName', 'VenueLatitude', 'VenueLongitude', 'VenueCategory']print(venues_df.shape)venues_df.head()
# Lets check how many venues were returned for each neighbourhoodvenues_df.groupby(["Neighborhood"]).count()# Lets check out how many unique categories can be curated from all the returned valuesprint('There are {} unique categories.'.format(len(venues_df['VenueCategory'].unique())))There are 174 unique categories# Displaying the first 50 Venue Category namesvenues_df['VenueCategory'].unique()[:50]
Analyzing each neighbourhood
Here we apply one hot encoding to all the venues. So now the number of columns becomes 175
# One hot encodingkl_onehot = pd.get_dummies(venues_df[['VenueCategory']], prefix="", prefix_sep="")# Adding neighborhood column back to dataframekl_onehot['Neighborhoods'] = venues_df['Neighborhood']# Moving neighbourhood column to the first columnfixed_columns = [kl_onehot.columns[-1]] + list(kl_onehot.columns[:-1])kl_onehot = kl_onehot[fixed_columns]print(kl_onehot.shape)(6684, 175)
Next, let’s group rows of neighbourhood by taking the sum of the frequency of occurrence of each category.
kl_grouped=kl_onehot.groupby(["Neighborhoods"]).sum().reset_index()print(kl_grouped.shape)kl_grouped(198, 175)
len((kl_grouped[kl_grouped["Shopping Mall"] > 0]))
There are 66 shopping malls in Hyderabad which is very high. So now we have to select a suitable location where the number of shopping malls is less so that our chances of setting up a shopping mall at that location should be good.
# Creating a dataframe for Shopping Mall data onlykl_mall = kl_grouped[["Neighborhoods","Shopping Mall"]]
Clustering the neighbourhoods
Now we need to cluster all the neighbourhoods into different clusters. The results will allow us to identify which neighbourhoods have a higher concentration of shopping malls while which neighbourhoods have a fewer number of shopping malls. Based on the occurrence of shopping malls in different neighbourhoods, it will help us answer the question as to which neighbourhoods are most suitable to open new shopping malls.
# Setting the number of clusterskclusters = 3kl_clustering = kl_mall.drop(["Neighborhoods"], 1)# Run k-means clustering algorithmkmeans = KMeans(n_clusters=kclusters,random_state=0).fit(kl_clustering)# Checking cluster labels generated for each row in the dataframekmeans.labels_[0:10]array([0, 0, 0, 1, 0, 1, 1, 0, 0, 0], dtype=int32)
We set the number of clusters to 3 and run the algorithm. After applying the K-Means clustering algorithm, all the neighbourhoods get segregated and form different clusters.
# Creating a new dataframe that includes the cluster as well as the top 10 venues for each neighborhood.kl_merged = kl_mall.copy()# Add the clustering labelskl_merged["Cluster Labels"] = kmeans.labels_kl_merged.rename(columns={"Neighborhoods": "Neighborhood"}, inplace=True)kl_merged.head(10)
Here the Shopping Mall column represents the number of shopping malls in that particular area and Cluster Labels represents the cluster number (either 0 or 1 or 2)
 # Adding latitude and longitude values to the existing dataframekl_merged['Latitude'] = kl_df['Latitude']kl_merged['Longitude'] = kl_df['Longitude']# Sorting the results by Cluster Labelskl_merged.sort_values(["Cluster Labels"], inplace=True)kl_merged
Now here we can clearly see all the places which belong to the 1st cluster(cluster number 0). Similarly, we see all the cluster numbers in the sorted order that is 0,1,2.
Visualizing the resulting clusters
# Creating the mapmap_clusters = folium.Map(location=[latitude, longitude], zoom_start=11)# Setting color scheme for the clustersx = np.arange(kclusters)ys = [i+x+(i*x)**2 for i in range(kclusters)]colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))rainbow = [colors.rgb2hex(i) for i in colors_array]# Add markers to the mapmarkers_colors = []for lat, lon, poi, cluster in zip(kl_merged['Latitude'], kl_merged['Longitude'], kl_merged['Neighborhood'], kl_merged['Cluster Labels']):label = folium.Popup(str(poi) + ' - Cluster ' + str(cluster), parse_html=True)  folium.CircleMarker([lat,lon],radius=5,popup=label,color=rainbow[cluster-1],fill=True,fill_color=rainbow[cluster-1],fill_opacity=0.7).add_to(map_clusters)map_clusters
Examining the clusters
len(kl_merged.loc[kl_merged['Cluster Labels'] == 0])len(kl_merged.loc[kl_merged['Cluster Labels'] == 1])len(kl_merged.loc[kl_merged['Cluster Labels'] == 2])
There are 132 places in cluster 0 which is the highest among the 3 clusters, and cluster 0 contains all the places which do not have a shopping mall. Cluster 1 contains 51 places and all of them contain exactly 1 shopping mall, while cluster 2 contains 15 places where all the places contain 2 or more shopping malls.
The results from the K-means clustering show that we can categorize the neighbourhoods into 3 clusters based on the frequency of occurrence for “Shopping Mall”:• Cluster 0: Neighbourhoods with very less number of shopping malls • Cluster 1: Neighbourhoods with a moderate concentration of shopping malls • Cluster 2: Neighbourhoods with a high concentration of shopping malls We visualize the results of the clustering in the map with cluster 0 in red colour, cluster 1 in purple colour, and cluster 2 in mint green colour.
A good number of shopping malls are concentrated in the central area of Hyderabad city. Cluster 0 has a very low number of malls. This represents a great opportunity and high potential areas to open new shopping malls, as there is very little to no competition from existing malls. Meanwhile, shopping malls in cluster 2 are likely suffering from intense competition because of oversupply and a high concentration of shopping malls. Therefore, this project recommends property developers to capitalize on these findings to open new shopping malls in neighbourhoods in cluster 0 with little to no competition. Property developers with unique selling propositions to stand out from the competition can also open new shopping malls in neighbourhoods in cluster 1 with moderate competition. Lastly, property developers are advised to avoid neighbourhoods in cluster 2 which already have a high concentration of shopping malls and suffering from intense competition.
So we can apply the same approach for large datasets and can easily distinguish the venues based on the category. Suppose if there are 400 restaurants in a city then we can easily segregate them into different clusters. We can apply this method not only for shopping malls but restaurants, coffee shops, and much more. In this project, we only consider one factor i.e. frequency of occurrence of shopping malls, there are other factors such as population and income of residents that could influence the location decision of a new shopping mall.
But for setting up a shopping mall we need to consider other factors such as the cost of rent, the surroundings around the shopping mall, the kind of people in the locality-if it's a luxurious area many people prefer going out, their lifestyle will be different from others and therefore spend a lot. If we decide a place where the competition is less, then we need to consider the people living in that locality as well. If the people in that area spend a lot and love going out then it’ll be a success. If the people staying near the mall don't prefer going out, then it's better to consider some other place with less competition and a good crowd.
You can always refer to my GitHub Repository for the entire project.
“We used to build civilizations. Now we build shopping malls.” -Billy Bryson
Connect with me on LinkedIn
I hope you found the article insightful. I would love to hear feedback to improvise it and come back with better content.
Thank you so much for reading !