Clustering of Hong Kong Districts’ by restaurants mix
By: Rex Lui
Tags: Geographical Plotting, Data Science, Hong Kong, Clustering, Python

GitHub repository: link | Jupyter Notebook: link

Clustering of Hong Kong Districts’ by restaurants mix

This project is my final assignment of my IBM professional certificate. I reviewed the script, added more details in the notebook and then published to medium.

Business problems

Hong Kong is well known as “food paradise” and visitors can enjoy both Eastern and Western food in such small city.

My curiosity is to analyze the mix of restaurant type of each district in Hong Kong to see if some districts do have similar mix of restaurants.

In this project, KMeans Clustering will be used for clustering.

Data collection and manipulation

First of all, the Hong Kong districts metadata are gathered from wikipedia by pandas.

For coordinates of each Hong Kong districts, it has already been collected previous by Google Map API. In this project, I will import the csv instead and merge it with wikipedia data together.

# Read table from Wiki

HK_district_wiki = pd.read_html('https://en.wikipedia.org/wiki/Districts_of_Hong_Kong')[5]



# Exclude the data which is irrelevant which are 'subtotal' and Marine

HK_district = HK_district_wiki[~HK_district_wiki['District'].str.contains('subtotal')&~HK_district_wiki['District'].str.contains('Marine')]['District'].to_frame()



# Read area table from Wiki

HK_district_wiki_area = pd.read_html('https://en.wikipedia.org/wiki/Districts_of_Hong_Kong')[6][['District','Area(km2)']]



# Merge the Area data

HK_district = HK_district.merge(HK_district_wiki_area)



# Reset index

HK_district.reset_index(inplace = True, drop = True)



# Load the HK District Geodata which I collected previously

hk_district_geo = pd.read_csv('HKDistrictGeo.csv')



# Merge to the District DataFrame

HK_district = HK_district.merge(hk_district_geo)

print(HK_district)

Now we have a table of Hong Kong districts with area and the coordinates

To examine whether the coordinates are correct, plot a map of districts using Folium.

# create map

# set location that make Hong Kong at the centre of the web page

# set zoom_start = 12 for better visualization

map = folium.Map(tiles='cartodbpositron', location=[22.3529808, 114.107615], zoom_start=12)



# add markers to the map to show the districts

markers_colors = []

for lat, lon, poi in zip(HK_district['latitude'], HK_district['longitude'], HK_district['District']):

label = folium.Popup(str(poi))

folium.CircleMarker(

[lat, lon],

radius=10,

popup=label,

fill=True,

fill_color='#FFFFFF',

fill_opacity=0.7).add_to(map)



map.save('hk_district.html')

display(IFrame('hk_district.html', width = 1500, height = 1200))

As a Hong Konger, this map completely match my understanding!

Above map correctly locate the coordinate of Hong Kong districts.

After that, Foursquare will be used to explore the restaurants data in the district.

By using the API, the venue location and the type of restaurant can be extracted.

# Define FourSquare Credential

CLIENT_ID = 'XXX' # your Foursquare ID

CLIENT_SECRET = 'XXX' # your Foursquare Secret

VERSION = '20180605' # Foursquare API version

LIMIT = 1000 # A default Foursquare API limit value # Create a function to explore the Food venure from a given coordinates

import requests

def getNearbyVenues(names, latitudes, longitudes, radius):



venues_list=[]

for name, lat, lng, rad in zip(names, latitudes, longitudes, radius):

print(name)



# create the API request URL

url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}&categoryId=4d4b7105d754a06374d81259'.format(

CLIENT_ID,

CLIENT_SECRET,

VERSION,

lat,

lng,

rad,

LIMIT)



# make the GET request

results = requests.get(url).json()["response"]['groups'][0]['items']



# return only relevant information for each nearby venue

venues_list.append([(

name,

lat,

lng,

v['venue']['name'],

v['venue']['location']['lat'],

v['venue']['location']['lng'],

v['venue']['categories'][0]['name']) for v in results])



nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])

nearby_venues.columns = ['District',

'District Latitude',

'District Longitude',

'Venue',

'Venue Latitude',

'Venue Longitude',

'Venue Category']



return(nearby_venues) # Run the function for each HK districts

# Since some districts are much larger than another districts, I assume each district to be a circle and use the district area to determine the radius using

HK_district_venues = getNearbyVenues(names = HK_district['District'],

latitudes = HK_district['latitude'],

longitudes = HK_district['longitude'],



#assume district is a circle, determine radius by the equation of circle area: Area = pi * r^2

radius = np.sqrt(HK_district['Area(km2)']/np.pi)*1000

)

By using the API, the restaurant name and the restaurant category can be extracted in the district

Analyze the restaurants in each district

Convert categorial attributes of the restaurant type to numerical by using one hot encoding. After that, compute the proportion of restaurant mix in the districts. This will be used as the attributes for clustering in the later stage.

# one hot encoding

HK_district_venues_onehot = pd.get_dummies(HK_district_venues[['Venue Category']], prefix="", prefix_sep="")



# add District column back to dataframe

HK_district_venues_onehot['District'] = HK_district_venues['District']



# move District column to the first column

fixed_columns = [HK_district_venues_onehot.columns[-1]] + list(HK_district_venues_onehot.columns[:-1])

HK_district_venues_onehot = HK_district_venues_onehot[fixed_columns] # Group the district by summing the count of restaurant type

# Calculate the proportion of the restaurant type in the districts

HK_district_grouped = HK_district_venues_onehot.groupby('District').mean().reset_index()

HK_district_grouped

Restaurant mix of each districts

Since it’s rather hard to observe which type of restaurant ranked the top in each district, another dataframe will be created for displaying the top 10th restaurant type in the districts.

# Find out the top 10 restaurant type in the district



def return_most_common_venues(row, num_top_venues):

row_categories = row.iloc[1:]

row_categories_sorted = row_categories.sort_values(ascending=False)



return row_categories_sorted.index.values[0:num_top_venues]



num_top_venues = 10



indicators = ['st', 'nd', 'rd']



# create columns according to number of top venues

columns = ['District']

for ind in np.arange(num_top_venues):

try:

columns.append('{}{} Most Restaurant Type'.format(ind+1, indicators[ind]))

except:

columns.append('{}th Most Restaurant Type'.format(ind+1))



# create a new dataframe

HK_district_dish_sorted = pd.DataFrame(columns=columns)

HK_district_dish_sorted['District'] = HK_district_grouped['District']



for ind in np.arange(HK_district_grouped.shape[0]):

HK_district_dish_sorted.iloc[ind, 1:] = return_most_common_venues(HK_district_grouped.iloc[ind, :], num_top_venues)



HK_district_dish_sorted

Now we can already observe which districts are similar in term of restaurant mix

Cluster the districts by the restaurant type

KMeans clustering will be used as the methodology of clustering.

As a simple trial, 4 clusters will be set in the classifier. After running the clustering, it will be merged back to the district dataframe for better display.

# set number of clusters

kclusters = 4 # set the attributes used for clustering

HK_district_grouped_clustering = HK_district_grouped.drop('District', 1) # run k-means clustering

kmeans = KMeans(n_clusters = kclusters, random_state = 2)

kmeans.fit(HK_district_grouped_clustering) # check cluster labels generated for each row in the dataframe

kmeans.labels_[0:10] # Merge the result into the district dataframe

# add clustering labels

HK_district_dish_sorted['Cluster Labels'] = kmeans.labels_

#HK_district_dish_sorted.insert(0, 'Cluster Labels', kmeans.labels_) HK_district_merged = HK_district # merge df_grouped with df_data to add latitude/longitude for each neighbourhood

HK_district_merged = HK_district_merged.join(HK_district_dish_sorted.set_index('District'), on='District') HK_district_merged

For better visualization, the clusters will be display in the map.

# create map

map_cluster = folium.Map(tiles='cartodbpositron', location=[22.3529808, 114.107615], zoom_start=12)



# set color scheme for the clusters

x = np.arange(kclusters)

ys = [i + x + (i*x)**2 for i in range(kclusters)]

colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))

rainbow = [colors.rgb2hex(i) for i in colors_array]





# add markers to the map

markers_colors = []

for lat, lon, poi, cluster in zip(HK_district_merged['latitude'], HK_district_merged['longitude'], HK_district_merged['District'], HK_district_merged['Cluster Labels']):

label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)

folium.CircleMarker(

[lat, lon],

radius=10,

popup=label,

color=rainbow[cluster-1],

fill=True,

fill_color=rainbow[cluster-1],

fill_opacity=0.7).add_to(map_cluster)



map_cluster.save('hk_district_cluster.html')

display(IFrame('hk_district_cluster.html', width = 1500, height = 1200))

Map of district with clusters

Examine Clusters

Now, it’s time to check whether the result make sense and really meaningful to users.

Cluster 1

Kolwoon City, Wong Tai Sin and Tsuen Wan are special areas in Hong Kong which clustered in Cluster 1

Kolwoon City is well known to have the best and most number of Thai restaurants in Hong Kong.

It’s reasonable why these three districts are clustered into the same clusters.

Cluster 2

Southern, Islands and Sai Kung are the remote areas in Hong Kong and many foreigners loves to live there.

As a result, to fit the taste of their need, there are lots of Cafe and western dishes restaurants in the areas.

Cluster 3

Cluster 3 are mostly business and urban area in Hong Kong which consists of different types of restaurants.

The variety of restaurant type in these districts is high that well demonstrated why Hong Kong is also known as “Food Paradise”.

Cluster 4

Most of Cluster 4 districts are living areas in Hong Kong.

The restaurants mainly serves locals and provide cheap and convenient options of meals.

As shown in the below tables, Chinese Restaurant and Fast Food Restaurant exist a lot in these districts.

Conclusion

By using clustering above, the districts can be briefly grouped into 4 segments by the mix of restaurants there.

Although it maybe not surprising to any Hong Kong people, this technic can still be replicated to other city even though we have no knowledge about the city.

Next step

Since free vision of FourSquare was used in the project, the number of venue can be extracted is limited and more number of restaurants can be grabbed for paid version. On the other hand, FourSquare is a US based company and may not have full detail of restaurant data in Hong Kong. If the restaurant data from Hong Kong popular dining rating website “Openrice” can be obtained instead, more accurate clustering can be obtained with more restaurant data.

Thanks for reading!