Interactive Time Series with Python and API: Generating Heatmap of Taxis in Singapore | by Shawn Tham | Towards Data Science

Land Transport Authority (LTA) of Singapore has an API that shows live coordinates of all taxis. It is quite a cool API that shows real-time taxi movement that is refreshed every 30 seconds. However, there is little work done with this API after searching up for more information on it. Hence, in this article, I will walk through the steps on how to collect data from this API and present it in a time series heatmap below using Folium!
Step 1: Importing the relevant packages
import foliumimport folium.plugins as pluginsimport pandas as pdimport jsonimport requestsimport timefrom folium.plugins import HeatMapWithTimeimport datetime
Step 2: Pulling Data from API and storing it as a dataset
I start by creating an empty data frame so that I can append the live data into it.
cumulative = pd.DataFrame()
Next, using a simple input function, I am able to key in the duration of taxi movement. This will determine the amount of times the loop will run.
mins = input('How many mins do you want to see? ')number_of_times = (int(mins)*60)/30
Next, write a for loop to continuously pull data from the API. As the API refreshes every 30 seconds, I use Python time Module to rerun the script every 30 seconds. Meanwhile, I only extract the coordinates and the timestamp from the Geojson object and append it into my pandas data frame. This will allow me to compile past data into a data frame for easy manipulation later on.
startTime = time.time()for i in range(int(number_of_times)):    url = ("https://api.data.gov.sg/v1/transport/taxi-availability")    response = requests.get(url)    data = response.json()    df = pd.io.json.json_normalize(data['features'])    coordinateslist = df['geometry.coordinates'].tolist()    df1 = pd.DataFrame(coordinateslist)    result = df1.transpose()    result.columns = ['coordinates']    result['Timestamp'] = (df['properties.timestamp'][0])    cumulative = cumulative.append(result)     time.sleep(30)endTime = time.time()elapsedTime = endTime - startTimeprint("Elapsed Time = %s" % elapsedTime,'seconds')
Step 3: Data Wrangling with for loops
Now it is time to manipulate the data so that it is easy for us to utilize it. I start by splitting the coordinates into latitude and longitude columns.
cumulative['coordinates'] = cumulative['coordinates'].astype(str)Latitude = []Longitude = []coordinates = []for i in cumulative['coordinates']:    i = i.split(", ")    lat = i[1][:-1]    long = i[0][1:]    Latitude.append(lat)    Longitude.append(long)cumulative['Latitude'] = Latitudecumulative['Longitude'] = Longitude
Your data frame should resemble the one below before we process it into the format we want in order to plot it into Folium.
Next, we need to put the latitude and longitude into a nested list in order for us to use folium.plugins.HeatMapWithTime to generate the time series heatmap.
lat_long_list = []for i in cumulative['Timestamp'].unique():    temp=[]    for index, instance in cumulative[cumulative['Timestamp'] == i].iterrows():        temp.append([instance['Latitude'],instance['Longitude']])    lat_long_list.append(temp)
Step 4: Creating a time index and formatted it
Next, we have to convert the Timestamp into a datetime format so that we can convert it into a neater and more readable layout. This will also be the index that is used in the play button to show the time and date.
#converting it to datetime formatcumulative['Timestamp']= pd.to_datetime(cumulative['Timestamp'])#creating a time indextime_index = []for i in cumulative['Timestamp'].unique():    time_index.append(i)#formatting the indexdate_strings = [d.strftime('%d/%m/%Y, %H:%M:%S') for d in time_index]
Step 5: Plotting the time series heatmap
Finally, we can generate the map! I choose a dark themed map of Singapore so that the heatmap will look more vibrant.
#Choosing the map type m = folium.Map(location=[1.352083,103.819839],zoom_start = 11, tiles="https://tiles.stadiamaps.com/tiles/alidade_smooth_dark/{z}/{x}/{y}{r}.png",attr="Stadia.AlidadeSmoothDark")#Plot it on the mapHeatMapWithTime(lat_long_list,radius=5,auto_play=True,position='bottomright',name="cluster",index=date_strings,max_opacity=0.7).add_to(m)# Display the mapm
Summary
Step 1: Importing the relevant packages
Step 2: Pulling Data from API and storing it as a dataset
Step 3: Data Wrangling with for loops
Step 4: Creating a time index and formatted it
Step 5: Plotting the time series heatmap
Feel free to save it as a html file so that you can send to your friends and colleagues. You can check out the full code on my Github here.