Automate Data Preparation using Google Colab: Read and Process Citi Bike Data in Zip File | by Huajing Shi | Towards Data Science

From time to time I get requests from colleagues to process some large data files and report some statistics from the data. Since they rely on Excel as their main data processing/analysis tool and don’t use Python, R or SQL, reading and processing data files with more than 1,048,576 rows (the row capacity of Excel) become a problem.
To process data in a zip file, normally I would download the zip file, unzip to get CSV file, and then import the CSV file into either SQL, R or Python to process. Once I got the results, I need to output the results to CSV files and sent the files to the colleagues who requested the work. This whole procedure could get cumbersome when I have to repeat the process every month when new data comes in.
Google Colab (i.e., Google Colaboratory) provides a platform to share Python code and run it in web browser. It is especially helpful to share your Python solution with non Python users.
Here is an example of Python program using Google Colab to automate the data preparation of monthly Citi Bike data. The program can read the zip file directly from the Citi Bike online portal, and process the data to report the total number of Citi Bike trips for each calendar day. The users only need to provide the URL of the data file that they want to process, and download/save the results to their local computers at the end.
Citi Bike website publishes user usage data every month, and maintained historical data back to June 2013. This is a good data source for analyzing micro mobility trend in New York metropolitan region.
The number of records in the Citi Bike data file could be very large. For instance, the data for the month of August 2020 contains more than two million records which exceeded the row capacity of Excel.
First go to Citi Bike Trip data portal and copy the link address of the file you would like to process:
https://s3.amazonaws.com/tripdata/index.html
To run the code, you need to sign in to your Google account. Once signed in, you can run the Python code by clicking on the arrow at the top left corner of the code window.
Paste the data file link address in the input box “Data url: ”, and hit “enter”. The code will download the zip file from the file address, and read in the data line by line, and save the data into a DataFrame.
data_address = input('Data url: ')url = urllib.request.urlopen(data_address)data = []df = pd.DataFrame()with ZipFile(BytesIO(url.read())) as my_zip_file:    for contained_file in my_zip_file.namelist():        for line in my_zip_file.open(contained_file).readlines():            s=str(line,'utf-8')            s = re.sub(r"\n", "", s)            s = re.sub(r"\"", "", s)            line_s = s.split(",")            data.append(line_s)                       df = pd.DataFrame(data)
Things get easier and fun once we have the data stored into a DataFrame. The first row in the original data contains variable names (or column names). Use the data in the first row as column names for the DataFrame and then remove the first row from the DataFrame. Then aggregate the DataFrame by the “starttime” of each trip to count the number of trips for each calendar day, and print the result to the output console.
col_name = df.iloc[0].astype(str)df.columns = col_namedf = df.drop([0])df['startdate']=df['starttime'].astype('datetime64[ns]').dt.datedate_count = df.groupby('startdate').count()[['bikeid']]date_count.columns = ['count']print(date_count)
Once the result is output to the console, download the result to your local computer by specifying the output file name in the input box “Output File Name:” at the bottom of the console.
output_file_name = input('Output File Name: ')date_count.to_csv(output_file_name+'.csv') files.download(output_file_name+'.csv')
The complete code is as follows:
from io import BytesIOfrom zipfile import ZipFileimport pandas as pdimport urllib.requestimport refrom google.colab import filesdata_address = input('Data url: ')url = urllib.request.urlopen(data_address)data = []df = pd.DataFrame()with ZipFile(BytesIO(url.read())) as my_zip_file:    for contained_file in my_zip_file.namelist():        for line in my_zip_file.open(contained_file).readlines():            s=str(line,'utf-8')            s = re.sub(r"\n", "", s)            s = re.sub(r"\"", "", s)            line_s = s.split(",")            data.append(line_s)                       df = pd.DataFrame(data)col_name = df.iloc[0].astype(str)df.columns = col_namedf = df.drop([0])df['startdate']=df['starttime'].astype('datetime64[ns]').dt.datedate_count = df.groupby('startdate').count()[['bikeid']]date_count.columns = ['count']print(date_count)output_file_name = input('Output File Name: ')date_count.to_csv(output_file_name+'.csv') files.download(output_file_name+'.csv')
Link to Google Colab Notebook:
https://colab.research.google.com/drive/1SpTToe6sL9OnB5-LhAPpOwGCIPe1miHw#scrollTo=kRMFFxaW30Rc
Link to GitHub: