Data Modeling for a Music Streaming App | by Joshua Yeung | Towards Data Science

I have joined Udacity Data Engineer Nanodegree. The program consists of many real-world projects. In the project, I have the role of a Data Engineer at a fabricated data streaming company called “Sparkify”. It is a startup and wants to analyze the data they’ve been collecting on songs and user activity on their new music streaming app. The analytics team is particularly interested in understanding what songs users are listening to. Currently, they don’t have an easy way to query their data, which resides in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in their app.
They’d like a data engineer to create a Postgres database with tables designed to optimize queries on song play analysis and bring me on the project. My role is to create a database schema and ETL pipeline for this analysis. I’ve tested my database and ETL pipeline by running queries given to me by the analytics team from Sparkify and compare my results with their expected results.
In this project, I’ve applied what I’ve learned on data modeling with Postgres and build an ETL pipeline using Python. I’ve defined fact and dimension tables for a star schema for a particular analytic focus and written an ETL pipeline that transfers data from files in two local directories into these tables in Postgres using Python and SQL.
The first dataset is a subset of real data from the Million Song Dataset. Each file is in JSON format and contains metadata about a song and the artist of that song. The files are partitioned by the first three letters of each song’s track ID. For example, here are file paths to two files in this dataset.
song_data/A/B/C/TRABCEI128F424C983.jsonsong_data/A/A/B/TRAABJL12903CDCF1A.json
And below is an example of what a single song file, TRAABJL12903CDCF1A.json, looks like.
{"num_songs": 1, "artist_id": "ARJIE2Y1187B994AB7", "artist_latitude": null, "artist_longitude": null, "artist_location": "", "artist_name": "Line Renaud", "song_id": "SOUPIRU12A6D4FA1E1", "title": "Der Kleine Dompfaff", "duration": 152.92036, "year": 0}
The second dataset consists of log files in JSON format generated by this event simulator based on the songs in the dataset above. These simulate activity logs from a music streaming app based on specified configurations.
The log files in the dataset you’ll be working with are partitioned by year and month. For example, here are file paths to two files in this dataset.
log_data/2018/11/2018-11-12-events.jsonlog_data/2018/11/2018-11-13-events.json
And below is an example of what the data in a log file, 2018–11–12-events.json, looks like.
If you would like to look at the JSON data within log_data files, you will need to create a pandas DataFrame to read the data. Remember to first import JSON and pandas libraries.
df = pd.read_json(filepath, lines=True)
For example, df = pd.read_json('data/log_data/2018/11/2018-11-01-events.json', lines=True) would read the data file 2018-11-01-events.json.
In case you need a refresher on JSON file formats, here is a helpful video.
Using the song and log datasets, I’ve created a star schema optimized for queries on song play analysis. This includes the following tables.
songplays — records in log data associated with song plays i.e. records with page NextSongsongplay_id, start_time, user_id, level, song_id, artist_id, session_id, location, user_agent
users — users in the appuser_id, first_name, last_name, gender, level
songs — songs in the music databasesong_id, title, artist_id, year, duration
artists — artists in the music databaseartist_id, name, location, latitude, longitude
time — timestamps of records in songplays broken down into the specific unitstart_time, hour, day, week, month, year, weekday
The conditions like Primary key and NOT NULL in columns are important while dealing with relational database systems. Again, as a data engineer, it's a must to specify primary key columns and not null depending on the business requirements. For instance, user_id and start_time should be considered as not null.
In addition to the data files, the project workspace includes six files:
test.ipynb displays the first few rows of each table to check the database.create_tables.py drops and creates the tables. I run this file to reset my tables before each time I run my ETL scripts.etl.ipynb reads and processes a single file from song_data and log_data and loads the data into the tables. This notebook contains a detailed explanation of the ETL process for each of the tables.etl.py reads and processes files from song_data and log_data and loads them into the tables.sql_queries.py contains all the SQL queries, and is imported into the last three files above.
test.ipynb displays the first few rows of each table to check the database.
create_tables.py drops and creates the tables. I run this file to reset my tables before each time I run my ETL scripts.
etl.ipynb reads and processes a single file from song_data and log_data and loads the data into the tables. This notebook contains a detailed explanation of the ETL process for each of the tables.
etl.py reads and processes files from song_data and log_data and loads them into the tables.
sql_queries.py contains all the SQL queries, and is imported into the last three files above.
To create all the tables I needed, I have done the following four steps:
Write CREATE statements in sql_queries.py to create each table according to the schema defined above.Write DROP statements in sql_queries.py to drop each table if it exists.Run create_tables.py to create the database and tables.Run test.ipynb to confirm the creation of the tables with the correct columns. Make sure to click "Restart kernel" to close the connection to the database after running this notebook.
Write CREATE statements in sql_queries.py to create each table according to the schema defined above.
Write DROP statements in sql_queries.py to drop each table if it exists.
Run create_tables.py to create the database and tables.
Run test.ipynb to confirm the creation of the tables with the correct columns. Make sure to click "Restart kernel" to close the connection to the database after running this notebook.
Inside create_tables.py, there is a function create_tables to execute all the queries listed in create_table_queries list.
You can see the tables have been created with the columns defined from test.ipynb.
It’s time to develop ETL processes for each table. In etl.py, there is a function called process_data. You can pass the file path and the process function you need to process your data. It gets all files matching extension (.json) from the directory. And then it iterates over files and processes using your function passed into.
First, let us handle song_data and create the songs and artists dimensional tables. I’ve builtprocess_song_file function to perform the following tasks:
First, get a list of all song JSON files in data/song_data
Read the song file
Extract Data for Songs Table and Insert Data into Songs Table
Extract Data for Artists Table and Insert Data into Artists Table
To extract data for Songs table, the following tasks are done:
Select columns for song ID, title, artist ID, year, and duration
Use df.values to select just the values from the DataFrame
Index to select the first (only) record in the DataFrame
Convert the array to a list and set it to song_data
Write the song_table_insert query in sql_queries.py and insert a record for this song into the songs table.
Data Extraction for Artists table is similar to that for Songs table, but this time I selected columns artist ID, name, location, latitude, and longitude.
Second, I’ve performed ETL on the second dataset, log_data, to create the time and users dimensional tables, as well as the songplays fact table.
I’ve filtered records by NextSong action
df = df[df['page']=='NextSong']
and converted the ts timestamp column to DateTime.
t = pd.to_datetime(df['ts'], unit='ms')
Note: the current timestamp is in milliseconds
I used pandas’ dt attribute to access easily datetime-like properties such as hour, day, week of year, month, year, and weekday from the ts column and set time_data to a list containing these values in order. I’ve also specified labels for these columns and set to column_labels. I created a DataFrame, time_df, containing the time data for this file by combining column_labels and time_data into a dictionary and converting this into a DataFrame. Finally, I inserted records for the timestamps in the log files into the time table.
Data extraction for users table is easy, just select columns for user ID, first name, last name, gender, and level and set to user_df and insert records for the users in this log file into the users table.
INSERT statements needs to be modified for each table to handle existing records where appropriate. As a data engineer, I need to give careful thought to how the tables will handle existing records when new data are being inserted into the tables.
For instance, use on conflict update for the user table since the level can change from free to paid.
This one is a little more complicated since the information from the songs table, artists table and original log file are all needed for the songplays table. Since the log file does not specify an ID for either the song or the artist, I needed to get the song ID and artist ID by querying the songs and artists tables to find matches based on the song title, artist name, and the song’s duration time.
Since this is a subset of the much larger dataset, the solution dataset will only have 1 row with values containing ID for both songid and artistid in the fact table. Those are the only 2 values that the query in the sql_queries.py will return that is not-NONE. The rest of the rows will have NONE values for those two variables.
That’s it. I’ve created a Postgres database with tables designed to optimize queries on song play analysis. The analytics team has now an easy to query their data and analyze understanding what songs users are listening to.
For more details, you can visit my GitHub repository.