Learning PySpark by Exploring and Visualizing Google App Data
By: Bee Guan Teo
Tags: Python, Data Science, Data Visualization, Pyspark

Exploration of Google App Data

In this section, we are going to start writing Python script in the Databricks Notebooks to perform exploratory data analysis using PySpark. This section will be broken down into seven parts and some common PySpark methods will be introduced along the way.

An online version of the Notebook with completed code can be accessed at this link or you may also download an offline Notebook from my Github.

Part 1: Import libraries

The first step started with importing prerequisite libraries/modules.

Python script to import libraries

Line 1–5:

Import all the necessary PySpark modules required for data exploratory tasks presented in this article .

Line 7:

PySpark requires a SQLContext to initiate the functionalities of Spark SQL. Spark SQL is one of the Spark modules which is used to work with structured data. We are going to use it to perform data query from our dataset in a later stage.

The keyword “sc” denotes SparkContext. SparkContext represents a connection to a Spark Cluster and is considered as a main entry point for Spark functionalities.

Part 2: Read file and count total rows of records

Next, we get the data from an external source (a CSV file in this case).

Python script to read file and count total row

Line 1:

In PySpark, we use read.csv method from SQLContext module to read csv file from the data uploaded to Databricks in an earlier step. Please note the csv file is stored at the directory “FileStore/tables” in Databricks by default.

method from SQLContext module to read csv file from the data uploaded to Databricks in an earlier step. Please note the csv file is stored at the directory “FileStore/tables” in Databricks by default. The read.csv method returns a Spark Dataframe and assign it to variable, df_spark. The Spark Dataframe is similar to Pandas Dataframe.

Line 2:

PySpark use count method to get number of rows of the records read from the csv file.

Total row of records

Part 3: Inspect data

Once we read the raw data from the CSV file, we might be interested to know some basic details of the dataset by having a quick overview of some records.

3.1 Show column details

Python script to print column details

printSchema method displays data type of each column and also to show if a column is nullable. At this point, we can observe that all the columns are string and are nullable.

Column details

3.2 Display first several rows of records

Python script to show first five rows of records

The show method is to display the first several rows of record. We can flexibly pass a different value to the show method to adjust the number of rows that we intend to display.

First five rows of records

3.3 Display specific columns

Python script to display specific columns

We can also use “select” method chained with the “show” method to only display some specified columns instead of show all the columns.

Top five rows of selected columns

Part 4: Data Wrangling

At the first glance of the raw data read from the CSV file, we might have noticed several issues:

There might be null or missing values in some columns since all the columns are nullable. The null or missing values can result in analytical errors.

All the columns are strings. This will pose a problem when we wish to perform statistical analysis or plotting graph using the data.

Besides, some columns are not presented in a format which permit numerical analysis. For example, the values in “Size” column are appended with a letter “M” right after the digit. And the value of “Installs” column are mixed with comma and plus characters. The “Price” value is also preceded with “$” character.

Some examples of bad data format

Data cleaning and transformation are needed here to permit easy data access and analysis.

4.1 Remove null values

Prior to removing the null values, we need to identify the columns where null values can be found.

Python script to show null values in each column

The code above check for the existence of null value for every columns and count its frequency and then display it in a tabulated format as below.

Number of null values exist in each column

The output shows that there is one null value in “Content Rating”, “Current Ver” and “Android Ver” columns. Next we are going to use dropna method to remove the null value from the columns.

Python script to remove null values

4.2 Remove unwanted characters

In PySpark, we can transform our data in a specified column into a format that is useful to us. To do so, we can use withColumn and translate methods.

Python script to remove unwanted characters

Line 1–3:

withColumn method is to specify the column with which we intent to transform (e.g. “Size”, “Installs” and “Price”)

method is to specify the column with which we intent to transform (e.g. “Size”, “Installs” and “Price”) translate method is to replace the character patterns with a target character. For example, translate(‘Size’, ‘Mk’, ‘’) will replace all the character ‘M’ or ‘k’ characters in the ‘Size’ column with an empty string, ‘’. This means all the character ‘M’ or ‘k’ that appended with the size value will be stripped off. The similar method is adopted to remove “+” or “,” from “Installs” column and “$” from “Price” column.

Line 5:

Show again the first five records after data transformation.

Sample of transformed data

4.3 Filter unwanted value

At this point we have managed to remove unwanted characters, “M” or “k”, from the “Size” column. However, there is still one more issue remained. There is existence of value “Varies with device” in that column. This is because the size of some apps can vary with device. Such string value is inconsistent with the rest of the values (numerical)in the column and therefore we have to remove them.

Python script to remove unwanted values

To filter out a particular value from a column, we can use where method to set a filter condition where only the value of a specified column (e.g. “Size”) that meet a specific condition (e.g. != “Varies with device”) will be selected and assign it back to df_spark.

method to set a filter condition where only the value of a specified column (e.g. “Size”) that meet a specific condition (e.g. != “Varies with device”) will be selected and assign it back to df_spark. This approach will screen out all the records with size value equal to “Varies with device”.

4.4 Change data type of columns

Remember that all the columns are still in string format even though we have gone through the data cleaning and transformation steps above. We have to convert some columns from string to numerical values.

Python script to change column’s data type

Line 1–4:

Again we use withColumn to set a specific column which we are going to work with. Next, we use cast method to set a specific data type (IntegerType or FloatType) to which we intend to convert the string.

Line 5:

Use printSchema again to show the updated data types in each column.

Updated data types

4.5 Rename columns

Now, we can proceed to rename the columns which we have just transformed their values to reflect the changes.

Python script to rename columns

Line 1–3:

To rename a column, we can use withColumnRenamed method. This method takes two arguments, 1) original column name and 2) new column name.

Line 4:

Show first five rows of the records to view the changes on the column names.

Renamed columns

4.6 Drop unwanted columns

Not all the columns are relevant in the study here and we can remove those irrelevant columns.

Python script to drop columns

Removal of columns is straightforward. Just use drop method and pass the target column names (e.g. Reviews, Genres & Current Ver)as arguments to the method.

Dataframe after dropping selected columns

Part 5: Query data

At last, we manage to obtain a clean data in a usable format and we are now ready to delve deeper to explore our data. In this part, we will use filter method to perform data query based on different type of conditions.

5.1 Query by single condition

In general, we can set a condition within the filter method and this will return all the records that match the condition. Let’s look at the examples below:

Python script to search the app dedicated to teen

This query is done to search for the app which are dedicated to teen.

Filtered records

Python script to search the app with rating above 4.2

The query above is done to search for the record with “Rating” greater than 4.2.

Filtered records

5.2 Query by value range

PySpark offers a method, between, to enable us to search for records between a lower limit and upper limit.

Python script to search by a range of values

The code above returns the records with the “Price” between $1 to $3.

Filtered records

5.3 Query by keyword

This is also possible to search for record based on the existence of some specific keywords that exist in a particular column. We can do so by one of the three methods: startswith, endswith and contains. Let’s look at several examples below:

5.3.1 startswith

Python script to use startswith to query data

The startswith method above return the records with “Android Ver” started with “4”. This means only the records of the app running on Android 4 will be returned.

Filtered records

5.3.2 endswith

Python script to use endswith to query data

The endswith method above return the records with “Last Updated” ended with “2018”. This means only the records of the app which are last updated in 2018 will be returned.

Filtered records

5.3.3 contains

Python script to use contains to query data

The contains method above return the records with “App” contains a keyword “photo”. This means only the records of the app that includes “photo” keyword as part of its name will be returned.

Filtered records

5.4 Query by multiple conditions

We can use logical operator such as &, |, and ~ to join multiple search conditions in our data query.

Python script to query by using logical operators to join multiple search conditions

The code above uses & operator to join three search conditions so that only the records of the app belonging to GAME category and the last updated year is 2018 and running on Android 5 above will be returned.

Filtered records

Part 6: Descriptive statistics

We can easily run a quick descriptive statistical study on our dataset using PySpark describe method.

Python script to calculate descriptive statistics