Apache Spark Is Fun
By: Nick Rafferty
Tags: Programming, Apache Spark, Big Data, Software Development, Learning To Code

Try Your First DataFrame!

Now for the fun part! Open up a new tab and navigate to this GitHub repository: https://github.com/nickrafferty78/Apache-Spark-Is-Fun.

By now I’m assuming you’ve created a GitHub account yourself. If you haven’t, you’ll need to do that first before moving on to the next step.

First, fork the repository.

Next, navigate to your own GitHub and open up the repository that you just forked. (If these concepts are unfamiliar to you I’d encourage you to read more here: https://guides.github.com/introduction/git-handbook/. Click the ‘Code’ button and copy the URL as shown below. Next, type this command into your terminal (make sure to replace the ‘your-github-username’ with your own username):

Now let’s get IntelliJ started. Open up IntelliJ and open the project that you just cloned to your computer. Navigate to the Tutorial class under src/main/scala/Tutorial.scala.

Creating the Spark Session

There is a little starter code to help this app get set up. It should look like this:

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.functions._

import org.apache.log4j.{Level, Logger}

import org.apache.spark.sql.types.LongType

import Utilities._



object Tutorial extends App {



Utilities.setupLogging()





}

Underneath Utilities.setupLogging() , copy and paste this code:

val spark = SparkSession

.builder()

.appName("InstrumentReviews")

.master("local[*]")

.getOrCreate()

Congrats, you’ve written your first Spark Session! Let’s break that down.

As we mentioned above, the Spark Session is your entry point to begin using DataFrames. The main things you need to know are:

The .appName method names your Spark Session. Since this section will be using the InstrumentReviews.json dataset, I am going to name it ‘InstrumentReviews’.

method names your Spark Session. Since this section will be using the InstrumentReviews.json dataset, I am going to name it ‘InstrumentReviews’. The .master method tells spark which URL to connect to. Since we are running this on your own personal computer, ‘local’ tells spark to use your personal computer instead of using an external machine. The ‘[*]’ after local is an interesting piece you might not know. This is one of the core tenets of spark, to be able to distribute data in an extremely efficient way. Your computer likely has more than one core. Each core on your computer can process jobs independently of each other. My own computer has 2 cores, so by telling spark [*], you are telling it to run on every single core in your computer.

method tells spark which URL to connect to. Since we are running this on your own personal computer, ‘local’ tells spark to use your personal computer instead of using an external machine. The ‘[*]’ after local is an interesting piece you might not know. This is one of the core tenets of spark, to be able to distribute data in an extremely efficient way. Your computer likely has more than one core. Each core on your computer can process jobs independently of each other. My own computer has 2 cores, so by telling spark [*], you are telling it to run on every single core in your computer. Finally, .getOrCreate() creates your session.

Reading Data

We’re going to be working with musical instrument review data in this example. Before reading the data, it’s always good to get a lay of the land with what data you’ll be working with. Navigate to /data/InstrumentReviews.json and you should see an extremely large file filled with instrument review data. Here is an example entry:

{

"reviewerId":"A2IBPI20UZIR0U",

"reviewerName":"cassandra tu \"Yeah, well, that's just like, u...",

"reviewText":"Not much to write about here, but it does exactly what it's supposed to. filters out the pop sounds. now my recordings are much more crisp. it is one of the lowest prices pop filters on amazon so might as well buy it, they honestly work the same despite their pricing,",

"overall":5.0,

"summary":"good"

}

Now back in Tutorial.scala, underneath the Spark Session you created, copy, and paste this code:

val firstDataFrame = spark

.read

.format("json")

.option("inferSchema", "true")

.load("data/InstrumentReviews.json")

You’ve already created the variable spark above, so in this code block, you are utilizing that spark session to read the incoming data.

The .format method specifies what type of structure the data is coming in. Our musical instrument review data is in JSON.

method specifies what type of structure the data is coming in. Our musical instrument review data is in JSON. .option with “inferSchema”, “true” is telling spark to infer the schema of the musical instrument review data without you explicitly defining it.

with “inferSchema”, “true” is telling spark to infer the schema of the musical instrument review data without you explicitly defining it. .load gives spark the path to the data file.

Let’s test if that will work!

Underneath your code for the firstDataFrame, write these two lines:

firstDataFrame.show()

firstDataFrame.printSchema()

If all went well you should see something like this:

root

|-- overall: double (nullable = true)

|-- reviewText: string (nullable = true)

|-- reviewerId: string (nullable = true)

|-- reviewerName: string (nullable = true)

|-- summary: string (nullable = true)

If you see that, congrats on writing your first DataFrame!