Data Science and Machine Learning with Scala and Spark (Episode 02/03) | by MA Raza, Ph.D. | Towards Data Science

Spark’s inventors chose Scala to write the low-level modules. In Data Science and Machine Learning with Scala and Spark (Episode 01/03), we covered the basics of Scala programming language while using a Google Colab environment. In this article, we learn about the Spark ecosystem and its higher-level API for Scala users. As before, we still use Spark 3.0.0 and Google Colab for practicing some code snippets.
According to Apache Spark and Delta Lake Under the Hood
Apache Spark is a unified computing engine and a set of libraries for parallel data processing on computer clusters. As of the time this writing, Spark is the most actively developed open source engine for this task; making it the de facto tool for any developer or data scientist interested in big data. Spark supports multiple widely used programming languages (Python, Java, Scala and R), includes libraries for diverse tasks ranging from SQL to streaming and machine learning, and runs anywhere from a laptop to a cluster of thousands of servers. This makes it an easy system to start with and scale up to big data processing or incredibly large scale.
I highly recommend reading the Apache Spark and Delta Lake Under the Hood quick reference booklet. It is a 45-page document with examples in Scala and pyspark APIs and would not take more than 30 minutes to go through.
Being a data scientist, we can ask the question “what is the importance of Spark for machine learning tasks?” With the increase in the use of electronic devices, social media platforms, and advanced IT systems rollouts, data is being generated at a level never seen before. Also, due to cheap data storage, clients are happy to collect big data for values to be extracted. Machine Learning models have been proven and work for business to better understand and strategize for future expansions. For having access to big data, Spark is the de-facto choice for machine learning to churn through enormous volumes of collected data to build models.
DataFrame API is the most important higher-level API for Machine Learning. Spark’s MLlib has dropped the support of RDD in favor of DataFrame API.
Scala was selected to be the core the language to write the Spark engine. However, Apache Spark provides high-level APIs in Java, Scala, Python, and R. In this article, we will use spark 3.0.0.
Below are the commands to get modules for Scala and Spark in Google colab
// Run below commandsimport $ivy.`org.apache.spark::spark-sql:3.0.0`import org.apache.spark.sql._import $ivy.`sh.almond::ammonite-spark:0.3.0`
There are two important abstractions of Spark, Spark Context, and Spark Session. Using the below lines, we create a spark session and context.
import org.apache.spark.SparkContextimport org.apache.spark.SparkConf
Usually, when running spark there are a lot of warnings, using below commands, you can turn them off.
// TIP: Turn off the millions lines of logsimport org.apache.log4j.{Level, Logger}Logger.getLogger(“org”).setLevel(Level.OFF)// define spark sessionval spark = {SparkSession.builder().master("local[*]").getOrCreate()}// Define Spark Contextdef sc = spark.sparkContext
Resilient Distributed Datasets (RDD) is the most common abstraction of Spark data structure related closely with Scala. It is very similar to Scala native parallel feature. Let's write some snippets about RRD in Scala.
// Spark RDDimport scala.util.Random// Define variable in Scalaval bigRng = scala.util.Random.shuffle(1 to 100000)// convert Scala variable to spark RDDval bigPRng = sc.parallelize(bigRng)
You can apply many operations on bigPRng , it will run on Spark.
// calculate the mean of the populationbigPRng.mean// Find the min of the populationbigPRng.min// Find the stanndard deviation of the populationbigPRng.popStdev
Applying the function on each element of RDD is very similar to Scala parallel map function.
// Map function on RDD, similar to Paralell in Scalaval bigPRng2 = bigPRng.map(_ * 2)// Scala function function to apply on RDDdef div3(x:Int) : Boolean = {val y:Int=(x%3); return(y==0)}val bigBool = bigPRng2.map(div3(_))
In the previous sections, we have included examples about the RDD and how RDD can be used to compute in parallel. Another popular API in spark is Dataframe. Coming from a Data Scientist background, probably, DataFrame API would make more sense. However, there is a major difference is how to spark data frames and pandas data frames operate under the hood.
Spark DataFrames can live on multiple physical machines and hence their computations are performed in distributed way opposed to pandas DataFrames.
Spark MLlib is implemented in DataFrame API going forward.
Let us learn some basic tricks of Spark DataFrame API mostly useful for machine learning tasks.
Read through Section 0 in the Google colab about the preparation of data for below snippets.
// Read the .txt fileval df_emps = spark.read.option("header", "true").csv(data_dir + "employee.txt")// print the schemadf_emps.printSchema()// show top 10 records similar to df.head(10) in pandasdf_emps.show(10, false)
Read the second table
// Read the .txt fileval df_cr = spark.read.option("header", "true").csv(data_dir + "country_region.txt")// print the schemadf_cr.printSchema()// show top 10 records similar to df.head(10) in pandasdf_cr.show(10, false)
Read the third table
// Read the .txt fileval df_dd = spark.read.option("header", "true").csv(data_dir + "dept_div.txt")// print the schemadf_dd.printSchema()// show top 10 records similar to df.head(10) in pandasdf_dd.show(10, false)
Merging is one of the most common and efficient operations in Spark compared with pandas.
// Merge all three tablesval df_joined = df_emps.join(df_cr, "region_id").join(df_dd, "department")
I have prepared a functional Google colab notebook. Feel free to use the notebook for practice.
In this tutorial, we will learn data science and machine learning with scala.
SCALA is functional and objecte oriented progarmming running with JVM with aim to be scalable hence name as SCALA.
Spark is written in scala which makes scala a native choice for processing big data. We will also cover the basic of using scala with spark and machine learning using spark and scala.
Key Objectives

Install Scala to work with Google Colab
Introduction and basics of Scala
Scala for Data Science
Scala with Spark Introduction
Machine Learning with Scala

Install Scala to work with Google Colab
Introduction and basics of Scala
Scala for Data Science
Scala with Spark Introduction
Machine Learning with Scala
Above is the bigger scope of this tutorial. The aim is to provide a working google colab notebook for all above sections.
In this tutorial, we will focus on spark computations using Scala.
To get a better understanding of spark framework and its modules, I would highly recommend [Apache SparkTM Under the Hood] https://databricks.com/p/ebook/apache-spark-under-the-hood. We will go through some of the examples given in this hand book. Therefore, download the associated data from this github data Location.
Another source is using excercise files from https://www.lynda.com/Scala-tutorials/Scala-Essential-Training-Data-Science/559182-2.html
I have downloaded the data and put it together in one of my repositories for easy access on this consoldated blog data
We need to mount the google drive to read the data stored in google drive. Below is the simplest way to mount it. You will asked to enter the token generated by your access procedure. Her eis the link to the article about mounting gdrive
from google.colab import drive
drive.mount('/content/drive')

Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive

# Location of exercise data (I have uploaded the data to my google drive). Youhave to change to your location.
root_example_data = '/content/drive/My Drive/scala_machine_learning/'

!ls '/content/drive/My Drive/scala_machine_learning/'

departuredelays.csv.gz	Ex_Files_Scala_EssT_Data_Science

I followed the tutorial Machine Learning with Scala in Google Colaboratory by Shadad Laddad to start Scala kernel in the google colab.
If you get a "scala" kernel not recognized warning when loading up the notebook for the first time, start by running the two cells below. Once you are done reload the page to load the notebook in the installed Scala kernel.
%%shell
SCALA_VERSION=2.12.8 ALMOND_VERSION=0.3.0+16-548dc10f-SNAPSHOT
curl -Lo coursier https://git.io/coursier-cli
chmod +x coursier
./coursier bootstrap \
    -r jitpack -r sonatype:snapshots \
    -i user -I user:sh.almond:scala-kernel-api_$SCALA_VERSION:$ALMOND_VERSION \
    sh.almond:scala-kernel_$SCALA_VERSION:$ALMOND_VERSION \
    --sources --default=true \
    -o almond-snapshot --embed-files=false
rm coursier
./almond-snapshot --install --global --force
rm almond-snapshot

%%shell
echo "{
  \"language\" : \"scala\",
  \"display_name\" : \"Scala\",
  \"argv\" : [
    \"bash\",
    \"-c\",
    \"env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libpython3.6m.so:\$LD_PRELOAD java -jar /usr/local/share/jupyter/kernels/scala/launcher.jar --connection-file {connection_file}\"
  ]
}" > /usr/local/share/jupyter/kernels/scala/kernel.json

Lets check if the scala is working or not. if it does not work refresh the page.
println("Hello, world!")

Hello, world!

Apache Spark is a lightning fast cluster computing system solving the limititaion of previously  favoutite MapReduce systems for large data sets. It is the framework of choice among data scientists and machine learning engineers to work with big data problems.
Scala was selected is preffered languange to write Spark engine. However Apcahe Spark privides high level  APIs in Java, Scala, Python and R.
According to Apache Spark Introduction for Beginners
Spark is one of Hadoop's sub venture created in 2009 in UC Berkeley's AMPLab by Matei Zaharia. It was Open Sourced in 2010 under a BSD license. It was given to Apache programming establishment in 2013, and now Apache Spark has turned into the best level Apache venture from Feb-2014. And now the results are pretty booming.
Spark 3.0.0 released on 18th June 2020 after passing the vote on 10th of June 2020. However, the preview of Spark 3.0.0 was released late 2019.
Spark 3.0 is roughly two times faster than Spark 2.4
In this tutorial, we will use Scala as higher level API to run spark computations.
To learn more about Spark, you may want to read below articles

https://towardsdatascience.com/introduction-to-apache-spark-207a479c3001
https://spark.apache.org/
https://medium.com/@amjadraza24/spark-ifying-pandas-databricks-koalas-with-google-colab-93028890db5
https://medium.com/@amjadraza24/getting-started-spark3-0-0-with-google-colab-9796d350d78

https://towardsdatascience.com/introduction-to-apache-spark-207a479c3001
https://spark.apache.org/
https://medium.com/@amjadraza24/spark-ifying-pandas-databricks-koalas-with-google-colab-93028890db5
https://medium.com/@amjadraza24/getting-started-spark3-0-0-with-google-colab-9796d350d78
// Run below commands
import $ivy.`org.apache.spark::spark-sql:3.0.0`

import $ivy.$                                  
import org.apache.spark.sql._

import org.apache.spark.sql._
import $ivy.`sh.almond::ammonite-spark:0.3.0`

import $ivy.$                                
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

import org.apache.spark.SparkConf
// TIP: Turn off the millions lines of logs
import org.apache.log4j.{Level, Logger}
Logger.getLogger("org").setLevel(Level.OFF)

import org.apache.log4j.{Level, Logger}

val conf = new SparkConf().setAppName("test_scala_machine_learning").setMaster("local[*]")
new SparkContext(conf)

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

conf: SparkConf = org.apache.spark.SparkConf@34a88b7a
res6_1: SparkContext = org.apache.spark.SparkContext@5368b6be
# spark.stop()

20/06/21 09:04:44 INFO SparkContext: SparkContext already stopped.

val spark = {
    SparkSession.builder()
      .master("local[*]")
      .getOrCreate()
  }

spark: SparkSession = org.apache.spark.sql.SparkSession@7dec4dc6
def sc = spark.sparkContext

defined function sc
Resilient Distributed Datasets (RDD) is the most common abstraction of Spark data structure related closely with Scala. It is very similar to Scala native parallel feature. Let's write some snippets about RRD in Scala.
// Spark RDD
import scala.util.Random

import scala.util.Random
val bigRng = scala.util.Random.shuffle(1 to 100000)

bigRng: collection.immutable.IndexedSeq[Int] = Vector(
  56811,
  43866,
  39908,
  15474,
  21501,
  96554,
  29846,
  56881,
  29337,
  12277,
  98105,
  31018,
  49771,
  86256,
  43551,
  66319,
  50308,
  57732,
  27969,
  15028,
  60441,
  4125,
  92598,
  29572,
  22279,
  39084,
  73300,
  64156,
  19123,
  50993,
  81177,
  59026,
  70870,
  42779,
  15839,
  52508,
  22562,
  69156,
...
// convert to RDD
val bigPRng = sc.parallelize(bigRng)

bigPRng: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at cmd11.sc:1
// calculate the mean of the population
bigPRng.mean

res12: Double = 50000.50000000018
// Find the min of the population
bigPRng.min

res25: Int = 1
// Find the max of the population
bigPRng.max

res13: Int = 100000
// Find the stanndard deviation of the population
bigPRng.popStdev

res14: Double = 28867.513458037993
bigPRng.take(25)

res28: Array[Int] = Array(
  95614,
  44945,
  85237,
  90417,
  92032,
  55788,
  14288,
  32807,
  44044,
  9755,
  3331,
  94537,
  5991,
  61000,
  35500,
  84313,
  49472,
  29350,
  92789,
  78992,
  42190,
  60047,
  15325,
  84589,
  49274
)
// Map function on RDD, similar to Paralell in Scala
val bigPRng2 = bigPRng.map(_ * 2)

bigPRng2: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[5] at map at cmd15.sc:1
bigPRng2.take(25)

res30: Array[Int] = Array(
  191228,
  89890,
  170474,
  180834,
  184064,
  111576,
  28576,
  65614,
  88088,
  19510,
  6662,
  189074,
  11982,
  122000,
  71000,
  168626,
  98944,
  58700,
  185578,
  157984,
  84380,
  120094,
  30650,
  169178,
  98548
)
bigPRng2.mean()

res16: Double = 100001.00000000036
// Scala function function to apply on RDD
def div3(x:Int) : Boolean = {val y:Int=(x%3); return(y==0)}

defined function div3
div3(9)

res18: Boolean = true
val bigBool = bigPRng2.map(div3(_))

bigBool: org.apache.spark.rdd.RDD[Boolean] = MapPartitionsRDD[8] at map at cmd19.sc:1
bigBool.take(25)

res20: Array[Boolean] = Array(
  true,
  true,
  false,
  true,
  true,
  false,
  false,
  false,
  true,
  false,
  false,
  false,
  false,
  true,
  true,
  false,
  false,
  true,
  true,
  false,
  true,
  true,
  true,
  false,
  false
)
In the previous sections, we have included examples about the RDD and how RDD can be used to compute in parallel. Another popular API in spark is Dataframe. Coming from a Data Scientist background, probably, DataFrame API would make more sense. However, there is a major difference is how to spark data frames and pandas data frames operate under the hood.
Spark DataFrames can live on multiple physical machines and hence their computations are performed in distributed way opposed to pandas DataFrames.
Spark MLlib is implemented in DataFrame API going forward.
Let us learn some basic tricks of Spark DataFrame API mostly useful for machine learning tasks.
//spark data frames

val data_dir  = "/content/drive/My Drive//scala_machine_learning/Ex_Files_Scala_EssT_Data_Science/Exercise Files/Chapter 5/05_01/"

data_dir: String = "/content/drive/My Drive//scala_machine_learning/Ex_Files_Scala_EssT_Data_Science/Exercise Files/Chapter 5/05_01/"
val df_emps = spark.read.option("header", "true").csv(data_dir + "employee.txt")

df_emps: DataFrame = [id: string, last_name: string ... 7 more fields]
// print schema
df_emps.printSchema()

root
 |-- id: string (nullable = true)
 |-- last_name: string (nullable = true)
 |-- email: string (nullable = true)
 |-- gender: string (nullable = true)
 |-- department: string (nullable = true)
 |-- start_date: string (nullable = true)
 |-- salary: string (nullable = true)
 |-- job_title: string (nullable = true)
 |-- region_id: string (nullable = true)


// show top 10 records similar to df.head(10) in pandas
df_emps.show(10, false)

+---+------------+----------------------------+--------+------------+------------+------+----------------------------+---------+
|id |last_name   |email                       |gender  |department  |start_date  |salary|job_title                   |region_id|
+---+------------+----------------------------+--------+------------+------------+------+----------------------------+---------+
|1  |'Kelley'    |'rkelley0@soundcloud.com'   |'Female'|'Computers' |'10/2/2009' |67470 |'Structural Engineer'       |2        |
|2  |'Armstrong' |'sarmstrong1@infoseek.co.jp'|'Male'  |'Sports'    |'3/31/2008' |71869 |'Financial Advisor'         |2        |
|3  |'Carr'      |'fcarr2@woothemes.com'      |'Male'  |'Automotive'|'7/12/2009' |101768|'Recruiting Manager'        |3        |
|4  |'Murray'    |'jmurray3@gov.uk'           |'Female'|'Jewelery'  |'12/25/2014'|96897 |'Desktop Support Technician'|3        |
|5  |'Ellis'     |'jellis4@sciencedirect.com' |'Female'|'Grocery'   |'9/19/2002' |63702 |'Software Engineer III'     |7        |
|6  |'Phillips'  |'bphillips5@time.com'       |'Male'  |'Tools'     |'8/21/2013' |118497|'Executive Secretary'       |1        |
|7  |'Williamson'|'rwilliamson6@ted.com'      |'Male'  |'Computers' |'5/14/2006' |65889 |'Dental Hygienist'          |6        |
|8  |'Harris'    |'aharris7@ucoz.com'         |'Female'|'Toys'      |'8/12/2003' |84427 |'Safety Technician I'       |4        |
|9  |'James'     |'rjames8@prnewswire.com'    |'Male'  |'Jewelery'  |'9/7/2005'  |108657|'Sales Associate'           |2        |
|10 |'Sanchez'   |'rsanchez9@cloudflare.com'  |'Male'  |'Movies'    |'3/13/2013' |108093|'Sales Representative'      |1        |
+---+------------+----------------------------+--------+------------+------------+------+----------------------------+---------+
only showing top 10 rows


val df_cr = spark.read.option("header", "true").csv(data_dir + "country_region.txt")

df_cr: DataFrame = [region_id: string, company_regions: string ... 1 more field]
df_cr.printSchema()

root
 |-- region_id: string (nullable = true)
 |-- company_regions: string (nullable = true)
 |-- country: string (nullable = true)


df_cr.show(10, false)

+---------+-------------------+---------+
|region_id|company_regions    |country  |
+---------+-------------------+---------+
|1        | 'Northeast'       | 'USA'   |
|2        | 'Southeast'       | 'USA'   |
|3        | 'Northwest'       | 'USA'   |
|4        | 'Southwest'       | 'USA'   |
|5        | 'British Columbia'| 'Canada'|
|6        | 'Quebec'          | 'Canada'|
|7        | 'Nova Scotia'     | 'Canada'|
+---------+-------------------+---------+


val df_dd = spark.read.option("header", "true").csv(data_dir + "dept_div.txt")

df_dd: DataFrame = [department: string, company_division: string]
df_dd.show(10, false)

+-------------+----------------------+
|department   |company_division      |
+-------------+----------------------+
|'Automotive' |'Auto & Hardware'     |
|'Baby'       |'Domestic'            |
|'Beauty'     |'Domestic'            |
|'Clothing'   |'Domestic'            |
|'Computers'  |'Electronic Equipment'|
|'Electronics'|'Electronic Equipment'|
|'Games'      |'Domestic'            |
|'Garden'     |'Outdoors & Garden'   |
|'Grocery'    |'Domestic'            |
|'Health'     |'Domestic'            |
+-------------+----------------------+
only showing top 10 rows


// merge all thres tables

val df_joined = df_emps.join(df_cr, "region_id").join(df_dd, "department")

df_joined: DataFrame = [department: string, region_id: string ... 10 more fields]
df_joined.columns

res35: Array[String] = Array(
  "department",
  "region_id",
  "id",
  "last_name",
  "email",
  "gender",
  "start_date",
  "salary",
  "job_title",
  "company_regions",
  "country",
  "company_division"
)
df_joined.show(20, false)

+-------------+---------+---+------------+----------------------------+--------+------------+------+-------------------------------+-------------------+---------+----------------------+
|department   |region_id|id |last_name   |email                       |gender  |start_date  |salary|job_title                      |company_regions    |country  |company_division      |
+-------------+---------+---+------------+----------------------------+--------+------------+------+-------------------------------+-------------------+---------+----------------------+
|'Computers'  |2        |1  |'Kelley'    |'rkelley0@soundcloud.com'   |'Female'|'10/2/2009' |67470 |'Structural Engineer'          | 'Southeast'       | 'USA'   |'Electronic Equipment'|
|'Sports'     |2        |2  |'Armstrong' |'sarmstrong1@infoseek.co.jp'|'Male'  |'3/31/2008' |71869 |'Financial Advisor'            | 'Southeast'       | 'USA'   |'Games & Sports'      |
|'Automotive' |3        |3  |'Carr'      |'fcarr2@woothemes.com'      |'Male'  |'7/12/2009' |101768|'Recruiting Manager'           | 'Northwest'       | 'USA'   |'Auto & Hardware'     |
|'Jewelery'   |3        |4  |'Murray'    |'jmurray3@gov.uk'           |'Female'|'12/25/2014'|96897 |'Desktop Support Technician'   | 'Northwest'       | 'USA'   |'Fashion'             |
|'Grocery'    |7        |5  |'Ellis'     |'jellis4@sciencedirect.com' |'Female'|'9/19/2002' |63702 |'Software Engineer III'        | 'Nova Scotia'     | 'Canada'|'Domestic'            |
|'Tools'      |1        |6  |'Phillips'  |'bphillips5@time.com'       |'Male'  |'8/21/2013' |118497|'Executive Secretary'          | 'Northeast'       | 'USA'   |'Auto & Hardware'     |
|'Computers'  |6        |7  |'Williamson'|'rwilliamson6@ted.com'      |'Male'  |'5/14/2006' |65889 |'Dental Hygienist'             | 'Quebec'          | 'Canada'|'Electronic Equipment'|
|'Toys'       |4        |8  |'Harris'    |'aharris7@ucoz.com'         |'Female'|'8/12/2003' |84427 |'Safety Technician I'          | 'Southwest'       | 'USA'   |'Games & Sports'      |
|'Jewelery'   |2        |9  |'James'     |'rjames8@prnewswire.com'    |'Male'  |'9/7/2005'  |108657|'Sales Associate'              | 'Southeast'       | 'USA'   |'Fashion'             |
|'Movies'     |1        |10 |'Sanchez'   |'rsanchez9@cloudflare.com'  |'Male'  |'3/13/2013' |108093|'Sales Representative'         | 'Northeast'       | 'USA'   |'Entertainment'       |
|'Jewelery'   |7        |11 |'Jacobs'    |'jjacobsa@sbwire.com'       |'Female'|'11/27/2003'|121966|'Community Outreach Specialist'| 'Nova Scotia'     | 'Canada'|'Fashion'             |
|'Clothing'   |7        |12 |'Black'     |'mblackb@edublogs.org'      |'Male'  |'2/4/2003'  |44179 |'Data Coordiator'              | 'Nova Scotia'     | 'Canada'|'Domestic'            |
|'Baby'       |3        |13 |'Schmidt'   |'sschmidtc@state.gov'       |'Male'  |'10/13/2002'|85227 |'Compensation Analyst'         | 'Northwest'       | 'USA'   |'Domestic'            |
|'Computers'  |4        |14 |'Webb'      |'awebbd@baidu.com'          |'Female'|'10/22/2006'|59763 |'Software Test Engineer III'   | 'Southwest'       | 'USA'   |'Electronic Equipment'|
|'Games'      |7        |15 |'Jacobs'    |'ajacobse@google.it'        |'Female'|'3/4/2007'  |141139|'Community Outreach Specialist'| 'Nova Scotia'     | 'Canada'|'Domestic'            |
|'Baby'       |1        |16 |'Medina'    |'smedinaf@amazonaws.com'    |'Female'|'3/14/2008' |106659|'Web Developer III'            | 'Northeast'       | 'USA'   |'Domestic'            |
|'Kids'       |6        |17 |'Morgan'    |'dmorgang@123-reg.co.uk'    |'Female'|'5/4/2011'  |148952|'Programmer IV'                | 'Quebec'          | 'Canada'|'Domestic'            |
|'Home'       |5        |18 |'Nguyen'    |'jnguyenh@google.com'       |'Male'  |'11/3/2014' |93804 |'Geologist II'                 | 'British Columbia'| 'Canada'|'Domestic'            |
|'Electronics'|3        |19 |'Day'       |'rdayi@chronoengine.com'    |'Male'  |'9/22/2004' |109890|'VP Sales'                     | 'Northwest'       | 'USA'   |'Electronic Equipment'|
|'Movies'     |5        |20 |'Carr'      |'dcarrj@ocn.ne.jp'          |'Female'|'11/22/2007'|115274|'VP Quality Control'           | 'British Columbia'| 'Canada'|'Entertainment'       |
+-------------+---------+---+------------+----------------------------+--------+------------+------+-------------------------------+-------------------+---------+----------------------+
only showing top 20 rows


for more examples, you can follow the official documentations at
In this episode, we have learned the basics of Spark with Scala and covered below key concepts with exercises.
Running Scala in Google colab
Basic's of Spark
Spark's RDD with Scala
Sparks' DataFrame API with Scala
In the next episode, we will learn about Machine Learning models with Spark and Scala using Google colab runtime.
 

 

For more examples, consult the Spark official documentation
In this episode, we have learned the basics of Spark with Scala and covered below key concepts with exercises.
Running Scala in Google Colab
Basic’s of Spark
Spark’s RDD with Scala
Sparks’ DataFrame API with Scala
In the next episode, we will learn about Machine Learning models with Spark and Scala using Google Colab runtime.
Machine Learning with Scala in Google Colaboratory
Scala Docs -https://docs.scala-lang.org/tour/tour-of-scala.html