Snowflake with Spark
By: Mouhamad Keita
Tags: Spark, Snowsql, Big Data, Snowflake, Cloud

Snowflake is a cloud-based SQL data warehouse that focuses on a great performance, zero-tuning, diversity of data sources, and security.

The Snowflake Connector for Spark enables using Snowflake as an Apache Spark data source, similar to other data sources (PostgreSQL, HDFS, S3, etc.). Snowflake supports three versions of Spark: Spark 2.2, Spark 2.3, and Spark 2.4. There is a separate version of the Snowflake Spark Connector for each version of Spark. Use the correct version of the connector for your version of Spark.

In this topic, we will use Spark 2.4 and see how to read and write Dataframe to Snowflake database.

Make sure you follow the topic about creating Snowflake Database here.

SBT dependency

libraryDependencies ++= Seq(

"net.snowflake" %% "spark-snowflake" % "2.7.0-spark_2.4"

)

Create a Snowflake TABLE

To create a Database in Snowflake, please refer to the topic SQL on SnowSQL. To access to Snowflake Database we have to create a new table. We can use the below program to create a table.

package com.sparkSnowFlake



import java.sql.DriverManager



object CreateSnowflakeTable extends App {



val properties = new java.util.Properties()

properties.put("user", "mouhamadkeita92")

properties.put("password", "*********")

properties.put("account", "re54891")

properties.put("warehouse", "COMPUTE_WH")

properties.put("db", "PEOPLE")

properties.put("schema", "public")

properties.put("role", "SYSADMIN")



val jdbcUrl = "jdbc:snowflake://re54891.east-us-2.azure.snowflakecomputing.com/"



println("Created connection")

val connection = DriverManager.getConnection(jdbcUrl, properties)

val statement = connection.createStatement

println("Done creating connection")



println("Creating table PEOPLE")

statement.executeUpdate("create or replace table PEOPLE(name VARCHAR, age INT, job VARCHAR)")

statement.close()

println("Done creating PEOPLE table")



connection.close()

}

In this Code we create a Database “PEOPLE” with Spark. We can also create the Database with SnowSQL CLI or Snowflake GUI.

Write Spark DataFrame to Snowflake

To read data from Snowflake into a Spark DataFrame we use the write() method of the DataFrame to construct a DataFrameWriter .

package com.sparkSnowFlake



import org.apache.spark.sql.{SaveMode, SparkSession}



object WriteDataToSnowflake extends App {



val spark = SparkSession.builder()

.master("local[*]")

.appName("SparkSnowflakeWrite")

.getOrCreate()

spark.sparkContext.setLogLevel("ERROR")

import spark.implicits._

var sfParameters = Map(

"sfURL" -> "https://re54891.east-us-2.azure.snowflakecomputing.com/",

"sfAccount" -> "re54891",

"sfUser" -> "mouhamadkeita92",

"sfPassword" -> "Ousmane92.",

"sfDatabase" -> "PEOPLE",

"sfSchema" -> "PUBLIC",

"sfRole" -> "ACCOUNTADMIN"

)



val data = Seq(("Jorge",23,"Developer"),

("Bob",28,"Developer"),

("Bill",31,"Admin"),

("John",40,"Project Manager")

)

val peopleDF = data.toDF("name","age","job")



peopleDF.write

.format("snowflake")

.options(sfParameters)

.option("dbtable", "PEOPLE")

.mode(SaveMode.Overwrite)

.save()

}

In this code, we insert data into table PEOPLE in the database PEOPLE. In the next section, we will read data saved in table PEOPLE into a Dataframe.

READ Snowflake table into Spark Dataframe

To read data from Snowflake into a Spark DataFrame we use the read() method of the SqlContext object to construct a DataFrameReader .

When using DataFrames, the Snowflake connector supports SELECT queries only.

package com.sparkSnowFlake



import org.apache.spark.sql.{DataFrame, SparkSession}



object ReadDataFromSnowflake extends App {



val spark = SparkSession.builder()

.master("local[*]")

.appName("SparkByExamples.com")

.getOrCreate()

var sfParameters = Map(

"sfURL" -> "https://re54891.east-us-2.azure.snowflakecomputing.com/",

"sfAccount" -> "re54891",

"sfUser" -> "mouhamadkeita92",

"sfPassword" -> "********",

"sfDatabase" -> "PEOPLE",

"sfSchema" -> "PUBLIC",

"sfRole" -> "ACCOUNTADMIN"

)



val df: DataFrame = spark.read

.format("snowflake")

.options(sfParameters)

.option("dbtable", "PEOPLE")

.load()

df.show(false)

}

The output

READ Snowflake table With Query

Above Snowflake with Spark example demonstrates reading the entire table from the Snowflake table using dbtable option and creating a Spark DataFrame, below example uses a query option to execute a filter SQL query.

val df: DataFrame = spark.read

.format("snowflake")

.options(sfParameters)

.option("query", "SELECT * FROM PEOPLE WHERE job = 'Developer'")

.load()

df.show(false)

The output

Check Data via SnowSQL CLI

We can read data from SnowSQL to make sure everything is OK.

❯ snowsql

* SnowSQL * v1.2.5

Type SQL statements or !help

mouhamadkeita92#COMPUTE_WH@PEOPLE.PUBLIC>SELECT * FROM "PEOPLE"."PUBLIC"."PEOPLE"

;

+-------+-----+-----------------+

| NAME | AGE | JOB |

|-------+-----+-----------------|

| Jorge | 23 | Developer |

| Bob | 28 | Developer |

| Bill | 31 | Admin |

| John | 40 | Project Manager |

+-------+-----+-----------------+

4 Row(s) produced. Time Elapsed: 1.795s

mouhamadkeita92#COMPUTE_WH@PEOPLE.PUBLIC>

References :