fabiochiu/medium-articles huggingface dataset:
Apache Drill vs. Apache Spark — Which SQL query engine is better for you?
By: Sunny Srinidhi
Tags: Technology, Big Data, Apache Spark, Data Science, Apache Drill

It is very important to understand that there is a fundamental difference between the two, how they are implemented, and what they are capable of.

With Apache Drill, we write SQL queries to fetch data from a variety of sources, such as SQL databases, MongoDB, AWS S3, Apache Kafka, JSON files, and many more. But with Apache Spark, we write “SQL-Like” queries to fetch data from various data sources. The difference here is that Drill is an ANSI SQL:2003 query engine, with more powerful features than a traditional SQL query engine. Apache Spark, on the other hand, supports only a subset of SQL queries, and in a limited fashion. Spark is a generic computation engine, which has support for SQL like queries to get the data that it needs to run those computations on. Spark also has drivers for various storage services on which you want to run your queries.

The list of data sources that SparkSQL supports is also quite long, but not as long as that of Drill. People who have been working with BI tools for long enough find themselves at home with SQL queries. Bringing that knowledge with them over to Apache Drill will enable them to query data from data sources which might be new to them, such as NoSQL databases, Parquet files, S3 files, etc. Such people can run complex aggregation queries using Apache Drill; and most of the time, they get the results they need. The same cannot be said about SparkSQL.

SparkSQL is ideal in cases where you to process data inside Spark, and you want to fetch data from external sources with some simple joins and where conditions. Once you get the data into your Spark Datasets, you can then apply your transformations and perform your actions.

The way these two tools handle the underlying data is quite different. Apache Drill is able to take complex queries which run on multiple data sources, and is able to push down those queries to native drivers of the concerned data sources. It maps the data internally to JSON formats, and it’s ability to work with JSON makes Drill much more flexible in a way compared to Spark.

Drill also extends the ANSI SQL queries with commands such as FLATTEN and KVGEN, which make it easier to handle simple and even complex JSON data. And since Drill works with JSON internally, it enables Drill to discover schema on the fly.
Video: Batched To Perfection — Modeling & Solving Business Problems With Apache Spark
By: Riskified Technology
Tags: Delta Lake, Data, Apache Spark, Videos, Data Engineering

Batched To Perfection: Modeling & Solving Business Problems With Apache Spark — Eliav Lavi

As data gets bigger, the applications we’re maintaining as developers are becoming increasingly more data hungry. There comes a point where simply querying all raw data and crunching it into some meaningful piece of information upon request at runtime is just not giving us good enough performance. Maybe our application starts lagging. This is when pre-calculating our aggregations become crucial.

In this talk, we examine how can Apache Spark helps in building elegant and accurate batch processing data pipelines. This allows us to maintain our pre-calculated aggregations and make our web applications run blazingly fast again. Along the way we make use of some other cool technologies, such as Databricks’ Delta Lake, and throw some functional programming goodness into the process.

Eliav Lavi is a technical lead @ Riskified, where he’s been working for the past 7 years. Previously a classically trained musician, he shifted to a developer position in order to pursue his long-time passion for tech. Today he is part of Riskified’s Account Protection team, preventing bad actors from taking over customers’ accounts.

*Talk is in English
Testing Spark Structured Streaming Applications:
By: Scott Haines
Tags: Testing, Big Data, Structured Streaming, Apache Kafka, Apache Spark

Let’s now walk through the test case above. The pattern for loading a scenario comes from trial by fire while trying to figure out the easiest way to generate repeatable tests based off of real fake data.

My team at Twilio (Voice Insights ) came up with some neat internal generators of fake telephony data — modeled off of real live traffic. Through the generator we can create true to life scenarios instead of just testing randomly generated data or some bogus nonsense! This sometimes means we generate thousands of scenarios that will be loaded and tested. This process is replicated in many spark applications.

Our TestHelper has a lot of nice utilities including the ability to load a scenario (basically read json and parse to our protobuf) which was shown below.

def loadScenario[T<: Message : ClassTag](file: String): Seq[T] = {

val fileString = Source.fromFile(file).mkString

val parsed = mapper.readValue(fileString, classOf[Scenario])

parsed.input.map { data =>

val json = mapper.writeValueAsString(data)

convert[T](json)

}

}

example of scenario loading method above ^^

We then create our MemoryStream and take our scenario iterator and generate a pile of Kafka data in the form of our MockKafkaDataFrame.

case class MockKafkaDataFrame(key: Array[Byte], value: Array[Byte])

This allows us to stream data through our unit tests in the same format that we will be reading from Kafka in the real world. This saves us a ton of time over running locally embedded Kafka.

We then create a streaming query that writes to a MemorySink. This allows us to test an end to end streaming query, without the need to Mock out the source and sink in our structured streaming application. This means you can plug in the tried and true DataSources from Spark and focus instead on ensuring that your application code is running hiccup free.

What the hell. That was easy right. Yep.
Apache Spark and Hadoop HDFS: Hello World
By: Teiva Harsanyi
Tags: Apache Spark, Big Data, Programming, Hadoop, Hdfs

This post intends to help people starting their big data journey by helping them to create a simple environment to test the integration between Apache Spark and Hadoop HDFS. It does not intend to describe what Apache Spark or Hadoop is.

Prerequisites

A Docker environment (local or remote).

A DNS entry on our local machine to map hadoop to the Docker host IP address.

Hadoop Docker Image

Pull the teivah/kafka:2.9.2 Docker image:

docker pull teivah/hadoop:2.9.2

This image starts a single-node Hadoop cluster.

Run the Docker image:

docker run -ti --hostname hadoop -p 50070:50070 -p 9000:9000 -p 50075:50075 -p 50010:50010 teivah/hadoop:2.9.2

The command exposes 3 ports: 50070 is the HDFS namenode port, 50075 is the HDFS datanode port and 9000 is the port to be accessed by Spark ( fs.defaultFS configuration).

Once the container is running, go to the web UI: http://hadoop:50070/ and then navigate to Utilities -> Browse the file system.

Upload a file at the root level (you may want to create hierarchies afterward).

Apache Spark Client (in Java)

Git clone the following project: https://github.com/teivah/spark-hdfs-helloworld.

Run mvn clean install to install the project and download the dependencies.

to install the project and download the dependencies. Open io.teivah.spark.SparkHdfsHelloWorld.java . This client connects to an HDFS cluster (hdfs://hadoop:9000/…) and executes a series of queries to counts how many times each word appears in the file.

. This client connects to an HDFS cluster (hdfs://hadoop:9000/…) and executes a series of queries to counts how many times each word appears in the file. Replace FILENAME (line 21) by the name of the file you just uploaded.

(line 21) by the name of the file you just uploaded. Execute the application. At the end of the execution, you should have a part-00000 file created in the /output folder representing the results.

You are now ready to play with Apache Spark and Hadoop HDFS.
How to Optimize Read CSV in Apache spark
By: Tarun Kumar
Tags: Optimization, Apache Spark

Reading CSV is very common practice in big data. Normally we use spark api to read csv.

Using below method you can read csv 30% faster in spark.

e.g. spark.read().csv(“path”)

In one of our application we were reading and processing 150 Tera bytes of raw csv files daily. In order to improve performance of pipeline, we started with read process.

On doing analysis of how much time is spent in various java methods’ got this result.

Most of time was spent on converting bytes to String and vice versa and trim methods. Spark’s internal catalyst engine uses, InternalRow format. Read CSV method was converting bytes from internal row format to string , splitting, trimming and again converting to bytes format.

We could eliminate this all together. We read text format and did all parsing and trimming in Internal Row Format. We didn’t call String split and String trim methods as they are very heavy. You can clearly see, now most of time is spent just for doing some internal

This resulted in 30% read time save in spark.

Code is attached below. You need to read spark text read and call this class in map partition to iterate on each map partition and split and read as csv format.
Airflow, Spark & S3, stitching it all together
By: Suman Kumar Gangopadhyay
Tags: Machine Learning, Airflow, Apache Spark, Artificial Intelligence, Data Engineering

As a first step, we will setup a MySQL RDS instance in AWS

in AWS, select RDS as the service and then click on ‘Create Database’ from the next page. Then follow the screenshots below for choosing the remaining options

make a note of the credentials that you provided here, as these will be needed while setting up airflow. Leave all the other options as-is until you arrive at the screen (by scrolling down) below

Note that, I have chosen the EMR Master’s security group as the VPC security group along with the default. This is essential to allow the communication between the edge node (where airflow will be setup) and the RDS

expand the ‘Additional configuration’ and ensure that the port is 3306. Scroll down further and expand the next ‘Additional Configuration’. Enter the database name of your choice and make a note of this name, it will also be needed during the airflow setup

Leave everything else as is and then hit ‘Create database’. Stay calm while AWS takes its own time to set up the MySQL database

Once the database is up and running, click on the DB Identifier

In the next screen, make a note of the End Point , this will be the database host

Next scroll down to see the security groups and click on the default one. We will have to allow inbound traffic from the edge node to this database

Hit Edit Inbound Rules and then add the following (the public IP of the edge node should go into the blank space)

Hit Save rules

Let us now test if we can actually connect to the database from the edge node. Perform the following steps to check this

SSH into the edge node (duh! obvious), then run the below command

mysql -h < > -u < > -p < > (then enter password on prompt)

You should be able to see the mysql prompt now, run the below command in the mysql prompt. This will ensure that airflow gets the necessary character support during the metadata setup (not running this will make airflow db initialisation fail)

alter database <<the database name that you gave in the RDS setup>> CHARACTER SET utf8 COLLATE utf8_unicode_ci;

Now your database is all set to support airflow installation and subsequent job execution.
Running Spark Jobs on a Remote Databricks Cluster using Databricks Connect
By: Charith Ekanayake
Tags: Intellij, Azure Databricks, Vscode, Apache Spark

Running Spark Jobs on a Remote Databricks Cluster using Databricks Connect Charith Ekanayake Apr 23·4 min read

Databricks Connect is a client library for Databricks Runtime. It allows you to write jobs using Spark APIs and run them remotely on a Databricks cluster instead of in the local Spark session.

This allows developers to develop locally in an IDE they prefer and run the workload remotely on a Databricks Cluster which has more processing power than the local spark session.

Supported Databricks Runtimes

-------------------------------------------------------------------

8.1 ML, 8.1

7.3 LTS ML, 7.3 LTS

6.4 ML, 6.4

5.5 LTS 5.5 ML

I would walk you through step by step how you could achieve this with Python and Scala APIs for Spark. In this example, I would be using the below environments and tools.

Azure Databricks Service

VisualStudio Code,IntelliJ Idea

Python 3.8, JDK 1.8, Scala 2.12.13

Miniconda installed on a PC. (Miniconda)

Hadoop setup on Windows with winutils fix

PySpark Edition

First, I would be creating a virtual environment using Conda prompt

conda create --name envdbconnect python=3.8

conda activate envdbconnect

The compatible Python versions are as below.

Databricks Runtime Version Python Version

---------------------------- -- -------------------

8.1 ML, 8.1 3.8

7.3 LTS ML,7.3 LTS 3.7

6.4 ML, 6.4 3.7

5.5 LTS ML 3.6

5.5 LTS 3.5

Setting up the Client

Uninstall locally installed PySpark if it’s already there, since it will conflict with databricks-connect package installation.

pip uninstall pyspark

Install Databricks Connect client (Available Client versions)

pip install -U "databricks-connect==8.1.*"

Once the installation is completed, we need to configure the client.

Below items need to be noted down before the configuration.

Workspace URL
Apache Spark 3 playground — Back to school with Zeppelin notebooks! Pt.3
By: Omri Keefe
Tags: Jupyter Notebook, Spark, Apache Zeppelin, Big Data, Apache Spark

In this third and final part of our “Apache Spark 3 playground” guide we will be adding a Zeppelin notebook into the mix.

Apache Zeppelin — from Zeppelin assets

Apache Zeppelin, from the Zepplin home page, is:

Web-based notebook that enables data-driven,

interactive data analytics and collaborative documents with SQL, Scala and more.

This will allow us to create a sandbox environment that will allow us to experiment and learn without needing to manage things like building, packaging, or deploying our code.

You can save these environments and share them with other developers, engineers, and analysts.

This guide will explain:

Setup a Zeppelin notebook in a local environment

Using Spark as a backend to explore data

The general guidance for solving an introduction data question (Word-count)

Zeppelin vs Jupyter?

Jupyter is usually to go-to notebook solution for data engines, it has amazing community support and is widely used.

I found Zeppelin to be easier and more intuitive to get up and running and decided to go with it, but both are great applications to know.

It would be interesting to try to recreate this guide using Jupyter for the sake of comparison and deciding what flavor you prefer.

The first two parts of this short series covered:

Setting up a local Spark environment

Adding an S3 mock and automating its creation

The only step you need to complete from them is cloning the guide’s Github repository, we did that in Part 1.

If you’re ready to go, let's start with spinning up our playground.
Sparkify: Predicting the user churn using Apache Spark
By: Abhinav Rohatgi
Tags: Data Science, Churn Prediction, Udacity, Apache Spark

This project is aimed at predicting the User Churn for a fictional digital music streaming service called Sparkify

Churn prediction is one of the most sought after use cases of Big Data in the industry and it helps businesses make proactive decisions of how to retain a customer if the customer is predicted to be churning in the near future by targeting them with promotional campaigns and giving them a reason to stay with the business.

We’ll analyze, explore and model our solution using the industry-wide practice of CRISP-DM (Cross Industry Process for Data Mining) whereby we will be explaining the solution under the following sections:

Business Understanding Data Understanding Prepare Data Data Modeling Evaluate the Results Deploy

Business Understanding

Sparkify is a digital music streaming service that is used for free by listening to some advertisement in between the songs or by paying a monthly subscription fee to have an ads-free experience. Whenever the users feel like then they can also decide to downgrade from premium to free, upgrade from free to premium or even cancel from the service.

When the users cancel from the service, that is when we denote that the user has churned.

Data Understanding & Data Preparation

The dataset provided is basically composed of the log of each user action that is taken on the platform. Each action is marked with a certain timestamp ts when that action took place

In the dataset, we have 691 users with 543705 events

These users perform different actions and click on different pages and as you can see in the distribution, the bulk of the actions that the users perform is clicking on the NextSong

There are also users that have upgraded/downgraded their service and then there are also 99 users that have Cancelled their service, these 99 users will be marked to be denoted as churned.

We will now look at some aspects of the users that have churned and how they compare to the users that haven’t

We will first look at how the gender is distributed compared to the churned users

Male customers are more likely to churn than female customers

We will now look at some metrics and how they compare between the churned and active users
A Simple Apache Spark Example
Tags: Scala, Apache Spark

Apache Spark is a data processing framework that can quickly perform processing tasks on very large data sets, and can also distribute data processing tasks across multiple computers, either on its own or in tandem with other distributed computing tools

About this example

In this post I am sharing a simple Apache Spark example project. The source code used for this example is available here: https://github.com/jobinesh/apache-spark-examples.git

Here is the quick overview of the modules that you may find in this project

spark-job-common : All common classes that you need for building a Spark job are parked here. This approach may help you to avoid boilerplate code in your Spark job implementation

spark-job-impl : A classic word count Spark example is available here. This class may help you to understand the structuring of the source and usage of common classes from spark-job-common module

spark-job-launcher : The SparkLauncher helps you to start Spark applications programmatically.

The Spark application that you may find in the spark-job-impl module reads the text from src/main/resources/demo.txt file

and generates an output file with total count for each word.

The output directory location is configured in src/main/resources/application.conf

How to run this example?

The detailed steps are available here:

https://github.com/jobinesh/apache-spark-examples/blob/master/spark-demo-project/README.md

Enjoy !

This post originally appeared on Jobinesh’s personal blog page: https://www.jobinesh.com/2020/04/tracing-api-calls-in-your-helidon.html

The views expressed on this post are my own and do not necessarily reflect the views of my employer.
The What, Why, and When of Apache Spark
By: Allison Honold
Tags: Big Data, Spark, Apache, Apache Spark

The What, Why, and When of Apache Spark

Photo by Ethan Hoover on Unsplash

What is Spark?

Spark has been called a “general purpose distributed data processing engine”1 and “a lightning fast unified analytics engine for big data and machine learning”². It lets you process big data sets faster by splitting the work up into chunks and assigning those chunks across computational resources.

It can handle up to petabytes (that’s millions of gigabytes!) of data, and manage up to thousands of physical or virtual machines.

Some distinctive features include its use of memory and focus on easy development. It tries to keep all data in memory, only writing to the disk(s) if there is not sufficient memory. Note that this does risk needing to restart from scratch in the case of failure, at the benefit of eliminating the time-intensive processes of reading and writing. It makes development easy with an emphasis on API design and integration with several languages and tools.

Why would you want to use Spark?

Spark has some big pros:

High speed data querying, analysis, and transformation with large data sets.

Compared to MapReduce, Spark offers much less reading and writing to and from the disk, multi-threaded tasks (from Wikipedia: the threads share the resources of a single or multiple cores) within Java Virtual Machine (JVM) processes

Great for iterative algorithms (using a sequence of estimations based on the previous estimate).

As mentioned earlier, the easy to use APIs make a big difference in terms of ease of development, readability, and maintenance.

Super fast, especially for interactive queries. (100x faster than classic Hadoop Hive queries without refactoring the code!)

Supports multiple languages and integrations with other popular products.

Helps make complex data pipelines coherent and easy.

When does Spark work best?

If you are already using a supported language (Java, Python, Scala, R)

Spark makes working with distributed data (Amazon S3, MapR XD, Hadoop HDFS) or NoSQL databases (MapR Database, Apache HBase, Apache Cassandra, MongoDB) seamless

When you’re using functional programming (output of functions only depend on their arguments, not global states)

Some common uses:

Performing ETL or SQL batch jobs with large data sets

Processing streaming, real-time data from sensors, IoT, or financial systems, especially in combination with static data

Using streaming data to trigger a response

Performing complex session analysis (eg. grouping users based on web activity)

Machine Learning tasks

When would you not want to use Spark?

For multi-user systems, with shared memory, Hive may be a better choice². For real time, low latency processing, you may prefer Apache Kafka⁴. With small data sets, it’s not going to give you huge gains, so you’re probably better off with the typical libraries and tools.
Compaction / Merge of parquet files
By: Chris Finlayson
Tags: Apache Spark, Big Data, Optimization, Apache Hadoop, Hdfs

Optimising size of parquet files for processing by Hadoop or Spark

The small file problem

One of the challenges in maintaining a performant data lake is to ensure that files are optimally sized for the storage medium.

General rule of thumb being:

Many small files == bad

Small numbers of large files ==better

(…although we can get a bit more scientific on the ideal file size)

Persisting large amounts of small files is a particular issue on HDFS as the namenode takes the strain in memory for tracking every file in the current snapshot.

An example of small files in a single data partition

Small files can often be generated as the result of a streaming process. e.g. If the rate of data received into an application is sub-optimal compared with how frequently the application writes out to storage. It can also be the result of incremental updates into a table partition.

Asides from memory strain, small files also present a major performance hit for read processing as the consumer process will need to spend additional handles for open/closing of many more files than is optimal for reading.

To handle this , it is good practice to run a compaction job on directories that contain many small files to ensure storage blocks are filled efficiently. It is common to do this type of compaction with MapReduce or on Hive tables / partitions and we will walk through a simple example of remediating this issue using Spark.

It is also helpful to not overly partition your data. Shallow and wide is a better strategy for storage of compacted files rather than deep and narrow.
Big Data Pipelines as Serverless Microservice’s
By: Joel Lutman
Tags: AWS, Aws Step Functions, Apache Spark, Serverless Architecture, Serverless

Apache Spark, Serverless, and Microservice’s are phrases you rarely hear spoken about together, but that’s all about to change…

As someone who works as a SME in Apache Spark it’s been common for me to be working with large Hadoop clusters (either on premise or as part of an EMR cluster on AWS), which run up large bills even though the clusters are mostly idle, seeing short periods of intense compute when pipelines run. On top of the monetary cost of running this, the added burden of keeping these clusters healthy eats into time and energy that should be spent on adding value to the data.

In contrast we have the Serverless movement, which aims to abstract away many of these issues with managed services, where you only pay for what you use. Examples of how Serverless 1.0 has materialised is with managed compute services such as AWS Lambda and AWS Glue, services where you simply submit small scripts and define an execution runtime, paying

only for the execution time and memory requirement; and managed storage services such as AWS DynamoDB and AWS S3, services where you read and write data via an API, paying only for the storage time and I/O throughput.

If we can bring these two concepts together we can leverage the power of Apache Spark to deliver value via big data solutions without the overheard of large Hadoop clusters. In doing so we can split stages into small micro-services that can easily scale independently regardless of cluster limitations, and orchestrate entire workflows (beyond apache spark) to make use of parallel execution.
Partitioned Delta Lake : Part 3. A tutorial about how to use partition…
Tags: Delta Lake, Spark, Programming, Scala, Apache Spark

Welcome to third part of this series on how to use partition in Delta Lake efficiently.

The previous posts can be found here:

What is Partition?

A big problem can be solved easily when it is chopped into several smaller sub-problems. That is what the partitioning technique does. It divides a big database containing data metrics and indexes into smaller and handy slices of data called as partitions. The partitioned tables are directly used by SQL queries without any alteration. Once the database is partitioned, the data definition language can easily work on the smaller partitioned slices, instead of handling the giant database altogether. This is how partitioning cuts down the problems in managing large database tables.

The partitioning key consists of a single or supplementary columns with the intention of determining the partition wherever the rows will be stored. Spark modifies the partitions by using these partition keys.

Partitioning Techniques

Partitioning Techniques

Create Partitioned table:

Method to create Partitioned table

Read Partitioned table:

Method to read Partitioned Table

Creating and Reading table from Delta lake

I have used sales_test.csv file for my demo. Creating table with partition column as date and reading table using above method.

When we can see storage location of table it is partitioned with date. You can see separate folder is created for particular date.

Updating Partitioned table:

I am going to modify no of units for specific date. After I am updating partitioned table. delta lake will update only data that matches predicates over partition columns.

Best Practices in Partitioned table:

Choose the right partition column:

You can partition a Delta table by a column. The most commonly used partition column is date . Follow these two rules of thumb for deciding on what column to partition by:

If the cardinality of a column will be very high, do not use that column for partitioning. For example, if you partition by a column userId and if there can be 1M distinct user IDs, then that is a bad partitioning strategy.

and if there can be 1M distinct user IDs, then that is a bad partitioning strategy. Amount of data in each partition: You can partition by a column if you expect data in that partition to be at least 1 GB.

In Upcoming series , I will explain about upsert and insert in Delta Lake.

Thanks for reading!!!!

See you soon :)
How to use Broadcast variable in UDF in pyspark
By: Diptiman Chakrabarti
Tags: Variables, Apache Spark, Pyspark, Python3

How to use Broadcast variable in UDF in pyspark

Apache spark is now used as ETL on big data hadoop platform or even on cloud with different essence of it. Complex joins across multiple files/tables and transformation are now part and parcel of any Apache spark script. With wake of complex implementations, performance tuning on spark has also become need of hour.

Complex joins with multiple tables in spark causes lots of shuffling. And higher shuffling impacts performance and sometime causes memory issues too. Issues with hash memory, serializable memories are very common.

User defined function(UDF) with Broadcast Variable on apache spark can help to reduce joins and thus improves performances.

What is Broadcast Variable and UDF

Broadcast variable is a global variable which is broadcasted across all clustered and when ever required can be referred by the transformation and actions in apache spark. Broadcast variables are created out of RDD.

User Defined Function(UDF) in Apache Spark facilitates special functionalities which are not possible with Spark built-in functions, and are specific to work requirement.

In Apache Spark if any Spark serialized data access is required by User defined function(UDF), that can only be done either with Broadcast variable or by Accumulator. Broadcast variable can take key-value pair which accumulator can’t. So Broadcast variable keys can be used as filter column in UDF and required value from broadcast variable can be returned via UDF. Filter is a transformation and does not involve shuffling.

In this topic I am trying to describe how a broadcast variable with multiple keys can be used in user defined function(UDF) in pyspark.
Cache Patterns with Apache Spark
By: Scott Haines
Tags: Big Data Analytics, Big Data, Optimization, Apache Spark, Data Engineering

Spark Storage

The notion of the StorageLevel comes up when you are caching off the lineage of your DataFrame.

val df = spark.read

.schema(Encoders.product[SomeCaseClass].schema)

.option("basePath", "hdfs://path/to/data/lake/root/table")

.parquet("hdfs://path/to/data/lake/root/table/year=2020/month=2")

.transform(expensiveTransformation)

.join(otherDataFrame, df("key").equalTo(other("uuid")))

Above we have an example of loading, transforming and joining data with an additional data frame. Given that this operation is lazy, in that nothing will be computed until you provide an action, we can then cheat and cache this lineage by calling head on our DataFrame to push Spark into action.

import org.apache.spark.storage._

df.persist(StorageLevel.MEMORY_ONLY_SER)

df.head // computes the expensive operations and caches

df.storageLevel // will show you how things are stored.

Once you have cached your computations, in this case calling persist with the option of StorageLevel.MEMORY_ONLY_SER you can proceed with as many operations against the results of your initial computations without having to restart each action at the source or your initial read command.

StorageLevel has options which range from DISK to OFF_HEAP to NONE. I suggest taking a look at the options there and testing different options that help to speed up your Spark Applications.

When you are all done with your cached data, you can remove it from Spark really easily by calling unpersist. * tap with wand and say mischief managed — for the Harry Potter fans out there.

df.unpersist

This will remove the cache either immediately or you can also use the blocking option.

df.unpersist(blocking=true)

Then you will free up your resources for the rest of your necessary computations.

The Cross Product of Big Data and Big Data is Out of Memory Exceptions — Holden Karau

Conclusions

The approach I take to caching and persistence with Spark is to treat it more like the JVM garbage collector. If I need to access the same data more than once then I store it in cache. This allows me to reduce round-trips to my source of truth for the data which is either s3a:// or hdfs://. Given the type of data I work with, which is mainly parquet and other FileSystem based data, is heavy on the IO and the total loaded files side. Once I have my data loaded it is often better to cache things off before even making another decision so I don’t inadvertently make needless trips back to hdfs or s3 to load the same data again.

About the Author

I work on the Voice Insights team at Twilio where we are ingesting millions of events every few seconds across a slew of Spark Streaming applications all running within our data lineage pipeline. We are a hybrid Data Platform / Data Engineering and Data Sciences team working in California.

See my other post on Streaming Trend Discovery to see how we handled some of the more complex use cases with Spark.
Data Sharing between multiple Spark Jobs in Databricks
By: Karthikeyan Siva Baskaran
Tags: Global Temp View, Spark, Data Sharing, Temp View, Apache Spark

Data Sharing between multiple Spark Jobs in Databricks

There are multiple use cases where you want to share the data across multiple Spark Jobs when the data is not too huge. Using createOrReplaceGlobalTempView , the data can be shared between spark jobs instead of storing intermediate data persistently in disk and later cleaning those intermediate data.

createOrReplaceTempView or createOrReplaceGlobalTempView creates a lazily evaluated “view” from the dataframe that you can then use like a hive table in Spark SQL. But, it does not persist into memory unless you cache the data set.

The temp view created by these both methods will create memory reference to the dataframe in use. It will create a temporary view of the table in memory, it is not persistent at this moment but you can run SQL query on top of that.

The lifetime of temp view created by createOrReplaceTempView() is tied to Spark Session in which the dataframe has been created.

The lifetime of global temp view created by createGlobalTempView() is tied to Spark Application . So, this memory reference can be used across Spark Sessions. it will be automatically dropped when the application terminates. It’s tied to a system preserved database global_temp(configurable via SparkConf), and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.my_view

In Databricks, you can share the data using this global temp view between different notebook when each notebook have its own Spark Session. If each notebook shares the same spark session, then using normal temp view also you can share the data across notebooks, but due to some security reasons, this option is set to false by default. But you can turn it on, based on your requirements.

Databricks command to view the configuration value

Spark Session Isolation is enabled by default. With Spark Session Isolation, different notebooks attached to a cluster are in different sessions with isolated runtime configurations and current database setting. In order to share temporary views across notebooks when session isolation is enabled, users can use global temporary views. Users can still disable session isolation by setting spark.databricks.session.share to true . If you enable this option, createOrReplaceTempView itself shares the data between different spark sessions(different notebooks). From Spark 2.0.2-db1 and above versions due to some security reasons and for user stability, session isolation is disabled by default. You can enable Spark session isolation so that every notebook uses its own SparkSession

To disable session isolation, declare it on cluster level and then restart the cluster. But, as a good practice session isolation shouldn’t be disabled.
Slowly Changing Dimensions (SCD) Type 2 and effective ways of handling it in Apache Spark
By: Jafeer Ali
Tags: Bigdata, Apachespark, Hive, Scdtype2, Data Science

Slowly Changing Dimensions (SCD) Type 2 and effective ways of handling it in Apache Spark Jafeer Ali May 8, 2019·3 min read

What is SCD type 2?

As per oracle documentation, “A Type 2 SCD retains the full history of values. When the value of a chosen attribute changes, the current record is closed. A new record is created with the changed data values and this new record becomes the current record. Each record contains the effective time and expiration time to identify the time period between which the record was active.”

Lets Start with example, Our current Hive/delta table looks like below

current table

New CSV with effective from June 2019 looks like below

new csv -june 2019

And current table should be updated as follow,

Explanation:

Employee 1 remains unchanged. So no change to Effective From and Effective Upto .

and . Employee 2 has been changed. So Effective Upto is updated to May 2019 for the old record and a new record is created with Effective From June 2019.

is updated to May 2019 for the old record and a new record is created with June 2019. Employee 3 has no record in new CSV file. So Effective Upto is updated to May 2019.

is updated to May 2019. Employee 4 is a new record in new CSV file. So new record is created with Effective From June 2019.

Implementation steps in Spark:

Assumptions:

Current data frame — it is the current dataframe which reads data from Hive/delta.

— it is the current dataframe which reads data from Hive/delta. New data frame — it is new data which is going to be updated to the current dataframe.

Record Hash Computation:

Concatenate all columns and apply hashing for New Data frame.

Let us assume record hash already exists in Current dataframe.

Meta information about table row like effective from, effective upto excluded as part of record hash computation. [ dropColsInHash ]

Steps:

From above example, we can clearly see we need to handle four scenarios.

1) Unchanged

2) New

3) Delete

4) Update

Unchanged

Inner join both dataframe with record hash as join column

2) New

Left anti join of new dataframe and current dataframe where effective up to is null.

join of new dataframe and current dataframe where effective up to is null. Effective from — new date

Effective upto — Null

3) Delete

Left anti join of current dataframe and new dataframe.

join of current dataframe and new dataframe. Effective from — No change

Effective upto — previous month of new date.

4) Update

Old record is handled as part of Delete.

New Record is handled as part of Insert.

Final Result is derived by union of unchanged, insert and delete and overwrite the Hive/delta table with result.

That’s it. Refer my github project for complete implementation.

Advantages of this method
How to UPSERT data into relational database using Apache Spark: Part 2
By: Thomas Thomas
Tags: Big Data Analytics, Big Data, Programming, Scala, Apache Spark

In my opinion, Database UPSERT won’t be complete with out talking about MERGE functionality. In this blog we will sail through how we can UPSERT using MERGE command.

SELECT from a database table and then INSERT or UPDATE based on the condition are the traditional way of handling data UPSERT. With the MERGE statement, you can combine those statements into a single one and improve overall performance of the individual transaction.

Most of the relational databases support MERGE command. In Part 1 example, I’ve used MySQL. But MERGE is not supported by MySQL, However, there is a different way of achieving the same in MySQL:

INSERT…ON DUPLICATE KEY UPDATE

If you specify the ON DUPLICATE KEY UPDATE option in the INSERT statement and the new row causes a duplicate value in the UNIQUE or PRIMARY KEY index, MySQL performs an update to the old row based on the new values.

In this example I’m going to use DB2 database and MERGE statement. Following is a syntax diagram for the MERGE statement showing all of its principal clauses:

In its simplest form the MERGE statement looks like this:

MERGE INTO <table-name>

USING <table-reference>

AS <correlation-name>

ON <search-condition>

WHEN NOT MATCHED THEN <insertSQL> THEN

WHEN MATCHED THEN <updateSQL>

Let’s re-write the Step 4 in Part 1 example using MERGE command.

4. Update database table records using Spark.

For our example let’s say the school city “Garden City” is changed to “NewYork” and “University at Albany, State University of New York” is changed to “Albany State University”.

Lets see how this is updated into school table.

Step 4.1: Get the data from CSV file to update the school table

Step 4.2: Setup JDBC connection parameters. Please note that I’ve modified the connection parameter to suite DB2 database.

Step 4.3: Broadcast JDBC connection parameters to cluster nodes.

Step 4.4: Browse through each partitioned data and establish the JDBC Connection for each partition and execute the MERGE statement.

In the above example Coalesce is used. You can use repartition also.

On a side note, here is the difference between coalesce and repartition

Step 4.5: Execute MERGE statement using the column values and Add the all the dataframe records to a batch and execute the batch.

I have benchmarked both MERGE statement( without optimizing the environment and Query) and SELECT/INSERT/UPDATE statements in a development cluster and MERGE statement took 3 minutes 40 secs to load 100000 records, whereas SELECT/UPDATE/INSERT took 8 minutes 6 seconds. In all my executions MERGE statement performed far superior than the traditional SELECT/and UPDATE/INSERT.
3 Reasons Developers Like Apache Spark for Data Engineering
By: Kartik Patel
Tags: Spark, Apache, Spark Interface, Apache Spark, Developer

When your business decides to develop a software application or a software product, its IT team or IT consulting partner must choose an appropriate development environment to support the needs of the project and plan for scalability, performance and upgrades. Many developers like the Apache Spark tools and libraries and supporting environment and there is good reason to choose Spark for your project.

‘These are just a few of the developer considerations that make Spark a good choice.’

Spark is a distributed open-source cluster-computing framework and includes an interface for programming a full suite of clusters with comprehensive fault tolerance and support for data parallelism. This open access and flexibility makes Spark a common choice for developers.

In this article, we will discuss three reasons your developer team or IT consulting partner may wish to choose Apache Spark for data engineering in software application or software product application development.

Here are three reasons developers like to use Apache Spark:

Flexibility and Scalability–Spark is not a restrictive environment. It offers support for Cloud apps, Kubermetes, Apache Mesos and Hadoop and can easily handle disparate data. Developers can leverage Spark in a standalone mode and it supports hundreds of types of data sources including Apache Hive, Apache Cassandra, Apache HBase, HDFS etc.

Libraries and Tools– With more thaneighty high-level operators and support for parallel applications, Spark offers many tools and features to support developers. Spark accommodates familiar application languages, and developers can work in SQL, R, Python, Scala and Java. Spark supports streaming, analytics and SQL and has libraries for machine learning, streaming, data frames and graphics.

Programming APIs — Developers have lots of choices for APIs including Python, Scala and Java and for R programmers, SparkR allows access to Spark data directly from the R environment. For developers working in analytics, the Python and R access is crucial as these are typically used in analytical and data science solutions.

These are just a few of the developer considerations that make Spark a good choice. In addition Apache Spark popularity is driven by developer community support. With so manycontributing developers and world-wide use of the features and tools, the Spark libraries and functionality are growing every day.

‘We discuss three reasons your developer team or IT consulting partner may wish to choose Apache Spark for data engineering in software applications and software product development’
How to Display a PySpark DataFrame in Table Format
By: Giorgos Myrianthous
Tags: Spark, Programming, Apache Spark, Python, Data Science

How to Display a PySpark DataFrame in Table Format

Photo by Mika Baumeister on unsplash.com

In the big data era, it is quite common to have dataframes that consist of hundreds or even thousands of columns. And in such cases, even printing them out can sometimes be tricky as you somehow need to ensure that the data is presented in a clear but also efficient way.

In this article, I am going to explore the three basic ways one can follow in order to display a PySpark dataframe in a table format. For each case, I am also going to discuss when to use or avoid it, depending on the shape of data you have to deal with.

Print a PySpark DataFrame

To get started, let’s consider the minimal pyspark dataframe below as an example:

spark_df = sqlContext.createDataFrame(

[

(1, "Mark", "Brown"),

(2, "Tom", "Anderson"),

(3, "Joshua", "Peterson")

],

('id', 'firstName', 'lastName')

)

The most obvious way one can use in order to print a PySpark dataframe is the show() method:

>>> df.show()

+---+---------+--------+

| id|firstName|lastName|

+---+---------+--------+

| 1| Mark| Brown|

| 2| Tom|Anderson|

| 3| Joshua|Peterson|

+---+---------+--------+
Apache Spark - Spark SQL ve Spark Streaming Bölüm 1
By: Soner Ömer Sarıuşak
Tags: Oracle Database, Spark Streaming, Spark Sql, Java, Apache Spark

SEPAL_LENGTH, SEPAL_WIDTH alanlarının avg fonksiyonu ile ortalamalarını alalım..

Dataset<Row> results = dataFrame.groupBy("VARIETY").agg(org.apache.spark.sql.functions.avg("SEPAL_LENGTH"),org.apache.spark.sql.functions.avg("SEPAL_WIDTH")); results.show();

Spark SQL’de SQL fonksiyonları haricinde, temel SQL yapısı gibi sorgular da yazmamıza izin veriliyor. Bunun için StructType ile bir şema oluşturuyoruz. Daha sonrasında ise elimizdeki veri ile şemayı birleştirip DataFrame oluşturuyoruz.

StructType schema = new StructType(new StructField[]{ new StructField("SepalLength", DataTypes.createDecimalType(), true, Metadata.empty()), new StructField("SepalWidth", DataTypes.createDecimalType(), false, Metadata.empty()), new StructField("PetalLength", DataTypes.createDecimalType(), true, Metadata.empty()), new StructField("PetalWidth", DataTypes.createDecimalType(),false, Metadata.empty()), new StructField("Variety", DataTypes.StringType,false, Metadata.empty())}); Dataset<Row> df = spark.createDataFrame(dataFrame.toJavaRDD(), schema); df.createOrReplaceTempView("IRIS"); // Use query in iris schema dataFrame.sparkSession().sql("SELECT PetalLength from IRIS WHERE PetalLength>3 ORDER BY PetalLength ASC").show();

Console Görünümü

Sorgu yapıları için bir kaç örnek göstermek istedim. Apache Spark SQL ile bunun gibi bir çok sorgu yazabilirsiniz..