Introduction to PySpark | Distributed Computing with Apache Spark - GeeksforGeeks

29 Apr, 2022
Datasets are becoming huge. Infact, data is growing faster than processing speeds. Therefore, algorithms involving large data and high amount of computation are often run on a distributed computing system. A distributed computing system involves nodes (networked computers) that run processes in parallel and communicate (if, necessary).
MapReduce – The programming model that is used for Distributed computing is known as MapReduce. The MapReduce model involves two stages, Map and Reduce.
Map – The mapper processes each line of the input data (it is in the form of a file), and produces key – value pairs.Input data → Mapper → list([key, value])Reduce – The reducer processes the list of key – value pairs (after the Mapper’s function). It outputs a new set of key – value pairs.list([key, value]) → Reducer → list([key, list(values)])
Map – The mapper processes each line of the input data (it is in the form of a file), and produces key – value pairs.Input data → Mapper → list([key, value])
Input data → Mapper → list([key, value])
Reduce – The reducer processes the list of key – value pairs (after the Mapper’s function). It outputs a new set of key – value pairs.list([key, value]) → Reducer → list([key, list(values)])
list([key, value]) → Reducer → list([key, list(values)])
Spark – Spark (open source Big-Data processing engine by Apache) is a cluster computing system. It is faster as compared to other cluster computing systems (such as, Hadoop). It provides high level APIs in Python, Scala, and Java. Parallel jobs are easy to write in Spark. We will cover PySpark (Python + Apache Spark), because this will make the learning curve flatter. To install Spark on a linux system, follow this. To run Spark in a multi – cluster system, follow this. We will see how to create RDDs (fundamental data structure of Spark).
RDDs (Resilient Distributed Datasets) – RDDs are immutable collection of objects. Since we are using PySpark, these objects can be of multiple types. These will become more clear further.
SparkContext – For creating a standalone application in Spark, we first define a SparkContext –
from pyspark import SparkConf, SparkContextconf = SparkConf().setMaster("local").setAppName("Test")# setMaster(local) - we are doing tasks on a single machinesc = SparkContext(conf = conf)
RDD transformations – Now, a SparkContext object is created. Now, we will create RDDs and see some transformations on them.
# create an RDD called lines from ‘file_name.txt’lines = sc.textFile("file_name.txt", 2)  # print lines.collect() prints the whole RDDprint lines.collect()
One major advantage of using Spark is that it does not load the dataset into memory, lines is a pointer to the ‘file_name.txt’ ?file.
A simple PySpark app to count the degree of each vertex for a given graph –
from pyspark import SparkConf, SparkContextconf = SparkConf().setMaster("local").setAppName("Test")# setMaster(local) - we are doing tasks on a single machinesc = SparkContext(conf = conf)def conv(line):    line = line.split()    return (int(line[0]), [int(line[1])])def numNeighbours(x, y):    return len(x) + len(y)lines = sc.textFile('graph.txt')edges = lines.map(lambda line: conv(line))Adj_list = edges.reduceByKey(lambda x, y: numNeighbours(x, y))print Adj_list.collect()
Understanding the above code –
Our text file is in the following format – (each line represents an edge of a directed graph)1    21    32    33    4.    ..    ..    .PySparkLarge Datasets may contain millions of nodes, and edges.First few lines set up the SparkContext. We create an RDD lines from it.Then, we transform the lines RDD to edges RDD.The function conv acts on each line and key value pairs of the form (1, 2), (1, 3), (2, 3), (3, 4), ... are stored in the edges RDD.After this the reduceByKey aggregates all the key – pairs corresponding to a particular key and numNeighbours function is used for generating each vertex’s degree in a separate RDD Adj_list, which has the form (1, 2), (2, 1), (3, 1), ...
Our text file is in the following format – (each line represents an edge of a directed graph)1    21    32    33    4.    ..    ..    .PySpark
Large Datasets may contain millions of nodes, and edges.
First few lines set up the SparkContext. We create an RDD lines from it.
Then, we transform the lines RDD to edges RDD.The function conv acts on each line and key value pairs of the form (1, 2), (1, 3), (2, 3), (3, 4), ... are stored in the edges RDD.
After this the reduceByKey aggregates all the key – pairs corresponding to a particular key and numNeighbours function is used for generating each vertex’s degree in a separate RDD Adj_list, which has the form (1, 2), (2, 1), (3, 1), ...
Running the code –
The above code can be run by the following commands –$ cd /home/arik/Downloads/spark-1.6.0/
$ ./bin/spark-submit degree.py
You can use your Spark installation path in the first line.
The above code can be run by the following commands –$ cd /home/arik/Downloads/spark-1.6.0/
$ ./bin/spark-submit degree.py

$ cd /home/arik/Downloads/spark-1.6.0/
$ ./bin/spark-submit degree.py

You can use your Spark installation path in the first line.
We will see more on, how to run MapReduce tasks in a cluster of machines using Spark, and also go through other MapReduce tasks.
References –
http://lintool.github.io/SparkTutorial/https://spark.apache.org/
http://lintool.github.io/SparkTutorial/
https://spark.apache.org/
This article is contributed by Arik Pamnani. If you like GeeksforGeeks and would like to contribute, you can also write an article using write.geeksforgeeks.org or mail your article to review-team@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.
simmytarika5
Web Technologies
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Remove elements from a JavaScript Array
Convert a string to an integer in JavaScript
How to fetch data from an API in ReactJS ?
Difference between var, let and const keywords in JavaScript
Differences between Functional Components and Class Components in React
How to apply style to parent if it has child with CSS?
How to execute PHP code using command line ?
Difference Between PUT and PATCH Request
REST API (Introduction)
How to redirect to another page in ReactJS ?