How to find the sum of Particular Column in PySpark Dataframe

29 Jun, 2021
In this article, we are going to find the sum of PySpark dataframe column in Python. We are going to find the sum in a column using agg() function. 
Let’s create a sample dataframe.
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "sravan", "vignan", 67, 89],        ["2", "ojaswi", "vvit", 78, 89],        ["3", "rohith", "vvit", 100, 80],        ["4", "sridevi", "vignan", 78, 80],        ["1", "sravan", "vignan", 89, 98],        ["5", "gnanesh", "iit", 94, 98]]  # specify column namescolumns = ['student ID', 'student NAME', 'college',           'subject 1', 'subject 2']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)  # display dataframedataframe.show()
Output:
The agg() method returns the aggregate sum of the passed parameter column.
 Syntax:
dataframe.agg({'column_name': 'sum'})
Where,
The dataframe is the input dataframeThe column_name is the column in the dataframeThe sum is the function to return the sum.
The dataframe is the input dataframe
The column_name is the column in the dataframe
The sum is the function to return the sum.
Example 1: Python program to find the sum in dataframe column
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "sravan", "vignan", 67, 89],        ["2", "ojaswi", "vvit", 78, 89],        ["3", "rohith", "vvit", 100, 80],        ["4", "sridevi", "vignan", 78, 80],        ["1", "sravan", "vignan", 89, 98],        ["5", "gnanesh", "iit", 94, 98]]  # specify column namescolumns = ['student ID', 'student NAME', 'college',           'subject 1', 'subject 2']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)    # find sum of subjects columndataframe.agg({'subject 1': 'sum'}).show()
Output:
Example 2: Get sum value from multiple columns
Python3
# importing moduleimport pyspark  # importing sparksession from pyspark.sql modulefrom pyspark.sql import SparkSession  # creating sparksession and giving an app namespark = SparkSession.builder.appName('sparkdf').getOrCreate()  # list  of students  datadata = [["1", "sravan", "vignan", 67, 89],        ["2", "ojaswi", "vvit", 78, 89],        ["3", "rohith", "vvit", 100, 80],        ["4", "sridevi", "vignan", 78, 80],        ["1", "sravan", "vignan", 89, 98],        ["5", "gnanesh", "iit", 94, 98]]  # specify column namescolumns = ['student ID', 'student NAME', 'college',           'subject 1', 'subject 2']  # creating a dataframe from the lists of datadataframe = spark.createDataFrame(data, columns)    # find sum of multiple  columndataframe.agg({'subject 1': 'sum', 'student ID': 'sum',               'subject 2': 'sum'}).show()
Output:
Python-Pyspark
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.