Create a Pipeline in Pandas - GeeksforGeeks

17 Jan, 2022
Pipelines play a useful role in transforming and manipulating tons of data. Pipeline are a sequence of data processing mechanisms. Pandas pipeline feature allows us to string together various user-defined Python functions in order to build a pipeline of data processing. There are two ways to create a Pipeline in pandas. By calling .pipe() function and by importing pdpipe package. 
Through pandas pipeline function i.e. pipe() function we can call more than one function at a time and in a single line for data processing. Let’s understand and create a pipeline by using the pipe() function.
Below are various examples that depict how to create a pipeline using pandas.
Example 1:
Python3
# importing pandas libraryimport pandas as pd # Create empty dataframedf = pd.DataFrame() # Creating a simple dataframedf['name'] = ['Reema', 'Shyam', 'Jai',              'Nimisha', 'Rohit', 'Riya']df['gender'] = ['Female', 'Male', 'Male',                'Female', 'Male', 'Female']df['age'] = [31, 32, 19, 23, 28, 33] # View dataframedf
Output:
Now, creating functions for data processing.
Python3
# function to find meandef mean_age_by_group(dataframe, col):       # groups the data by a column and    # returns the mean age per group    return dataframe.groupby(col).mean()   # function to convert to uppercasedef uppercase_column_name(dataframe):       # Converts all the column names into uppercase    dataframe.columns = dataframe.columns.str.upper()         # And returns them    return dataframe 
Now, creating a pipeline using .pipe() function.
Python3
# Create a pipeline that applies both the functions created abovepipeline = df.pipe(mean_age_by_group, col='gender').pipe(uppercase_column_name) # calling pipelinepipeline
Output:
Now, let’s understand and create a pipeline by importing pdpipe package.
The pdpipe Python package provides a concise interface for building pandas pipelines that have pre-conditions. The pdpipe is a pre-processing pipeline package for Python’s panda data frame. The pdpipe API helps to easily break down or compose complex-ed panda processing pipelines with few lines of codes. 
We can install this package by simply writing:
pip install pdpipe
Example 2:
Python3
# importing the packageimport pdpipe as pdpimport pandas as pd # creating a empty dataframe named datasetdataset = pd.DataFrame() # Creating a simple dataframedataset['name'] = ['Reema', 'Shyam', 'Jai',                   'Nimisha', 'Rohit', 'Riya'] dataset['gender'] = ['Female', 'Male', 'Male',                     'Female', 'Male', 'Female'] dataset['age'] = [31, 32, 19, 23, 28, 33] dataset['department'] = ['Accounts', 'Management',                         'IT', 'IT', 'Management',                         'Advertising'] dataset['index'] = [1, 2, 3, 4, 5, 6] # View dataframedataset
Output:
Removing a column from dataframe using pdpipe.
Python3
# creating a pipeline and# dropping the unwanted columndropCol = pdp.ColDrop("index").apply(dataset) # display the new dataframe# after column dropdropCol
Output:
There is another way to drop columns through pdpipe.
Python3
# creating a pipeline and# dropping the unwanted columndropCol2 = pdp.ColDrop("index") # applying the ColDrop to dataframedf2 = dropCol2(dataset) # display dataframedf2
Output:
Here, the column is dropped in two steps. In the first step, we created a pipeline and in the second step, we applied it to the dataframe.
Now we are adding one column to dataframe using pdpipe.
Python3
# importing the packageimport pdpipe as pdpimport pandas as pd # creating a empty dataframe named datasetdataset = pd.DataFrame() # Creating a simple dataframedataset['name'] = ['Reema', 'Shyam', 'Jai',                   'Nimisha', 'Rohit', 'Riya'] dataset['gender'] = ['Female', 'Male', 'Male',                     'Female', 'Male', 'Female'] dataset['age'] = [31, 32, 19, 23, 28, 33] dataset['department'] = ['Accounts', 'Management',                         'IT', 'IT', 'Management',                         'Advertising'] dataset['index'] = [1, 2, 3, 4, 5, 6] # View dataframedataset
Output:
Now, dropping the values from dataframe.
Python3
#dropping the values using ValDropdf3 = pdp.ValDrop(['IT'],'department').apply(dataset) #display dataframedf3
  
Output:
 
 
The row containing ‘ IT ‘ value is dropped.
 
abhishek0719kadiyan
prachisoda1234
rajeev0719singh
rs1686740
saurabh1990aror
Picked
Python-pandas
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
How to Install PIP on Windows ?
How to drop one or multiple columns in Pandas Dataframe
Python OOPs Concepts
Python | Get unique values from a list
Check if element exists in list in Python
Python Classes and Objects
Python | os.path.join() method
How To Convert Python Dictionary To JSON?
Python | Pandas dataframe.groupby()
Create a directory in Python