8 Ways to Transform Pandas Dataframes | by Soner Yıldırım | Towards Data Science

Pandas is a data analysis and manipulation library for Python. The core data structure of Pandas is dataframe which stores data in tabular form with labelled rows and columns.
Pandas provides a variety of functions to modify or manipulate its core structure. In this article, we will go over 8 different ways of transforming dataframes.
We start by importing the dependencies.
import numpy as npimport pandas as pd
We will do the first a few examples using the following sample dataframe.
The first and foremost way of transformation is adding or dropping columns. A new column can be added as follows:
df['new'] = np.random.random(5)
We give the values as an array or list and assign a name to the new column. Make sure the size of the array is compatible with the size of the dataframe.
The drop function is used to drop a column.
df.drop('new', axis=1, inplace=True)
We pass the name of the column to be dropped. The axis parameter is set to 1 to indicate we are dropping a column. Finally, the inplace parameter needs to be True to save the changes.
We can use the loc method to add a single row to a dataframe.
df.loc[5,:] = ['Jack', 3, 3, 4, 5, 1]
The loc function specifies rows and columns with their labels. The [5, :] expression indicates row with label 5 and all columns.
The drop function with axis parameter set to zero can be used to drop a row.
df.drop(5, axis=0, inplace=True)
We have just dropped the row that was added in the previous step.
The insert function adds a column into a specific position.
df.insert(0, 'new', np.random.random(5))
The first parameter specifies the column index. The second and third parameters are the name of the column and the values, respectively.
The melt function converts a dataframe from wide (high number of columns) to narrow form (high number of rows). It is best explained via an example. Consider following dataframe.
It contains consecutive daily measurements for 3 people. The long format of this dataframe can be achieved using the melt function.
pd.melt(df, id_vars='names').head()
The column passed to the id_vars parameter remains the same and the other columns are combined under the variable and value columns.
The concat function combines multiple dataframes along an axis (index or columns). The shapes of the dataframes must be compatible with respect to the axis.
Consider the following two dataframes.
Here is how we can combine them:
pd.concat([df, df2], axis=0, ignore_index=True)
We set axis parameter as zero to combine along the index. The ignore_index parameter resets the index. Otherwise, the original indices are kept (0,1,2,0,1,2).
To combine along columns, the axis parameter is set to 1.
Merge function also combines dataframes based on common values in a given column or columns. Consider the following two dataframes.
We can merge them based on the id column.
customer.merge(order, on='id')
By default, only the values that exist in both dataframes are taken. We can change it with the “how” parameter. For instance, if we set is as “left”, all the values from the left dataframe are taken and only the matching ones are taken from the right dataframe.
Some machine learning models cannot handle categorical variables. In such cases, we should encode the categorical variables in a way that each category is represented as a column.
Assume we have the following dataframe.
The get_dummies function encodes the name and ctg columns.
pd.get_dummies(df)
For instance, in the first row, the name is Jane and the ctg is A. Thus, the columns that represent these values are 1 and all other columns are 0.
The pivot_table function transforms a dataframe to a format that explains the relationship among variables.
We have the dataframe on the left that contains two categorical features (i.e. columns) and a numerical feature.
We want to see the average value of the categories in both columns. The pivot_table function transforms the dataframe in a way that the average values or any other aggregation can be seen clearly.
df.pivot_table(index='name', columns='ctg', aggfunc='mean')
We have covered 8 different ways to transform a dataframe. Some of them just change the structure whereas some performs a calculation during the transformation.
We have also seen how to add or drop rows and columns. Furthermore, the merge and concat functions transform dataframes by means of combining them. All these operations are important parts of data analysis and manipulation process.
Thank you for reading. Please let me know if you have any feedback.