7 Ways to Manipulate Pandas Dataframes | by Soner Yıldırım | Towards Data Science

Pandas is a very powerful and versatile Python data analysis library that expedites the data analysis and exploration process. One of the advantages of Pandas is that it provides a variety of functions and methods for data manipulation.
A dataframe is the core data structure of Pandas. In order to master Pandas, you should be able to play around with dataframes easily and smoothly. In this post, we will go over different ways to manipulate or edit them.
Let’s start with importing NumPy and Pandas and creating a sample dataframe.
import numpy as npimport pandas as pdvalues = np.random.randint(10, size=(3,7))df = pd.DataFrame(values, columns=list('ABCDEFG'))df.insert(0, 'category', ['cat1','cat2','cat3'])df
The first way for manipulation we will mention is the melt function which converts wide dataframes (high number of columns) to narrow ones. Some dataframes are structured in a way that consecutive measurements or variables are represented as columns. In some cases, representing these columns as rows may fit better with our task.
#1 meltdf_melted = pd.melt(df, id_vars='category')df_melted.head()
The column specified with the id_vars parameter remains the same and the other columns are represented under the variable and value columns.
The second way is the stack function which increases the index level.
If dataframe has a simple column index, stack returns a series whose indices consist of row-column pairs of original dataframe.
If dataframe has multi-level index, stack increases the index level.
Consider the following dataframe:
#2 stackdf_stacked = df_measurements.stack().to_frame()df_stacked[:6]
The stack function, in this case, returns a Series object but we converted it to a dataframe using the to_frame function.
The unstack function, as the name indicates, reverses the operation of the stack function.
#3 unstackdf_stacked.unstack()
Adding or dropping columns is probably the manipulation we do most. Let’s both add a new column and drop some of the existing ones.
#4 add or drop columnsdf['city'] = ['Rome','Madrid','Houston']df.drop(['E','F','G'], axis=1, inplace=True)df
We created a new column with a list. Pandas Series or NumPy array can also be used to create a column.
To drop columns, in addition to the name of the columns, the axis parameters should be set to 1. The inplace parameter is set to True in order to save the changes.
New columns are added at the end of dataframe by default. If you want the new columns to be placed at a specific location, you should use the insert function.
#5 insertdf.insert(0, 'first_column', [4,2,5])df
We may also want to add or drop rows.
The append function can be used to add new rows.
#6 add or drop rowsnew_row = {'A':4, 'B':2, 'C':5, 'D':4, 'city':'Berlin'}df = df.append(new_row, ignore_index=True)df
We can drop a now just like dropping columns. The only change is the axis parameter value.
df.drop([3], axis=0, inplace=True)df
Another modification on dataframes can be achieved by the pivot_table function. Consider the following dataframe with 30 rows:
import randomA = np.random.randint(10, size=30)B = np.random.randint(10, size=30)city = random.sample(['Rome', 'Houston', 'Berlin']*10, 30)cat = random.sample(['cat1', 'cat2', 'cat3']*10 ,30)df = pd.DataFrame({'A':A, 'B':B, 'city':city, 'cat':cat})df.head()
The pivot_table function can also be considered as a way to look at the dataframe from a different perspective. It is used to explore the relationships among variables by allowing them present data in different formats.
#7 pivot_tabledf.pivot_table(index='cat', columns='city', aggfunc='mean')
The return dataframe contains the mean values for each city-cat pair.
We have covered 7 ways to edit or manipulate a dataframe. Some of them are so common that you are probably using them almost every day. There will also be cases in which you need to use the rare ones.
I think the success and prevalence of Pandas come from the versatile, powerful, and easy-to-use functions to manipulate and analyze data. There are almost always multiple ways to do a task with Pandas. Since a big portion of the time spent on a data science project is spent during data cleaning and preprocessing steps, it is highly encouraged to learn Pandas.
Thank you for reading. Please let me know if you have any feedback.