A Comprehensive Guide to Pandas’ Advanced Features in 20 Minutes | by Fabian Bosler | Towards Data Science

I am going out on a limb here and assume that red pandas are smarter and thus more advanced than their black-and-white brethren. Hence, the cover picture.
In Part I of this Pandas series, we explored the basics of Pandas, including:
How to load data;
How to inspect, sort, and filter data;
How to analyze data using and groupby/transform
If those concepts are new to you, head back to Part I, and get a quick refresher.
towardsdatascience.com
Data Types and ConversionsUseful accessor methods for certain data typesCombining DataFramesReshaping DataFrames
Data Types and Conversions
Useful accessor methods for certain data types
Combining DataFrames
Reshaping DataFrames
A Python environment (I suggest Jupyter Notebook). If you haven’t set this up, don’t worry. It is effortless and takes less than 10 minutes.
towardsdatascience.com
Before doing any data manipulation, let’s get some data. We will be using some fictitious sales data. This GitHub Repo holds the data and the code for this article.
Create a new notebook and import Pandas (import pandas as pd). I tend to adjust my notebook settings a bit like this:
from IPython.core.display import display, HTMLdisplay(HTML("<style>.container {width:90% !important;}</style>"))
These commands make the Notebookwider and thus utilize more space on the screen (typically the notebook has a fixed width, which sucks with wide screens).
invoices = pd.read_csv('https://raw.githubusercontent.com/FBosler/you-datascientist/master/invoices.csv')
Before diving into the data, let’s quickly summarize all the available Pandas data types. In total, there are seven types:
object : This data type is used for strings (i.e., sequences of characters)
int64 : Used for integers (whole numbers, no decimals)
float64 : Used for floating-point numbers (i.e., figures with decimals/fractions)
bool : Used for values that can only be True/False
datetime64 : Used for date and time values
timedelta : Used to represent the difference between datetimes
category : Used for values that take one out of a limited number of available options (categories don’t have to, but can have explicit ordering)
After reading the data and running a quick invoices.sample(5) we observe that the dataset seems to be fairly large, but well structured.
To get a feel for the data, I would usually follow a first sample up with info and describe, but we are here to learn about data types and will skip the typical exploration steps for now. Not surprisingly, there is a command to print out the data types of a DataFrame.
IN:invoices.dtypesOUT:Order Id               objectDate                   objectMeal Id                objectCompany Id             objectDate of Meal           objectParticipants           objectMeal Price            float64Type of Meal           objectSuper Hero Present       booldtype: object
We loaded the data without any type-conversion, so Pandas made its best guess when assigning types. We can see that all columns, but Meal Price and Super Hero Present are of the type object (i.e., string). From the quick inspection we did earlier, it seems like some of the columns could have been assigned a more explicit data type. So let’s change that.
There are two standard ways of converting pandas data types:
<column>.astype(<desired type>)
conversion helper functions, like pd.to_numeric or pd.to_datetime
astype is quick and works well with clean data and when the conversion is straight forward, e.g., from int64 to float64 (or vice versa). astype has to be called directly on the column that you want to convert. Like this:
invoices['Type of Meal'] = invoices['Type of Meal'].astype('category')invoices['Date'] = invoices['Date'].astype('datetime64')invoices['Meal Price'] = invoices['Meal Price'].astype('int')
To validate the effect of our type conversion we just run invoices.dtypes again:
OUT:Order Id                      objectDate                  datetime64[ns]Meal Id                       objectCompany Id                    objectDate of Meal                  objectParticipants                  objectMeal Price                     int64Type of Meal                categorySuper Hero Present              booldtype: object
Note: There is a bit of a difference between Pandas versions. For Pandas version 0.23.x it is possible to convert the Date of Meal column by using .astype('datetime64') and Pandas would then automatically convert into UTC. The UTC format is helpful because it is a standardized time format and allows us to subtract or add dates from other dates.
However, this does not work any longer for Pandas version 0.25.x where we will get a ValueError that lets us know that Tz-aware (timezone-aware) datetimes can not be converted without further adjustment.
There are three pd.to_<some_type> functions, but for me, only two of them come up frequently:
pd.to_numeric()
pd.to_datetime()
pd.to_timedelta() (I can’t remember if I’ve ever used this one, to be honest)
Their main advantage over astype, is that it is possible to specify the behavior in case a value is encountered, that can not be converted. Both functions accept an additional parameter errors that defines how errors should be treated. We could choose to ignore errors by passingerrors='ignore' , or turn the offending values into np.nan values by passing errors='coerce'. The default behavior is to raise errors.
I find that there is no cutty cutter solution, and I would typically investigate before making a decision. The fewer offending values compared to the number of observations you have, the more likely I am to coerce them.
pd.to_numeric()For the sake of argument, let’s mess up our data a bit:
invoices.loc[45612,'Meal Price'] = 'I am causing trouble'invoices.loc[35612,'Meal Price'] = 'Me too'
invoices['Meal Price'].astype(int) will now fail with a ValueError: invalid literal for int() with base 10: ‘Me too.’ Because there is no obvious way to convert the string into an integer. Whenever I encounter unexpected conversion errors, I typically check the values of the column explicitly to get a better understanding of the magnitude of the strange values.
IN:invoices['Meal Price'].apply(lambda x: type(x)).value_counts()OUT:<class 'int'>    49972<class 'str'>        2Name: Meal Price
You could then identify the offending rows by doing this:
IN:invoices['Meal Price'][invoices['Meal Price'].apply(  lambda x: isinstance(x,str))]OUT:35612                  Me too45612    I am causing troubleName: Meal Price, dtype: object
From there on it is a quick step to either fix the values or make an informed decision about how you want to handle failing conversion.
The example above is a prime example, where it would be very reasonable to just convert the values into np.nan by passing errors='coerce' to pd.to_numeric() like this:
pd.to_numeric(invoices['Meal Price'], errors='coerce')
Now it should be noted that you will by construction have two np.nan value in your data, so it might be a good idea to handle them on the spot. The problem with the np.nan`s is that integer columns don’t know how to handle them. Thus the column will be a float column.
# convert the offending values into np.naninvoices['Meal Price'] = pd.to_numeric(invoices['Meal Price'],errors='coerce')# fill np.nan with the median of the datainvoices['Meal Price'] = invoices['Meal Price'].fillna(invoices['Meal Price'].median())# convert the column into integerinvoices['Meal Price'].astype(int)
pd.to_datetime()Does what the name implies, the method converts a string into a datetime format. To call to_datetime on a column you would do: pd.to_datetime(invoices['Date of Meal']). Pandas will then guess the format and try to parse the date from the Input. And it does so impressively well:
IN:print(pd.to_datetime('2019-8-1'))print(pd.to_datetime('2019/8/1'))print(pd.to_datetime('8/1/2019'))print(pd.to_datetime('Aug, 1 2019'))print(pd.to_datetime('Aug - 1 2019'))print(pd.to_datetime('August - 1 2019'))print(pd.to_datetime('2019, August - 1'))print(pd.to_datetime('20190108'))OUT:2019-08-01 00:00:002019-08-01 00:00:002019-08-01 00:00:002019-08-01 00:00:002019-08-01 00:00:002019-08-01 00:00:002019-08-01 00:00:002019-01-08 00:00:00
However, now and then you might encounter some atypical formatting, like the last example from the list above. In this case, you probably want to provide a custom format, which can you can do like this: pd.to_datetime('20190108',format='%Y%d%m'). Think of the format string as a mask to check against the date string, and if the mask fits, the conversion will take place. Check this link out for a list of all the possible date format components. One additional noteworthy parameter when working with custom formats is exact=False. print(pd.to_datetime('yolo 20190108', format='%Y%d%m', exact=False)) will work, while it would fail without the exact parameter. With exact=False Pandas tries to match the pattern anywhere in the date string.
Before moving on let’s convert our Date of Meal column, like this invoices['Date of Meal'] = pd.to_datetime(invoices['Date of Meal'], utc=True)
If your only tool is a hammer, every problem looks like a nail.
Think of a Pandas accessor as a property that acts as an interface to methods specific to the type you are trying to access. Those methods are highly specialized. They serve one job and one job only. However, they are excellent and extremely concise for that particular job.
There are three different accessors:
dt
str
cat
All of the methods are accessed by calling .<accessor>.method on the column of choice, like this: invoices['Date of Meal'].dt.date
This one I think is the most useful and most straight forward of the accessor methods:
date (returns the date of the datetime value), or
IN:invoices['Date of Meal'].dt.dateOUT:0        2015-01-291        2017-10-30            ...    49972    2017-09-0649973    2015-08-20Name: Date of Meal, Length: 49974, dtype: object
weekday_name (returns the name of the day), month_name() (this is implemented somewhat inconsistently, as weekday_name is a property, but month_name is a method and needs to be called with the parenthesis)
IN:invoices['Date of Meal'].dt.weekday_nameOUT:0         Thursday1           Monday           ...    49972    Wednesday49973     ThursdayName: Date of Meal, Length: 49974, dtype: objectIN:invoices['Date of Meal'].dt.month_name()OUT:0          January1          October           ...    49972    September49973       AugustName: Date of Meal, Length: 49974, dtype: object
days_in_month
IN:invoices['Date of Meal'].dt.days_in_monthOUT:0        311        31         ..49972    3049973    31Name: Date of Meal, Length: 49974, dtype: int64
nanosecond,microsecond,second,minute,hour ,day, week, month, quarter, year gets the integer of the corresponding frequency.
EXAMPLES: (same logic applies for nanosecond, microsecond, second, minute, hour)IN:invoices['Date of Meal'].dt.dayOUT:0        291        30         ..49972     649973    20Name: Date of Meal, Length: 49974, dtype: int64IN:invoices['Date of Meal'].dt.weekOUT:0         51        44         ..49972    3649973    34Name: Date of Meal, Length: 49974, dtype: int64IN:invoices['Date of Meal'].dt.monthOUT:0         11        10         ..49972     949973     8Name: Date of Meal, Length: 49974, dtype: int64IN:invoices['Date of Meal'].dt.quarterOUT:0        11        4        ..49972    349973    3Name: Date of Meal, Length: 49974, dtype: int64IN:invoices['Date of Meal'].dt.yearOUT:0        20151        2017         ... 49972    201749973    2015Name: Date of Meal, Length: 49974, dtype: int64
is_leap_year, is_month_start, is_month_end, is_quarter_start, is_quarter_end, is_year_start, is_year_end returns True or False respectively for each value. See the following example for is_month_end
IN:invoices['Date of Meal'].dt.is_month_endOUT:0        False1        False         ...  49972    False49973    FalseName: Date of Meal, Length: 49974, dtype: bool
We can use the results to filter our data down to only rows, where Date of Meal is at the month's end.
to_pydatetime(), which converts the Pandas datetime into a regular Python datetime format (which you might need sometimes) and to_period(<PERIOD>) [available periods are W, M, Q, and Y], which converts the dates into periods.
IN:invoices['Date of Meal'].dt.to_pydatetime()OUT:array([datetime.datetime(2015, 1, 29, 8, 0),       datetime.datetime(2017, 10, 30, 20, 0),       datetime.datetime(2015, 2, 10, 11, 0), ...,       datetime.datetime(2017, 3, 7, 19, 0),       datetime.datetime(2017, 9, 6, 6, 0),       datetime.datetime(2015, 8, 20, 18, 0)], dtype=object)IN:invoices['Date of Meal'].dt.to_period('W')OUT:0       2015-01-26/2015-02-011       2017-10-30/2017-11-05                 ...         49972   2017-09-04/2017-09-1049973   2015-08-17/2015-08-23Name: Date of Meal, Length: 49974, dtype: object
The str accessor is also quite helpful, not because it enables additional functionality, but makes for much more readable code.
lower() / upper() to manage capitalization of strings
IN:invoices['Type of Meal'].str.lower()OUT:0        breakfast1           dinner           ...    49972    breakfast49973       dinnerName: Type of Meal, Length: 49974, dtype: objectIN:invoices['Type of Meal'].str.upper()OUT:0        BREAKFAST1           DINNER           ...    49972    BREAKFAST49973       DINNERName: Type of Meal, Length: 49974, dtype: object
ljust(width), rjust(width), center(width), zfill(width) to control the positioning of strings. All of them take a total width of the desired resulting string as an input. ljust, rjust, and center fill the difference to the desired length with whitespaces.zfill adds that many leading zeroes. ljustis left-bound, rjustis right-bound.
IN:invoices['Type of Meal'].str.ljust(width=15)OUT:0        Breakfast      1        Dinner                       ...       49972    Breakfast      49973    Dinner         Name: Type of Meal, Length: 49974, dtype: objectIN:invoices['Type of Meal'].str.rjust(width=15)OUT:0              Breakfast1                 Dinner              ...       49972          Breakfast49973             DinnerName: Type of Meal, Length: 49974, dtype: objectIN:invoices['Type of Meal'].str.center(width=15)OUT:0           Breakfast   1             Dinner                  ...       49972       Breakfast   49973         Dinner    Name: Type of Meal, Length: 49974, dtype: objectIN:invoices['Type of Meal'].str.zfill(width=15)OUT:0        000000Breakfast1        000000000Dinner              ...       49972    000000Breakfast49973    000000000DinnerName: Type of Meal, Length: 49974, dtype: object
startswith(<substring>), endswith(<substring>), contains(<substring>) checks for the presence of a substring
IN:invoices['Type of Meal'].str.endswith('ast')OUT:0         True1        False         ...  49972     True49973    FalseName: Type of Meal, Length: 49974, dtype: bool
swapcase(), repeat(times) for kicks and giggles
IN:invoices['Type of Meal'].str.swapcase()OUT:0        bREAKFAST1           dINNER           ...    49972    bREAKFAST49973       dINNERName: Type of Meal, Length: 49974, dtype: objectIN:invoices['Type of Meal'].str.repeat(2)OUT:0        BreakfastBreakfast1              DinnerDinner                ...        49972    BreakfastBreakfast49973          DinnerDinnerName: Type of Meal, Length: 49974, dtype: object
In my opinion the least powerful of the three, or at least the one I use the most infrequently. cat provides access to a couple of categorial operations, like:
ordered lets you know if the column is ordered
IN:invoices['Type of Meal'].cat.orderedOUT:False
categories to return the categories
IN:invoices['Type of Meal'].cat.categoriesOUT:Index(['Breakfast', 'Dinner', 'Lunch'], dtype='object')
codes for quick conversion of the category into its numerical representation
IN:invoices['Type of Meal'].cat.codesOUT:0        01        12        23        24        2        ..49969    149970    249971    149972    049973    1Length: 49974, dtype: int8
reorder_categories to change the existing order of the categories
IN:invoices['Type of Meal'].cat.reorder_categories(    ['Lunch','Breakfast','Dinner'])OUT:0        Breakfast1           Dinner2            Lunch3            Lunch4            Lunch           ...    49969       Dinner49970        Lunch49971       Dinner49972    Breakfast49973       DinnerName: Type of Meal, Length: 49974, dtype: categoryCategories (3, object): [Lunch, Breakfast, Dinner]
Concatenating comes in handy when you have similar data (structurally and in terms of content) spread out across multiple files. You can concatenate data vertically (i.e., stack the data on top of each other) or horizontally (i.e., stack the data next to each other).
As an example, imagine that a client provides you one file per month or year of data (because their reporting runs at months end, for example). Those files are conceptually identically and thus a prime example of vertical stacking.
Let’s have a look at how we can do this with Python’s Pandas. Let’s artificially split out invoices file by years.
# Use dt accessor to count all the yearsIN:invoices['Date of Meal'].dt.year.value_counts().sort_index()OUT:2013       382014     99882015    100602016     99612017     99642018     99312019       32Name: Date of Meal, dtype: int64# Split the data y_2013 = invoices[invoices['Date of Meal'].dt.year == 2013].copy()y_2014 = invoices[invoices['Date of Meal'].dt.year == 2014].copy()y_2015 = invoices[invoices['Date of Meal'].dt.year == 2015].copy()y_2016 = invoices[invoices['Date of Meal'].dt.year == 2016].copy()y_2017 = invoices[invoices['Date of Meal'].dt.year == 2017].copy()y_2018 = invoices[invoices['Date of Meal'].dt.year == 2018].copy()y_2019 = invoices[invoices['Date of Meal'].dt.year == 2019].copy()
We use .copy() to make sure that the resulting DataFrames are going to be a copy of the data and not just a reference to the original DataFrame.
Let’s validate that the splitting worked.
Now for concatenating the data into a unified DataFrame, we would call pd.concat(<LIST OF DATAFRAMES>) like this:
pd.concat takes a couple of optional parameters next to the list of DataFrames that you call concat on:
axis : 0 for vertical, 1 for horizontal. axis defaults to 0
join : 'inner' for the intersection, 'outer' for the union of indices of the non-concatenating axis. When we use axis=0 and join='inner' we will consider only overlapping columns. When using axis=1 and join='inner' we consider only overlapping indices. In the case of outer non-overlapping columns/indices will be filled with nan values. join defaults to outer
ignore_index : True to ignore preexisting indices and instead use labels from 0 to n-1 for the resulting DataFrame. ignore_index defaults to False
keys : If we provide a list (has to be the same length as the number of DataFrames) a hierarchical index will be constructed. keys defaults to None. Use keys for example, to add the source of the data. Best used in combination with names.
names : Assuming that you provide keys, the names will be used to label the resulting hierarchical index. names defaults to None.
A use case for horizontal stacking is a case where you have multiple time series with overlapping but not identical indices. In which case, you wouldn’t want to end up with a DataFrame with potentially thousands of columns, but much rather a DataFrame with thousands of rows.
Running the following snippet in Python will result in the following screenshot:
range_a = pd.date_range(    datetime.datetime(2019,1,2),    datetime.datetime(2019,1,8))df_a = pd.DataFrame(    index=range_a,     data=np.random.randint(2,10,size=len(range_a)),    columns=['observations_A'])range_b = pd.date_range(    datetime.datetime(2019,1,5),    datetime.datetime(2019,1,12))df_b = pd.DataFrame(    index=range_b,     data=np.random.randint(2,10,size=len(range_b)),    columns=['observations_B'])pd.concat([df_a,df_b],axis=1)
Note on Append :You might have seen the usage of appendto the same end asconcat. I advise against the usage of appendas it is just a special case of concatand does not provide a benefit over concatenating.
Merging, as opposed to concatenating DataFrames together, allows us to combine two DataFrames in a more traditional SQL-query kind of way. When merging DataFrames, most of the time you want some information from one source and another piece of information from another source. Whereas when concatenating your DataFrames are structurally and in terms of content quite similar, and you want to combine them into one unified DataFrame.
A simple example I like to use during interviews is something along the lines of:
Suppose you have two tables. One table contains the names of your employees and a location id, the other table contains location ids and a city name. How can you get a list of every employee and the city they work in?
Merging two DataFrames in Pandas is done with pd.merge. Let’s have a look at the function`s signature (signature means a list of all the possible parameters for the function and typically also the output). I bolded the most relevant parameters.
pd.merge( left,  right,  how='inner',  on=None,  left_on=None,  right_on=None,          left_index=False, right_index=False,  sort=True, suffixes=('_x', '_y'),  copy=True,  indicator=False, validate=None) -> pd.DataFrame
Let’s go through the parameters one by one:
left/right : the left, respectively right, DataFrame you want to merge
how : 'left', 'right', 'outer', 'inner'. how defaults to 'inner'. See below a schematic overview of what each of them does. We will discuss specific examples a little bit later.
left_index/right_index : If True, use the index from the left/right DataFrame to merge on. left_index/right_index defaults to False
on : Column name(s) to merge on. Column name(s) have to exist in both the left and right DataFrame. If not passed and left_index and right_index are False, the intersection of the columns in both DataFrames will be inferred to join on.
left_on/right_on : Column name(s) from the left/right DataFrame to join on. Typical use case: Keys you are joining on are differently labeled in your DataFrames. E.g., what is location_id in your left DataFrame, might be _id in your right DataFrame. In this case, you would do left_on='location_id', right_on='_id' .
suffixes: A tuple of string suffixes to apply to overlapping columns. suffixes defaults to ('_x', '_y'). I like to use ('_base', '_joined').
Enough theory, let’s have a look at some examples. To do this, we need some additional data to merge on.
Load data:
# Load some additional Dataorder_data = pd.read_csv('https://raw.githubusercontent.com/FBosler/you-datascientist/master/order_leads.csv', parse_dates=[3])# Note the parse_dates? We need this to have consistent types with invoices, otherwise the subsequent merges would throw errorssales_team = pd.read_csv('https://raw.githubusercontent.com/FBosler/you-datascientist/master/sales_team.csv')
Inspect data:
We see pretty much right away, that Order ID ,Company ID , and Date appear in multiple of the DataFrames and are thus good candidates to merge on.
Let’s merge invoices with order_data and experiment with the parameters a bit.
No parameters provided: All parameters will use their defaults. The merge is going to be an inner merge (equivalent to how='inner'). The merge is going to be done on all common columns, i.e., Date, Order Id, and Company Id(equivalent to on=['Date','Order Id','Company Id]).Suffixes are not relevant as all common columns will be used to merge on, so there will no be duplicated columns remaining.Runpd.merge(order_data,invoices) :
how='left' : Again, the merge is based on all common columns. Suffixes are not relevant as all common columns will be used to merge on, so there will no be duplicated columns remaining. However, this time around, we merge with how='left' which means that we will take all rows from the left frame and only add data from the right frame where we find some. Runpd.merge(order_data,invoices,how='left') :
how='right' : Again, the merge is based on all common columns. Suffixes are not relevant as all common columns will be used to merge on, so there will no be duplicated columns remaining. However, this time around, we merge with how='right' which means that we will take all rows from the right frame and only add data from the left frame where we find some. This case is equivalent to 'inner' in our example as every row in the left DataFrame has a corresponding row in the right DataFrame. Runpd.merge(order_data,invoices,how='right') :
how='outer' : Again, the merge is based on all common columns. Suffixes are not relevant as all common columns will be used to merge on, so there will no be duplicated columns remaining. However, this time around, we merge with how='outer'. We take all rows from either, the left or the right DataFrame and add corresponding data where we find some in the other DataFrame. This case is equivalent to 'left' in our example as every row in the left DataFrame has a corresponding row in the right DataFrame. Runpd.merge(order_data,invoices,how='outer') :
If we do explicitly provide an on parameter this will override the default behavior and try to find the provided column in both DataFrames. Remaining duplicated columns that are not being used to merge on will be suffixed.
In the following example, we only merge on Order Id. However, since the Date and Company Id columns are also present in both DataFrames. Those columns will be sufficed to indicate from which source DataFrame they are originating from as can be seen in the following example.
Run: pd.merge(order_data,invoices,on='Order Id')
We can also specify custom suffixes like this:pd.merge(order_data,invoices,on='Order Id',suffixes=('_base','_join'))
You would typically use the left_on and right_on parameters when the columns are named differently in the two DataFrames.
Note on Join :You might have seen the usage of jointo the same end asmerge. join by default merges on the index of both DataFrames. I advise against the usage of joinas it is just a special case of mergeand does not provide a benefit over merging.
To wrap up the chapter on combining DataFrames, we should quickly talk about map. map can be called on a DataFrame column or its Index like this:
The argument (in our case lookup) for map always has to be a series or a dictionary. While not precisely the same Pandas` series and regular dictionaries share a lot of functionalities and can often be used interchangeably. Like through a dictionary you could loop through a series by calling for k,v in series.items(): .
Use pd.concat to “stack” multiple DataFrame on top or next to each other. By default, the stacking is vertical. To overwrite the default use axis=1. By default, stack will carry over all columns/indices. To limit to common columns/indices use join='inner'.
Do not use pd.DataFrame.append as it is only a special, limited case of pd.concat and will only make your code less standardized and less coherent. Instead, use pd.concat.
Use pd.merge to combine information from two DataFrames. Merge defaults to an inner join and will infer the columns to merge on from the largest subset of common columns across the DataFrames.
Do not use pd.DataFrame.join as it is only a special, limited case of pd.merge and will only make your code less standardized and less coherent. Instead, use pd.merge.
Use pd.Series.map as a lookup-esque kind of functionality to get the value for a specific index/key from a series/dictionary.
Transposing a DataFrame means to swap the index and column. In other words, you are rotating the DataFrame around the origin. Transposing does not change the content of the DataFrame. The DataFrame only changes the orientation. Let’s visualize this with an example:
A DataFrame is transposed by simply calling .T on the DataFrame e.g.invoices.T).
Melt transforms a DataFrame from wide format to long format. Melt gives flexibility around how the transformation should take place. In other words, melt allows grabbing columns and transforming them into rows while leaving other columns unchanged. Melt is best explained with an example. Let’s create some sample data:
melt_experiment = pd.merge(    invoices,    pd.get_dummies(invoices['Type of Meal']).mul(invoices['Meal Price'].values,axis=0),    left_index=True,    right_index=True)del melt_experiment['Type of Meal']del melt_experiment['Meal Price']melt_experiment
Admittedly, this example is a bit contrived but illustrates the point. We turned the Type of Meal into columns and assigned the prices into the corresponding rows. Now to turn this back into a version where the Type of Meal is a column and the value is the price we could use pd.melt like this:
pd.melt(    frame=melt_experiment,    id_vars=['Order Id', 'Date', 'Meal Id', 'Company Id', 'Date of Meal','Participants', 'Heroes Adjustment'],    value_vars=['Breakfast', 'Dinner', 'Lunch'],    var_name='Type of Meal',    value_name='Expenses')
resulting in:
Melt is useful to transform a DataFrame into a format where one or more columns are identifier variables (id_vars), while all other columns, considered measured variables (value_vars), are moved to the row axis, leaving just two non-identifier columns. For each column we melt (value_vars), the corresponding existing row is duplicated to accommodate fusing data into a single column and our DataFrame extends. After melting, we have three times as many rows as before (because we used three value_vars and thus triplicated every row).
We discussed in the previous article, but a quick recap makes sense here, especially in the light of stacking and unstacking, which we will talk about later. Where transpose and melt leave the content of the DataFrame intact and “only” rearrange the appearance groupby and the following methods are aggregating the data in one form or another. Groupby will result in an aggregated DataFrame with a new index (the values of the columns, by which you are grouping). If you are grouping by more than one value, the resulting DataFrame will have a multi-index.
invoices.groupby(['Company Id','Type of Meal']).agg(    {'Meal Price':np.mean})
Starting with pandas version 0.25.1, there are also named aggregation, which make groupby a little more readable.
invoices.groupby(['Company Id','Type of Meal']).agg(    Avg_Price = pd.NamedAgg(column='Meal Price', aggfunc=np.mean))
Resulting in virtually the same as the previous calculation. However, the column is renamed during the process.
Pandas also incorporates a pivot_table functionality, not unlike Excel’s Pivot Table. I must admit though, I never use pivot, as I don’t see the advantage over groupby operations. Generally speaking, I think it makes sense to use what you are comfortable with and stick to that. I would advise against mixing different approaches when there is no necessity for doing so. The one good thing pivot_table has in its favor though is margin=True
pd.pivot_table(    invoices,    index=['Company Id','Type of Meal'],    values='Meal Price',    aggfunc=np.mean,    margins=True)
The result should look somewhat familiar, as we basically recreated the groupby functionality. However, we get the added benefit of getting the calculated value across all groups in addition to the individual group’s results (as indicated by the last line).
We can also specify the columns for the pivot table:
pd.pivot_table(    invoices,    index=['Company Id'],    columns=['Type of Meal'],    values='Meal Price',    aggfunc=np.mean,    margins=True)
Stack and unstack come in really handy when rearranging your columns and indices. Unstack will by default be called on the outmost level of the index as can be seen nicely in the following example, where calling unstack() turns the Heroes Adjustment Index into two columns.
We can also unstack a specific level of the index, like in the following example, where we unstack the Type of Meal column
Stacking, on the other hand, does the opposite. Stacking turns columns into rows, but also extends the index in the process (as opposed to melt). Let’s have a look at an example.
We have to build some data first where stacking would come in handy. Executing the following code snippet results in a multi-index, multi-level-columns DataFrame.
stack_test = invoices.groupby(['Company Id','Type of Meal']).agg({    'Meal Price':[max,min,np.mean],    'Date of Meal':[max,min],})stack_test
If we want to stack this DataFrame, we would call stack_test.stack() and get:
Here stack used the outmost level of our multi-level columns and turned it into an index. We now have only single-level columns. Alternatively, we could also call stack with level=0 and get the following result:
In this article, you learned how to become a real Pandas Ninja. You learned how to convert data into desired types and between types. You learned how to use unique methods for those types to get access to functionalities, that would otherwise take lines and lines of code. You learned how to combine different DataFrames by just stacking them on top of each other, or logically extracting information from the DataFrames and combining them into something more meaningful. You learned how to flip your DataFrame around as if it were a pancake. You learned to rotate on its origin point, to move columns into rows, to aggregate data through pivot or groupby and then to stack and unstack the results.
Good job, and thanks for reading!