Using Panda’s “transform” and “apply” to deal with missing data on a group level | by Fabian Bosler | Towards Data Science

According to an analysis conducted by Businessbroadway, a data professional spends up to 60% of their time gathering, cleaning data, and visualizing data.
A critical aspect of cleaning and visualizing data revolves around how to deal with missing data. Pandas offers some basic functionalities in the form of the fillna method. While fillna works well in the simplest of cases, it falls short as soon as groups within the data or order of the data become relevant. This article is going to discuss techniques to address those more complex cases.
Those cases are typically datasets that consist of different regimes (time-series), groups, and even subgroups. Examples of different regimes are months, quarters (time ranges in general), or a period of heavy rain. An example of groups within the data is gender. Examples of subgroups are age cohorts and ethnicity.
The article is intended as a code-along article. So feel free to boot up a Notebook and dive right in.
Recap of Pandas’ fillnaDealing with missing data, when the order is not relevantDealing with missing data, when the order is relevant
Recap of Pandas’ fillna
Dealing with missing data, when the order is not relevant
Dealing with missing data, when the order is relevant
Pandas has three modes of dealing with missing data via calling fillna():
method='ffill': Ffill or forward-fill propagates the last observed non-null value forward until another non-null value is encountered
method='bfill': Bfill or backward-fill propagates the first observed non-null value backward until another non-null value is met
explicit value: It is also possible to set an exact value to replace all missings. Such a substitution could be -999, for example, to indicate that the value is missing.
Example:
IN:demo = pd.Series(range(6))demo.loc[2:4] = np.nandemoOUT:0    0.01    1.02    NaN3    NaN4    NaN5    5.0dtype: float64# Forward-FillIN:demo.fillna(method='ffill')OUT:0    0.01    1.02    1.03    1.04    1.05    5.0dtype: float64# Backward-FillIN:demo.fillna(method='bfill')OUT:0    0.01    1.02    5.03    5.04    5.05    5.0dtype: float64# Explicit valueIN:demo.fillna(-999)OUT:0      0.01      1.02   -999.03   -999.04   -999.05      5.0dtype: float64
Frequently, when dealing with missing data, the sequencing does not matter, and thus, the values used to replace missing values can be based on the entirety of available data. In such cases, you would typically replace the missing values with your best guess (i.e., mean or medium of the available data)
Let’s quickly go over why you should use this option with care. Let’s assume you survey 1000 boys and 1000 girls for their weights. During the process of collecting the data, unfortunately, some of the data went missing.
# importsimport numpy as np# sample 1000 boys and 1000 girlsboys = np.random.normal(70,5,1000)girls = np.random.normal(50,3,1000)# unfortunately, the intern running the survey on the girls got distracted and lost 100 samplesfor i in range(100):    girls[np.random.randint(0,1000)] = np.nan# build DataFrameboys = pd.DataFrame(boys, columns=['weight'])boys['gender'] = 'boy'girls = pd.DataFrame(girls, columns=['weight'])girls['gender'] = 'girl'df = pd.concat([girls,boys],axis=0)df['weight'] = df['weight'].astype(float)
Without giving it much thought, we might fill the missing values with the entire sample’s mean. The results look somewhat strange, though. The girl’s KDE has two humps. One might conclude that there is a subgroup of heavier girls among our sample. Because we built the distributions beforehand, we know this isn’t the case. But if this were real data, we might draw wrong conclusions from this data.
# PLOT CODE:sns.set_style('white')fig, ax = plt.subplots(figsize=(16, 7))mean = df['weight'].mean()sns.distplot(    df[df['gender'] == 'girl']['weight'].fillna(mean),    kde=True,    hist=False,    ax=ax,    label='girls')sns.distplot(    df[df['gender'] == 'boy']['weight'],    kde=True,    hist=False,    ax=ax,    label='boys')plt.title('Kernel density estimation of weight for boys and girls')sns.despine()
In such situations, Panda’s transform function comes in handy. Using transform gives a convenient way of fixing the problem on a group level like this:
df['filled_weight'] = df.groupby('gender')['weight'].transform(    lambda grp: grp.fillna(np.mean(grp)))
Running the above command and plotting the KDE of the filled_weight values results in:
# PLOT CODE:sns.set_style('white')fig, ax = plt.subplots(figsize=(16, 7))sns.distplot(    df[df['gender'] == 'girl']['filled_weight'],    kde=True,    hist=False,    ax=ax,    label='girls')sns.distplot(    df[df['gender'] == 'boy']['filled_weight'],    kde=True,    hist=False,    ax=ax,    label='boys')plt.title('Kernel density estimation of weight for boys and girls')sns.despine()
Let’s use the previous example, but this time, we go a bit further and additionally segment our data into age groups. Let’s create some mock data:
# paramter for the weight distribution (mean, std)param_map = {    'boy':{        '<10':(40,4),        '<20':(60,4),        '20+':(70,5),    },    'girl':{        '<10':(30,2),        '<20':(40,3),        '20+':(50,3),    }}# generate 10k recordsdf = pd.DataFrame({    'gender':np.random.choice(['girl','boy'],10000),    'age_cohort':np.random.choice(['<10','<20','20+'],10000)})# set random weight based on parametersdf['weight'] = df.apply(    lambda x: np.random.normal(        loc=param_map[x['gender']][x['age_cohort']][0],        scale=param_map[x['gender']][x['age_cohort']][1]    ),axis=1)# set 500 values missingfor i in range(500):    df.loc[np.random.randint(0,len(df)),'weight'] = np.nan
Plotting the data reveals somewhat strange bimodal distributions (code below).
# PLOT CODEdf['filled_weight'] = df['weight'].fillna(    df['weight'].mean())g = sns.FacetGrid(    df,    col='age_cohort',    row='gender',    col_order=['<10','<20','20+'])g.map(sns.kdeplot,'filled_weight')
Now, if we would only replace the missing values with the mean for the respective gender, this would not go far enough as not only boys and girls are different in weight, but also different age groups are quite different in weight.
Luckily, transform can be applied just as before. We would group on both columns instead of only one like this:
df['filled_weight'] = df.groupby(['gender','age_cohort'])['weight'].transform(    lambda grp: grp.fillna(np.mean(grp)))
Running the above snippet will generate this much cleaner plot:
Two situations frequently come up when working with time-series data.
Aligning date ranges: Say you look at countries’ GDP, education level, and population growth by year. For some countries, you are missing the very first, some of the later years, or some years in between. Sure, you could ignore them. Still, for visualization purposes, you might want to keep them around and project your first observed value to the beginning, your last observed value to the end of your survey period, and come up with something that makes sense for the values in the middle.Interpolation: Looking at time-series data interpolation and thus ordering becomes very relevant. Replacing missing stock data from 2012 with an average that is calculated based on data up to 2019 is bound to produce some wacky results.
Aligning date ranges: Say you look at countries’ GDP, education level, and population growth by year. For some countries, you are missing the very first, some of the later years, or some years in between. Sure, you could ignore them. Still, for visualization purposes, you might want to keep them around and project your first observed value to the beginning, your last observed value to the end of your survey period, and come up with something that makes sense for the values in the middle.
Interpolation: Looking at time-series data interpolation and thus ordering becomes very relevant. Replacing missing stock data from 2012 with an average that is calculated based on data up to 2019 is bound to produce some wacky results.
We will look at an example based on data from the World Happiness Report 2019, where we address both situations. The World Happiness Report tries to answer which factors influence happiness throughout the world. The report surveys data from the year 2005 to 2018.
# Load the datadf = pd.read_csv('https://raw.githubusercontent.com/FBosler/you-datascientist/master/happiness_with_continent.csv')
df.sample(5) as opposed to df.head(5) selects five random rows, thus giving you a more unbiased view of the data.
Let’s take a look at the number of countries per year we have data for.
# PLOT CODE:df.groupby(['Year']).size().plot(    kind='bar',    title='Number of countries with data',    figsize=(10,5))
We can see that, especially for the earlier years, we don’t have many countries with data, and also there is some fluctuation across the entire sample period. To mitigate the effect of missing data, we will do the following:
Group by country and reindex to the whole date rangeInterpolate for years between and extrapolate for years outside the range for which we have observations on a per-country basis
Group by country and reindex to the whole date range
Interpolate for years between and extrapolate for years outside the range for which we have observations on a per-country basis
# Define helper functiondef add_missing_years(grp):    _ = grp.set_index('Year')    _ = _.reindex(list(range(2005,2019)))    del _['Country name']    return _# Group by country name and extenddf = df.groupby('Country name').apply(add_missing_years)df = df.reset_index()
We now have roughly 600 more rows. However, those observations are null right now
# Define helper functiondef fill_missing(grp):    res = grp.set_index('Year')\    .interpolate(method='linear',limit=5)\    .fillna(method='ffill')\    .fillna(method='bfill')    del res['Country name']    return res# Group by country name and fill missingdf = df.groupby(['Country name']).apply(    lambda grp: fill_missing(grp))df = df.reset_index()
The fill_missing function interpolates and extrapolates towards the end and the beginning, resulting in:
Perfect! Now we have data for the years 2005 to 2018 for all countries in our sample. The above methods were quite relevant for me when writing this article about visualizations. Check it out if you want to learn more about the happiness report.
towardsdatascience.com
That’s it for today. In this article, you learned how to use transformand apply to replace missing values with something more meaningful than the previous or subsequent value.
Reach out to me on LinkedIn if you discover some fantastic new visualizations, want to give feedback or have a chat.
If you liked what you’ve read, check out the other articles I wrote on Medium.