How to master Python’s main data analysis library in 20 Minutes | by Fabian Bosler | Towards Data Science

This article addresses typical problems and challenges aspiring data scientists and data analysts are facing regularly. We will solve those problems and tackle those challenges by utilizing Python’s most powerful library for data manipulation and analysis, Pandas.
SetupLoading DataInspecting / Sorting / Filtering dataAnalytical Functions
Setup
Loading Data
Inspecting / Sorting / Filtering data
Analytical Functions
You can find the full Jupyter Notebook here. But I urge you to go through the steps yourself. After all, Practice makes perfect.
A working Python environment (I suggest Jupyter Notebook). If you haven’t set this up, don’t worry. In last week’s article, we talked about how to set up Anaconda and explained how to open your first Jupyter Notebook. In case you haven’t done that yet, have a look at the linked article. Getting all set up and ready to go takes less than 10 minutes.
towardsdatascience.com
Before doing any data manipulation, let’s get some data. We will use data from the World Happiness Report 2019. I am going to provide a slightly adjusted version of the raw data that also includes continents.
This GitHub Repo holds the data and the code. If you are not familiar with GitHub, you can also download a packaged zip file from this link! Unpack the file and move the content (especially thehappiness_with_continent.csv) into the same folder as your Jupyter Notebook (if you don’t have one yet, create one).
In the new notebook run import pandas as pd (i.e., import the Pandas library into the workbook to get access to the features.
I like to adjust my notebook settings a bit like this:
from IPython.core.display import display, HTMLdisplay(HTML("<style>.container {width:90% !important;}</style>"))
These commands make the Notebook wider and thus utilize more space on the screen (typically the notebook has a fixed width, which sucks with wide screens).
Pandas stores data either as Series (one column) or as DataFrames (one or more columns), where the latter is simply one or more Series combined.
Note: Whenever we load data with one of the following load functions, the result will be stored in DataFrame though.
For me the main way of loading data is pandas read_csv. It simply aligns best with the way I think about data, which is largely tabular.
You can load data from a local file like this:
data = pd.read_csv('happiness_with_continent.csv')
Or you can read data from the web directly into a dataframe like this:
data = pd.read_csv('https://raw.githubusercontent.com/FBosler/you-datascientist/master/happiness_with_continent.csv')
Reading data from Excel is pretty straight forward. Google Sheets is a bit trickier as it requires you to go through an authentication procedure first. You can read up all on extracting data from Excel and Google Sheets here:
towardsdatascience.com
This one I use rarely, but certainly works for smaller tables. Just mark and copy (ctrl+c) a table from google sheets for example and run pd.read_clipboard().
Example: navigate here (first public sheet I found) and mark an area as indicated in the screenshot.
Some noteworthy parameters for the read_csv based functions (and read_clipboard):
sep : separator for the columns (defaults to , but could also be tab)
header : defaults to 'infer' (i.e., Pandas guesses as to what your header is), alternatives are an integer or a list of integers (for multi-level names). E.g., you could do header=3 and the dataframe would start with row 4 (because Python is 0-indexed) as the header. If your data has no header use header=None
names : names of the columns. If you want to use this parameter to override whatever column names Pandas had inferred, you should specify header=0 (or whatever line your columns names are in), if you do not do this, you will have your names as the column names and then the original column names in the first row. The names parameter expects a list, e.g. ['your col 1', 'your col 2', ... 'your last col name']
index_col : set’s an index on load (i.e., we set index to be name . We will learn more about indexes later)
skiprows : skips the first x rows, this is useful when the file contains some metadata at the beginning like author and other information
skipfooter : skips the last x rows, useful when there is metadata at the end of the file (e.g., footnotes)
parse_date : This parameter tells Pandas, which columns it should interpret as a date (e.g. pd.read_csv(happiness_with_continent.csv,parse_dates=['Year']). The default parser works reasonably well out of the box. In cases where you encounter funky data formatting, Pandas can work with custom date parsers (for which you would have to specify the parsing logic).
There are a bunch of additional (less used) parameters. You can read up on those by running pd.read_csv? in a cell (adding a question mark after a command will print the help text).
No matter how we read the data, we want to store it in a variable. We do this by assigning the read result to a variable like this data = pd.read_clipboard() or data = pd.read_csv('NAME_OF_YOUR_FILE.csv')
The following read methods rarely come up for me but are all also implemented in Pandas:
read_feather
read_fwf
read_gbq
read_hdf
read_html
read_json
read_msgpack
read_parquet
read_pickle
read_sas
read_sql
read_sql_query
read_sql_table
read_stata
read_table
There are three standard ways of displaying data in a Notebook, head, tail, and sample. head shows the first, tail the last, and sample a random selection of rows.
Note that there are dots before the column gini of household income reported in Gallop, by wp5-year . The dots mean that there are columns that are not displayed. To change your Notebooks settings to display more columns/rows run the following commands:
pd.set_option('display.max_columns', <number of columns you want>)pd.set_option('display.max_rows', <number of rows you want>)# I typically usepd.set_option('display.max_columns', 50)pd.set_option('display.max_rows', 8)
However, be aware of the fact that frequently the files that you are loading are so large (1GB+) that it is not going to be possible to display all the data for performance reasons. For this reason, you should try to familiarise yourself with the data on a higher level and not depend on visually going through the rows.
data.shape returns the dimensions of the DataFrame. In our example, 1704 rows, 27 columns.
IN:data.shapeOUT:(1704, 27)
data.columns returns a list of all the column names in the DataFrame.
IN:data.columnsOUT:Index(['Country name', 'Year', 'Life Ladder', 'Log GDP per capita',       'Social support', 'Healthy life expectancy at birth',       'Freedom to make life choices', 'Generosity',       'Perceptions of corruption', 'Positive affect', 'Negative affect',       'Confidence in national government', 'Democratic Quality',       'Delivery Quality', 'Standard deviation of ladder by country-year',       'Standard deviation/Mean of ladder by country-year',       'GINI index (World Bank estimate)',       'GINI index (World Bank estimate), average 2000-16',       'gini of household income reported in Gallup, by wp5-year',       'Most people can be trusted, Gallup',       'Most people can be trusted, WVS round 1981-1984',       'Most people can be trusted, WVS round 1989-1993',       'Most people can be trusted, WVS round 1994-1998',       'Most people can be trusted, WVS round 1999-2004',       'Most people can be trusted, WVS round 2005-2009',       'Most people can be trusted, WVS round 2010-2014',       'Continent'],      dtype='object')
data.index returns information about the index. We will talk about indices in more detail in the sort and filter section. Think of an index as the row names/numbers.
IN:data.indexOUT:RangeIndex(start=0, stop=1704, step=1)
data.info() returns information regarding types and number of non-null observations in the DataFrame
IN:data.info()OUT:<class 'pandas.core.frame.DataFrame'>RangeIndex: 1704 entries, 0 to 1703Data columns (total 27 columns):Country name                                                1704 non-null objectYear                                                        1704 non-null datetime64[ns]Life Ladder                                                 1704 non-null float64Log GDP per capita                                          1676 non-null float64Social support                                              1691 non-null float64Healthy life expectancy at birth                            1676 non-null float64Freedom to make life choices                                1675 non-null float64Generosity                                                  1622 non-null float64Perceptions of corruption                                   1608 non-null float64Positive affect                                             1685 non-null float64Negative affect                                             1691 non-null float64Confidence in national government                           1530 non-null float64Democratic Quality                                          1558 non-null float64Delivery Quality                                            1559 non-null float64Standard deviation of ladder by country-year                1704 non-null float64Standard deviation/Mean of ladder by country-year           1704 non-null float64GINI index (World Bank estimate)                            643 non-null float64GINI index (World Bank estimate), average 2000-16           1502 non-null float64gini of household income reported in Gallup, by wp5-year    1335 non-null float64Most people can be trusted, Gallup                          180 non-null float64Most people can be trusted, WVS round 1981-1984             125 non-null float64Most people can be trusted, WVS round 1989-1993             220 non-null float64Most people can be trusted, WVS round 1994-1998             618 non-null float64Most people can be trusted, WVS round 1999-2004             491 non-null float64Most people can be trusted, WVS round 2005-2009             630 non-null float64Most people can be trusted, WVS round 2010-2014             671 non-null float64Continent                                                   1704 non-null objectdtypes: datetime64[ns](1), float64(24), object(3)memory usage: 372.8+ KB
data.describe() returns some descriptive statistical information (count, mean, std, min, 25%, 50%, 75%, max) regarding the numerical columns of the DataFrame:
Calling sort_values on data without parameter will not do us any good. As a matter of fact, it will raise an error, telling us that it is missing an argument called by. This error makes sense. We have to tell Pandas by which column(s) we want to sort.
We could, for example, sort our data by year, or by year and country name like this:
data.sort_values(by='Year')data.sort_values(by=['Year','Country name'])data.sort_values(by=['Country name','Year'])
Note: If you pass multiple values, it will sort by the values in order of the values.
The sorting will be done “lowest value” first by default. However, it’s easy to change this behavior.
data.sort_values(by='Year', ascending=True)data.sort_values(  by=['Country name','Year'],   ascending=[False,True])
Note: Ascending is by default True, i.e., the smallest values first, if you want the largest values first you have to specify ascending=False
In addition to column-based sorting, there is also index-based sorting. To sort by index call: data.sort_index() or data.sort_index(ascending=False) . The first is for ascending order, the latter one for descending order.
Sorting is all good and nice, but often we are concerned with a particular subset of the data. Sometimes you might want to look at only one or multiple columns.
Selecting one column:There are two ways of selecting one particular column. Let’s say that we want to select the Year column. We could do either:
data['Year'] , or
data.Year (don’t use this approach)
Both do the same thing.
Note: Why are there two ways of doing precisely the same thing you might ask? The reason is convenience. The second approach is a tad faster, as you only need two dots and the column name. Whereas in the first method you need the column name, two upper ticks and two brackets.
However, I strongly advocate for using the first approach, as it avoids some minor problems down the road and is consistent with selecting multiple columns.
Selecting multiple columns:Say you want to select Country name and Life Ladder then this is how you would do it (careful: double brackets):
Note: Be especially mindful of the double brackets before the first and after the last column you want to select! Whenever you use double brackets, the result will be a DataFrame (even when only selecting one column with double brackets).I can’t stress this enough as even I still sometimes run into these errors! If you want to select multiple columns, but only open one set of brackets, the content between the brackets will be considered ONE column. Needless to say that your data does not contain that accidentally combined column.
Being able to select specific columns is only halfway there. However, selecting rows is just as easy.
Rows in Pandas are selected by index. You can think of indices as the names of the rows. Whenever you select rows from a DataFrame, this happens by overlaying the DataFrame with a Series with the same index containing only True and False values ( True meaning that the row should be selected, False that the row should not be selected). However, most of the time, this explicit index selection is abstracted away from the user. I still think it is tremendously important to understand how the row selection process works under the hood.
You can select one row or multiple rows by index. There are two methods for doing that:
data.iloc or
data.loc
iloc:data.iloc allows selecting rows (and optionally columns) by position (i.e., by the number of the row).
iloc — Selecting one row: The syntax looks like this data.iloc[row_number (,col_number)] where the part in parenthesis is optional.
Note: The formatting looks a bit unconventional, this is because when selecting one row and one row only, a Series is going to be returned.
iloc — Selecting multiple rows:The syntax looks like this data.iloc[start_row:end_row (,start_col:end_col)] where the part in parenthesis is optional.
Alternatively, you can additionally specify which columns to select.
loc:data.loc as opposed to iloc allows selecting rows (and columns) by:
label/index orwith a boolean/conditional lookup
label/index or
with a boolean/conditional lookup
To better explain point one and also better distinguish it from iloc we will turn the country name into the DataFrame`s index. To achieve this, run the following command:
data.set_index('Country name',inplace=True)
The set_index command sets a new index on the DataFrame. By also specifying inplace=True we make sure that the DataFrame will be changed. If we did not specify inplace=True, we would only see how the DataFrame would look like after applying the operation, but no change to the underlying data would take place.
The DataFrame should now look like this:
We can see that the DataFrame lost its row numbers (the previous) index and gained a new index:
loc — Selecting row(s) by one index label:The syntax looks like this data.loc[index_label (,col_label)] where the part in parenthesis is optional.
loc — Selecting rows and a column by index label and column label:
loc — Selecting row(s) by multiple index labels:
Notes:
Like earlier, when selecting multiple columns, we have to make sure to put them in double brackets. If we forget to do that, the columns would be considered to be one long (non-existing) name.
We use sample(5) to show that there is some Germany in the mix. Assuming we used head(5) instead, we would only start seeing Germany after 12 rows of United States.
Loc returns the rows in the provided order, irrespective of their actual ordering. For example, had we specified Germany first and United States second, we would have 13 rows of Germany and then 12 rows of United States
loc — Selecting rows and columns by multiple index labels:You can also specify the column names for the selected rows that you want to return.
Note: We spread the row selection ['Germany','United States] and the column selection ['Year','Life Ladder'] out across two lines. I find that splitting the statement makes for better readability.
loc — Selecting row(s) by a range of index labels:This way of selecting rows can feel a bit strange, as a range of labels ('Denmark':'Germany') is not quite as intuitive as using a range of numbers ( 903:907) for iloc.
Specifying a range of labels is based on the current ordering of the index, and it will fail on an unsorted index.
However, assuming that your index is sorted, or you sorted it before selecting a range, you can do the following:
loc — boolean/conditional lookupBoolean or conditional lookup is really where the meat is. As mentioned earlier, whenever selecting rows, this happens by overlaying the DataFrame with a mask of True and False values.
In the following example, we create a small DataFrame with the index ['A','B','A','D'] and some random values between 0 and 10 as values.
We then create an overlay with the same Index with the values [True,False,True,False].
We then use df.loc[overlay] to only select the rows with True value for their index.
IN:from numpy.random import randintindex = ['A','B','A','D']## create dummy DataFrame ##df = pd.DataFrame(    index = index,    data = {    'values':randint(10,size=len(index))})print('DataFrame:')print(df)OUT:DataFrame:   valuesA       8B       2A       3D       2IN:## create dummy overlay ##overlay = pd.Series(    index=index,    data=[True,False,True,False])print('\nOverlay:')print(overlay)OUT:Overlay:A     TrueB    FalseA     TrueD    Falsedtype: boolIN:## select only True rows ##print('\nMasked DataFrame:')print(df.loc[overlay])OUT:Masked DataFrame:   valuesA       8A       3
The same logic can be used to select rows based on a (or multiple) condition(s).
We first create a boolean mask like this:
And then use this mask to only select the rows that meet the specified condition like this:
Option 1, as well, as the alternative yield precisely the same result. However, the alternative is a little more legible. The improved legibility becomes even more apparent when applying multiple conditions:
Note: We used & (bitwise and) to filter for rows, where multiple conditions apply at the same time. We can use | (bitwise or) to filter for columns, where one of the conditions applies.
loc — advanced conditional lookup with custom formulas
It is also possible and quite easy to use customized functions as a condition and apply them to select columns.
In the following example, we only select years that are cleanly divisible by three and continents that contain the word America. The case is contrived but makes a point.
Instead of lambda (anonymous) functions, you could also define and use much more complicated functions. You could even (not that I recommend it) make API calls in a custom function and use the results of the calls to filter your dataframe.
Now that we are comfortable with filtering and sorting the data front to back and vice versa, let’s move to some more advanced analytical functionalities.
Like the read functions, there are also a lot of analytical functions implemented in Pandas.
I will highlight and explain the ones I use most frequently. However, and that’s part of the beauty of it, even I will find new useful functions from time to time. So never start reading and exploring!
1 max/min
2 sum
3 mean/median/quantile
4 idxmin/idxmax
Note: All functions can be applied column-wise, but also row-wise. The row-wise application makes very little sense in our example. However, frequently, you have data, where you want to compare different columns, in which case the row-wise application does make sense.
Whenever we call the aforementioned functions, a default parameter axis=0 is passed (for column-wise application). However, we can override this parameter and pass axis=1 (for row-wise application).
1 max/minCalling max() on data, will return (wherever possible) the maximum for each column. min() does the exact opposite.
IN:data.max() # COLUMNWISE MAXIMUMOUT: Year                                                        2018Life Ladder                                              8.01893Log GDP per capita                                       11.7703Social support                                          0.987343                                                       ...      Most people can be trusted, WVS round 1999-2004         0.637185Most people can be trusted, WVS round 2005-2009         0.737305Most people can be trusted, WVS round 2010-2014         0.661757Continent                                          South AmericaLength: 26, dtype: objectIN:data.max(axis=1) # ROW-WISE MAXIMUMOUT: Country nameAfghanistan    2008.0Afghanistan    2009.0Afghanistan    2010.0Afghanistan    2011.0                ...  Zimbabwe       2015.0Zimbabwe       2016.0Zimbabwe       2017.0Zimbabwe       2018.0Length: 1704, dtype: float64
2 sumCalling sum() on data, will return (wherever possible) the sum for each column.
IN:data.sum()OUT:Year                                                                                         3429014Life Ladder                                                                                  9264.91Log GDP per capita                                                                           15456.8Social support                                                                               1370.67                                                                         ...                        Most people can be trusted, WVS round 1999-2004                                              131.623Most people can be trusted, WVS round 2005-2009                                              166.532Most people can be trusted, WVS round 2010-2014                                              159.358Continent                                          AsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaEu...Length: 26, dtype: object
Note: Sum will concatenate strings into one long string, which results in AsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaAsiaEu... for the Continent column.
3 mean/median/quantileCalling mean ,median , or quantile on data will return the mean or median, respectively.
IN:data.mean()OUT:Year                                               2012.332160Life Ladder                                           5.437155Log GDP per capita                                    9.222456Social support                                        0.810570                                                      ...     Most people can be trusted, WVS round 1994-1998       0.249574Most people can be trusted, WVS round 1999-2004       0.268070Most people can be trusted, WVS round 2005-2009       0.264336Most people can be trusted, WVS round 2010-2014       0.237493Length: 25, dtype: float64IN:data.median()OUT:Year                                               2012.000000Life Ladder                                           5.339557Log GDP per capita                                    9.406206Social support                                        0.833098                                                      ...     Most people can be trusted, WVS round 1994-1998       0.229924Most people can be trusted, WVS round 1999-2004       0.232000Most people can be trusted, WVS round 2005-2009       0.198380Most people can be trusted, WVS round 2010-2014       0.193531Length: 25, dtype: float64IN:data.quantile(q=.8)OUT:Year                                               2016.000000Life Ladder                                           6.497157Log GDP per capita                                   10.375623Social support                                        0.913667                                                      ...     Most people can be trusted, WVS round 1994-1998       0.304498Most people can be trusted, WVS round 1999-2004       0.388611Most people can be trusted, WVS round 2005-2009       0.415082Most people can be trusted, WVS round 2010-2014       0.373906Name: 0.8, Length: 25, dtype: float64
4 idxmin/idxmaxCalling idxmax or idxmin on data will return the index of the row where the first minimum/maximum is found. However, it is only possible to call this on columns with some ordinality to them.
IN:data.iloc[:,:-1].idxmax() # We exclude the Continent ColumnOUT:Year                                               AfghanistanLife Ladder                                            DenmarkLog GDP per capita                                       QatarSocial support                                     New Zealand                                                      ...     Most people can be trusted, WVS round 1994-1998         NorwayMost people can be trusted, WVS round 1999-2004         SwedenMost people can be trusted, WVS round 2005-2009         NorwayMost people can be trusted, WVS round 2010-2014    NetherlandsLength: 25, dtype: object
That means, for example, Denmark has the highest Life Ladder , Qatar has the highest Log GDP per capita and New Zealand the highest value for Social support.
idxmin works the same as idxmax .
Recap: Don’t forget that you can apply all these functions column-wise (axis=0) or row-wise (axis=1)
You can also write custom functions and use them on your rows or columns. There are two kinds of custom functions:
Named Functions
Lambda Functions
Named functions are user-defined functions. They are defined through the usage of the reserved keyword def like this:
Named Functions:
FUNCTION:def above_1000_below_10(x):    try:        pd.to_numeric(x)    except:        return 'no number column'        if x > 1000:        return 'above_1000'    elif x < 10:        return 'below_10'    else:        return 'mid'IN:data['Year'].apply(above_1000_below_10)OUT:Country nameAfghanistan    above_1000Afghanistan    above_1000Afghanistan    above_1000Afghanistan    above_1000                  ...    Zimbabwe       above_1000Zimbabwe       above_1000Zimbabwe       above_1000Zimbabwe       above_1000Name: Year, Length: 1704, dtype: object
Here we did define a function called above_1000_below_10 and applied that to our data.
The function initially checks, if the value is convertible to a number and if not will return ‘no number column.’ Otherwise, the function returns above_1000 if the value is above 1000 and below_10 if the value is below 10, else it returns mid.
Lambda Functions:For me, lambda functions come up much frequently than named functions. In essence, those are short, throw-away functions for one-time use only. The name sounds clunky, but once you got the hang of it, they are quite convenient. For example, we could first split the continent column on space and then grab the last word of the results.
IN:data['Continent'].apply(lambda x: x.split(' ')[-1])OUT:Country nameAfghanistan      AsiaAfghanistan      AsiaAfghanistan      AsiaAfghanistan      Asia                ...  Zimbabwe       AfricaZimbabwe       AfricaZimbabwe       AfricaZimbabwe       AfricaName: Continent, Length: 1704, dtype: object
Note: Both, named and lambda functions, we applied to individual columns as opposed to the entire dataframe. When applying a function to a particular column, the function goes row by row. When applying a function to a whole DataFrame, the function goes column by column and is applied to the entire column then and has to be written a little differently, like so:
IN:def country_before_2015(df):    if df['Year'] < 2015:        return df.name    else:        return df['Continent']# Note the axis=1data.apply(country_before_2015, axis=1)OUT:Country nameAfghanistan    AfghanistanAfghanistan    AfghanistanAfghanistan    AfghanistanAfghanistan    Afghanistan                  ...     Zimbabwe            AfricaZimbabwe            AfricaZimbabwe            AfricaZimbabwe            AfricaLength: 1704, dtype: object
In this example, we also go row by row (as specified by the axis=1 ). We return the name of the row (which happens to be the index) when the Year of that row is smaller than 2015 or else the continent of that row. Tasks like this do come up when you have to do conditional data cleaning.
Sometimes you want to add, subtract or otherwise combine two or multiple columns, which really could hardly be any simpler.
Say we want to add Year and Life Ladder (contrived, I know, but let’s do it for the sake of argument).
IN:data['Year'] + data['Life Ladder']OUT:Country nameAfghanistan    2011.723590Afghanistan    2013.401778Afghanistan    2014.758381Afghanistan    2014.831719                  ...     Zimbabwe       2018.703191Zimbabwe       2019.735400Zimbabwe       2020.638300Zimbabwe       2021.616480Length: 1704, dtype: float64
Same with -, *, / and many more you could also do string operations, like this:
IN:data['Continent'] + '_' + data['Year'].astype(str)OUT:Country nameAfghanistan      Asia_2008Afghanistan      Asia_2009Afghanistan      Asia_2010Afghanistan      Asia_2011                  ...     Zimbabwe       Africa_2015Zimbabwe       Africa_2016Zimbabwe       Africa_2017Zimbabwe       Africa_2018Length: 1704, dtype: object
Note: In the example above, we want to combine two columns as strings. To do this, we have to interpret data['Year'] as a string. We do that by using .astype(str) on the column. For the sake of brevity, we will not dive into types and type conversion in this article but discuss those topics in another article.
So far, all the calculations we have applied were to the entire set, a row, or a column. However — and this is where it gets exciting — we can also group our data and calculate metrics for the individual groups.
So let’s say we want to know the highest Life Ladder value per country.
IN:data.groupby(['Country name'])['Life Ladder'].max()OUT:Country nameAfghanistan    4.758381Albania        5.867422Algeria        6.354898Angola         5.589001                 ...   Vietnam        5.767344Yemen          4.809259Zambia         5.260361Zimbabwe       4.955101Name: Life Ladder, Length: 165, dtype: float64
Let’s say we want per year the country with the highest Life Ladder.
IN:data.groupby(['Year'])['Life Ladder'].idxmax()OUT:Year2005    Denmark2006    Finland2007    Denmark2008    Denmark         ...   2015     Norway2016    Finland2017    Finland2018    FinlandName: Life Ladder, Length: 14, dtype: object
Or multi-level groups, let’s say we want per continent/year combination the country with the highest Life Ladder.
IN:data.groupby(['Year','Continent'])['Life Ladder'].idxmax()OUT:Year  Continent    2005  Africa                  Egypt      Asia             Saudi Arabia      Europe                Denmark      North America          Canada                           ...     2018  Europe                Finland      North America          Canada      Oceania           New Zealand      South America           ChileName: Life Ladder, Length: 83, dtype: object
Like before we can use many standard functions or custom functions (named or unnamed) to, for example, return a random country per group:
IN:def get_random_country(group):    return np.random.choice(group.index.values)# Named functiondata.groupby(['Year','Continent']).apply(get_random_country)# Unnamed functiondata.groupby(['Year','Continent']).apply(  lambda group: np.random.choice(group.index.values))OUT:Year  Continent    2005  Africa                  Egypt      Asia                   Jordan      Europe                 France      North America          Mexico                           ...     2018  Europe           North Cyprus      North America       Nicaragua      Oceania             Australia      South America           ChileLength: 83, dtype: object
Note: groupby always returns ONE value per group. So unless you are grouping by a column that only contains unique values, the result will be a smaller (an aggregated) dataset.
Sometimes you don’t want only one value per group, but instead, want the value you calculated for the group for every row belonging to that group. You can do this the following way:
IN:data.groupby(['Country name'])['Life Ladder'].transform(sum)OUT:Country nameAfghanistan    40.760446Afghanistan    40.760446Afghanistan    40.760446Afghanistan    40.760446                 ...    Zimbabwe       52.387015Zimbabwe       52.387015Zimbabwe       52.387015Zimbabwe       52.387015Name: Life Ladder, Length: 1704, dtype: float64
Where we get the sum of all Life Ladder scores for a country. We could also do:
IN:data.groupby(['Country name'])['Life Ladder'].transform(np.median)OUT:Country nameAfghanistan    3.782938Afghanistan    3.782938Afghanistan    3.782938Afghanistan    3.782938                 ...   Zimbabwe       3.826268Zimbabwe       3.826268Zimbabwe       3.826268Zimbabwe       3.826268Name: Life Ladder, Length: 1704, dtype: float64
To get the median per country. We can then calculate the difference to the value of every single year like this (as transform preserves the index):
IN:data.groupby(['Country name'])['Life Ladder'].transform(np.median) \- data['Life Ladder']OUT:Country nameAfghanistan    0.059348Afghanistan   -0.618841Afghanistan   -0.975443Afghanistan   -0.048782                 ...   Zimbabwe       0.123077Zimbabwe       0.090868Zimbabwe       0.187968Zimbabwe       0.209789Name: Life Ladder, Length: 1704, dtype: float64
This article should give you quite some food for thought. Initially, I also wanted to include accessors, type manipulation and concatenating, merging and joining DataFrames, but given the length of the article, I moved those topics to part II of this series:
towardsdatascience.com
See you then, and keep exploring!
Ohh and if you enjoy reading stories like these and want to support me as a writer consider signing up to become a Medium member. It’s $5 a month, giving you unlimited access to stories on Medium. If you sign up using my link, I’ll even get some 🍩.