Coding in R: Nest and map your way to efficient code | by Rebecca O’Dwyer | Towards Data Science

I wrote my first line of R code in March 2019, but only discovered the power of nest and map very recently. Prior to this I wrote a lot of inefficient code (C&P, C&P, C&P...).
This is short walk through of how to use nest and map to find the mean population of 184 countries between 1960 and 2016 using the gapminder data set (which comes built in with dslabs). gapminder has health and income data for 184 countries for every year from 1960 to 2016. The data is presented in a tidy format, which means for each country there are 57 rows of data (one row for each year).
library(dslabglimpse(gapminder)
## Observations: 10,545## Variables: 9## $ country <fct> Albania, Algeria, Angola, Antigua and Barbuda, ...## $ year <int> 1960, 1960, 1960, 1960, 1960, 1960, 1960, 1960,...## $ infant_mortality <dbl> 115.40, 148.20, 208.00, NA, 59.87, NA, NA, 20.3...## $ life_expectancy <dbl> 62.87, 47.50, 35.98, 62.97, 65.39, 66.86, 65.66...## $ fertility <dbl> 6.19, 7.65, 7.32, 4.43, 3.11, 4.55, 4.82, 3.45,...## $ population <dbl> 1636054, 11124892, 5270844, 54681, 20619075, 18...## $ gdp <dbl> NA, 13828152297, NA, NA, 108322326649, NA, NA, ...## $ continent <fct> Europe, Africa, Africa, Americas, Americas, Asi...## $ region <fct> Southern Europe, Northern Africa, Middle Africa...
nest only works if proceeded by group_by. You must group your data before you can nest it. Grouping gapminder by country and then calling nest results in this:
gapminder_nest <- gapminder %>%   group_by(country) %>%   nest()head(gapminder_nest)## # A tibble: 6 x 2## # Groups:   country [185]##   country                       data##   <fct>               <list<df[,8]>>## 1 Albania                   [57 × 8]## 2 Algeria                   [57 × 8]## 3 Angola                    [57 × 8]## 4 Antigua and Barbuda       [57 × 8]## 5 Argentina                 [57 × 8]## 6 Armenia                   [57 × 8]
When I first used nest, I was terrified by this. I do not come from a coding or IT background and seeing <list<df[,8]>> was enough to make me delete the code and revert back to cutting and pasting to get the job done.
But it’s not actually scary. What you have inside this newly created column called data is the neatly arranged health and income data for each country. It can be accessed by calling gapminder_nest$data[[x]] where x is the row. So to call Australia (where I’m from), the code is gapminder_nest$data[[8]].
gapminder_nest$data[[8]]## # A tibble: 57 x 8##     year infant_mortality life_expectancy fertility population     gdp##    <int>            <dbl>           <dbl>     <dbl>      <dbl>   <dbl>##  1  1960             20.3            70.9      3.45   10292328 9.67e10##  2  1961             20              71.1      3.55   10494911 9.90e10##  3  1962             19.5            70.9      3.43   10691220 1.00e11##  4  1963             19.2            71.0      3.34   10892700 1.07e11##  5  1964             18.8            70.6      3.15   11114995 1.14e11##  6  1965             18.6            71.0      2.97   11368011 1.21e11##  7  1966             18.3            70.8      2.89   11657281 1.24e11##  8  1967             18.3            71.1      2.85   11975795 1.31e11##  9  1968             18.2            70.7      2.89   12305530 1.38e11## 10  1969             18.1            71.1      2.89   12621240 1.48e11## # ... with 47 more rows, and 2 more variables: continent <fct>, region <fct>
map allows you to iterate a function over all columns in a data frame. The syntax is: map(.x, .f) where .x is the data frame to be iterated over and .f is the function. The little wrinkle with nested data is that you have data frames inside your data frame. Inside the gapminder_nest$data column there are more columns of data.
So instead of using gapminder_nest as .x in the map call, we need to use gapminder_nest$data as .x. This means map will iterate over each country’s nested data frame.
Let’s find the mean population in each country between 1960 and 2016 using our nested data and map. Set .x as gapminder_nest$data. The function isn’t simply mean. If you just code that, your code will throw an error. Remember, there are more columns inside the data column. We want mean population, so the code for .f is ~mean(.x$population). In simple terms, this code says iterate over all the nested country data sets, calculating the mean of the population column for each country. Don’t forget the tilde ~ in the code for .f!
(In the code below I’ve set na.rm = T to avoid any errors for countries/years where the data is missing)
pop_mean <- map(.x = gapminder_nest$data, .f = ~mean(.x$population, na.rm = T))head(pop_mean)## [[1]]## [1] 2708629## ## [[2]]## [1] 24231378## ## [[3]]## [1] 11909433## ## [[4]]## [1] 71053.36## ## [[5]]## [1] 31638376## ## [[6]]## [1] 2925011
However, this output is pretty useless because we don’t know what the country names are. A much neater solution is to mutate the existing gapminder_nest and add a column called pop_mean.
gapminder_nest <- gapminder_nest %>%  mutate(pop_mean = map(.x = data, .f = ~mean(.x$population, na.rm = T)))head(gapminder_nest)## # A tibble: 6 x 3## # Groups:   country [185]##   country                       data pop_mean ##   <fct>               <list<df[,8]>> <list>   ## 1 Albania                   [57 × 8] <dbl [1]>## 2 Algeria                   [57 × 8] <dbl [1]>## 3 Angola                    [57 × 8] <dbl [1]>## 4 Antigua and Barbuda       [57 × 8] <dbl [1]>## 5 Argentina                 [57 × 8] <dbl [1]>## 6 Armenia                   [57 × 8] <dbl [1]>
But this still isn’t right. We can see the names of the countries, but not the mean population. This is because map always returns a list, so mean population is inside a nested list in gapminder_nest.
To see mean population, simply call unnest on the column you want to be visible, in this case pop_mean.
gapminder_nest %>%   unnest(pop_mean)## # A tibble: 185 x 3## # Groups:   country [185]##    country                       data  pop_mean##    <fct>               <list<df[,8]>>     <dbl>##  1 Albania                   [57 × 8]  2708629.##  2 Algeria                   [57 × 8] 24231378.##  3 Angola                    [57 × 8] 11909433.##  4 Antigua and Barbuda       [57 × 8]    71053.##  5 Argentina                 [57 × 8] 31638376.##  6 Armenia                   [57 × 8]  2925011.##  7 Aruba                     [57 × 8]    74148.##  8 Australia                 [57 × 8] 16601155.##  9 Austria                   [57 × 8]  7800180.## 10 Azerbaijan                [57 × 8]  6897604.## # ... with 175 more rows
Et voila! Just for fun, here’s a plot of the 10 smallest countries by mean population. (Scroll to end for code)
Combining nest and map creates highly efficient code. However, for newbies there are a few traps to be avoided:
You can’t nest without calling group_by first. If you don’t, the code won’t know what variable to base the nesting on.Your data hasn’t disappeared! To access anything inside your nested data frame, call nested_df$data[[x]] where x is the row you want to see.Columns from the nested data frame have their own columns. It’s Russian Dolls.map allows you to iterate over the columns of a data frame with a function. For a nested data frame, you have columns inside columns, so you can code .x = nested_df$data and .f = ~some_function(.x$nested_column_inside_data_column).Don’t forget the tilde ~ in your map function! Don’t worry why (that’s for another post), just do it.
You can’t nest without calling group_by first. If you don’t, the code won’t know what variable to base the nesting on.
Your data hasn’t disappeared! To access anything inside your nested data frame, call nested_df$data[[x]] where x is the row you want to see.
Columns from the nested data frame have their own columns. It’s Russian Dolls.
map allows you to iterate over the columns of a data frame with a function. For a nested data frame, you have columns inside columns, so you can code .x = nested_df$data and .f = ~some_function(.x$nested_column_inside_data_column).
Don’t forget the tilde ~ in your map function! Don’t worry why (that’s for another post), just do it.
gapminder_plot <- gapminder_nest %>%   unnest(pop_mean) %>%   select(country, pop_mean) %>%   ungroup() %>%   top_n(pop_mean, n = -10) %>%   mutate(pop_mean = pop_mean/10^3)gapminder_plot %>%   ggplot(aes(x = reorder(country, pop_mean), y = pop_mean)) +  geom_point(colour = "#FF6699", size = 5) +  geom_segment(aes(xend = country, yend = 0), colour = "#FF6699") +  geom_text(aes(label = round(pop_mean, 0)), hjust = -1) +  theme_minimal() +  labs(title = "Countries with smallest mean population from 1960 to 2016",       subtitle = "(thousands)",       x = "",       y = "") +  theme(legend.position = "none",        axis.text.x = element_blank(),        plot.title = element_text(size = 14, face = "bold"),        panel.grid.major.y = element_blank()) +  coord_flip() +  scale_y_continuous(limits = c(0, 150))