Python Map Reduce Filter Tutorial Introduction
Tags: Python, Mapreduce, Tutorial, Programming, Intro

Map, Filter And Reduce In Pure Python

The concepts of map, filter and reduce are a game changer. The usage of these methods goes way beyond Python and are an essential skill for the future.

Map, Filter and Reduce (Image by Author)

The Basics

Map, filter and reduce are functions that help you handle all kinds of collections. They are at the heart of modern technologies such as Spark and various other data manipulation and storage frameworks. But they can also very powerful helpers when working with vanilla Python.

Map

Map is a function that takes as an input a collection e.g. a list [‘bacon’,’toast’,’egg’], and a function e.g. upper(). Then it will move every element of the collection through this function and produce a new collection with the same count of elements. Let’s look at an example

map_obj = map(str.upper,['bacon','toast','egg'])

print(list(map_obj))

>>['BACON', 'TOAST', 'EGG']

What we did here is using the map(some_function, some_iterable) function combined with the upper function (this function capitalizes each character of a string). As we can see we produced for every element in the input list another element in the output list. We receive always the same amount of elements in the output as we will put into it! Here we send 3 in and received 3 out, this is why we call it an N to N function. Let’s look at how one can use it.

def count_letters(x):

return len(list(x)) map_obj = map(count_letters,['bacon','toast','egg'])

print(list(map_obj))

>>[6, 5, 3]

In this example we defined our own function count_letters(). The collection was passed through the function and in the output, we have the number of letters of each string! Let’s make this a little bit sexier using a lambda expression.

map_obj = map(lambda x:len(list(x)),['bacon','toast','egg'])

print(list(map_obj))

>>[6, 5, 3]

A lambda expression is basically just a shorthand notation for defining a function. If you are not familiar with them you can check out how they work here. However, it should be fairly easy to understand how they work from the following examples.

Filter

In contrast to Map, which is an N to N function. Filter is a N to M function where N≥M. What this means is that it reduces the number of elements in the collection. In other words, it filters them! As with map the notation goes filter(some_function, some_collection). Let’s check this out with an example.

def has_the_letter_a_in_it(x):

return 'a' in x # Let's first check out what happens with map

map_obj = map(has_the_letter_a_in_it,['bacon','toast','egg'])

print(list(map_obj))

>>[True,True,False] # What happens with filter?

map_obj = filter(has_the_letter_a_in_it,['bacon','toast','egg'])

print(list(map_obj))

>>['bacon', 'toast']

As we can see it reduces the number of elements in the list. It does so by calculating the return value for the function has_the_letter_a_in_it() and only returns the values for which the expression returns True.

Again this looks much sexier using our all-time favorite lambda!

map_obj = filter(lambda x: 'a' in x, ['bacon','toast','egg'])

print(list(map_obj))

>>['bacon', 'toast']

Reduce

Let’s meet the final enemy and probably the most complicated of the 3. But no worries, it is actually quite simple. It is an N to 1 relation, meaning no matter how much data we pour into it we will get one result out of it. The way it does this is by applying a chain of the function we are going to pass it. Out of the 3, it is the only one we have to import from the functools. In contrast to the other two it can most often be found using three arguments reduce(some_function, some_collection, some_starting_value), the starting value is optional but it is usually a good idea to provide one. Let’s have a look.

from functools import reduce map_obj = reduce(lambda x,y: x+" loves "+y, ['bacon','toast','egg'],"Everyone")

print(map_obj)

>>'Everyone loves bacon loves toast loves egg'

As we can see we had to use a lambda function which takes two arguments at a time, namely x,y. Then it chains them through the list. Let’s visualize how it goes through the list

x=“Everyone”, y=” bacon”: return ”Everyone loves bacon“ x=”Everyone loves bacon“, y=”toast”: return ”Everyone loves bacon loves toast“ x=”Everyone loves bacon loves toast“, y=”egg” : return ”Everyone loves bacon loves toast loves eggs“

So we have our final element ”Everyone loves bacon loves toast loves eggs“. Those are the basic concepts to move with more ease through your processing pipeline. One honorable mention here is that you can not in every programming language assume that the reduce function will handle the element in order, e.g. in some languages it could be “‘Everyone loves egg loves toast loves bacon’”.

Combine

To make sure we understood the concepts let’s use them together and build a more complex example.

from functools import reduce vals = [0,1,2,3,4,5,6,7,8,9]

# Let's add 1 to each element >> [1,2,3,4,5,6,7,8,9,10]

map_obj = map(lambda x: x+1,vals)

# Let's only take the uneven ones >> [1, 3, 5, 7, 9]

map_obj = filter(lambda x: x%2 == 1,map_obj)

# Let's reduce them by summing them up, ((((0+1)+3)+5)+7)+9=25

map_obj = reduce(lambda x,y: x+y,map_obj,0)

print(map_obj)

>> 25

As we can see we can build pretty powerful things using the combination of the 3. Let’s move to one final example to illustrate what this might be used for in practice. To do so we load up a small subset of a dataset and will print the cities which are capitals and have more than 10 million inhabitants!

from functools import reduce #Let's define some data

data=[['Tokyo', 35676000.0, 'primary'], ['New York', 19354922.0, 'nan'], ['Mexico City', 19028000.0, 'primary'], ['Mumbai', 18978000.0, 'admin'], ['São Paulo', 18845000.0, 'admin'], ['Delhi', 15926000.0, 'admin'], ['Shanghai', 14987000.0, 'admin'], ['Kolkata', 14787000.0, 'admin'], ['Los Angeles', 12815475.0, 'nan'], ['Dhaka', 12797394.0, 'primary'], ['Buenos Aires', 12795000.0, 'primary'], ['Karachi', 12130000.0, 'admin'], ['Cairo', 11893000.0, 'primary'], ['Rio de Janeiro', 11748000.0, 'admin'], ['Ōsaka', 11294000.0, 'admin'], ['Beijing', 11106000.0, 'primary'], ['Manila', 11100000.0, 'primary'], ['Moscow', 10452000.0, 'primary'], ['Istanbul', 10061000.0, 'admin'], ['Paris', 9904000.0, 'primary']] map_obj = filter(lambda x: x[2]=='primary' and x[1]>10000000,data)

map_obj = map(lambda x: x[0], map_obj)

map_obj = reduce(lambda x,y: x+", "+y, map_obj, 'Cities:')

print(map_obj)

>> Cities:, Tokyo, Mexico City, Dhaka, Buenos Aires, Cairo, Beijing, Manila, Moscow

If you enjoyed this article, I would be excited to connect on Twitter or LinkedIn.

Make sure to check out my YouTube channel, where I will be publishing new videos every week.