How to Code Memory Efficient Functions with Python Generators | by Erdem Isbilen | Towards Data Science

Generators are special functions that return a lazy iterator which we can iterate over to handle one unit of data at a time. As lazy iterators do not store the whole content of data in the memory, they are commonly used to work with data streams and large datasets.
Generators in Python are very similar to normal functions with some characteristic differences listed below;
Generator functions have yield expression, instead of return used in normal functions.
Both yield and return statements return a value from a function. While the return statement ends the function completely, yield statement suspends the function by keeping all its state in the memory for later use.
When the generator function yields, the function is not terminated. Instead, yield expression pauses the function and gives control over to the caller.
After fully iterated, generators terminate and raise stopIteration exception.
This post will introduce you to the basics of generators in Python.
Let’s write a Python3 code that contains simple examples of implementing generators:
# Simple Generatordef my_simple_generator(): k = 0 yield kk = 1 yield kk = 2 yield kmy_numbers = my_simple_generator()print(next(my_numbers))print(next(my_numbers))print(next(my_numbers))# Below call raises StopIteration as generator is fully iterated# print(next(my_numbers))# Defining Generators with Loopdef my_generator_with_loop(my_str): length = len(my_str) for k in range(length):  yield my_str[k]my_text = my_generator_with_loop("Coding")for char in my_text: print(char)# Defining Generators with Expressionsmy_generator_expression = (number**2 for number in range(4))print (sum(my_generator_expression))# Defining Generator Pipelinemy_generator_01 = (number**2 for number in range(40))my_generator_02 = (number-5 for number in my_generator_01)print(sum(my_generator_02))
# Simple Generatordef my_simple_generator(): k = 0 yield kk = 1 yield kk = 2 yield kmy_numbers = my_simple_generator()print(next(my_numbers))print(next(my_numbers))print(next(my_numbers))# Below call raises StopIteration as generator is fully iterated# print(next(my_numbers))#Output012
Generator functions are defined as normal functions. The only difference in the code structure is the yield expression used instead of return.
We have three yield expression in our example above. This means that we can iterate the generator maximum of 3 times. If we iterate it more than 3 times, the generator will raise a StopIteration exception.
Once we define the generator function, we can assign it to a variable. With the assignment, my_numbers variable points to the generator object. Note that we are not iterating the generator with the assignment. To manually iterate the generator, we can use next(my_number) notation.
In every iteration step, the value of k, and the inner state of the my_simple_generator are remembered by Python.
To simplify the generator definition we can define and iterate the generators with the help of for loops.
# Defining Generators with Loopdef my_generator_with_loop(my_str): length = len(my_str) for k in range(length):  yield my_str[k]my_text = my_generator_with_loop("Coding")for char in my_text: print(char)#OutputCoding
We can further simplify the definition of generators with generator expressions. The syntax for creating a generator expression is very similar to the one used for list comprehension. Round brackets are used to create generator expressions, instead of square brackets used in the list comprehension.
As list comprehension returns and stores the whole content in the memory, generator expressions just returns and stores the portion of data when it is demanded.
# Defining Generators with Expressionsmy_generator_expression = (number**2 for number in range(4))print (sum(my_generator_expression))#Output14
To create complex pipelines, multiple generators can be stacked easily as shown in the below example.
# Defining Generator Pipelinemy_generator_01 = (number**2 for number in range(40))my_generator_02 = (number-5 for number in my_generator_01)print(sum(my_generator_02))#Output20340
Generators are memory-friendly as they return and store the portion of data only when it is demanded.
Generators simplify the iterator definition process
We can define generators with generators expressions or generator functions.
We can develop memory-efficient data pipelines by using multiple generators.
In this post, I explained the basics of generators in Python.
The code in this post is available in my GitHub repository.
I hope you found this post useful.
Thank you for reading!