How to Code Memory Efficient Functions with Python Generators
By: Erdem Isbilen
Tags: Python3, Python Generators, Programming, Python, Coding

Let’s write a Python3 code that contains simple examples of implementing generators:

# Simple Generator

def my_simple_generator():

k = 0

yield kk = 1

yield kk = 2

yield kmy_numbers = my_simple_generator()print(next(my_numbers))

print(next(my_numbers))

print(next(my_numbers))

# Below call raises StopIteration as generator is fully iterated

# print(next(my_numbers))# Defining Generators with Loop

def my_generator_with_loop(my_str):

length = len(my_str)

for k in range(length):

yield my_str[k]my_text = my_generator_with_loop("Coding")for char in my_text:

print(char)# Defining Generators with Expressions

my_generator_expression = (number**2 for number in range(4))

print (sum(my_generator_expression))# Defining Generator Pipeline

my_generator_01 = (number**2 for number in range(40))

my_generator_02 = (number-5 for number in my_generator_01)print(sum(my_generator_02))

Defining Generator Functions

# Simple Generator

def my_simple_generator():

k = 0

yield kk = 1

yield kk = 2

yield kmy_numbers = my_simple_generator()print(next(my_numbers))

print(next(my_numbers))

print(next(my_numbers))

# Below call raises StopIteration as generator is fully iterated

# print(next(my_numbers))#Output

0

1

2

Generator functions are defined as normal functions. The only difference in the code structure is the yield expression used instead of return.

We have three yield expression in our example above. This means that we can iterate the generator maximum of 3 times. If we iterate it more than 3 times, the generator will raise a StopIteration exception.

Once we define the generator function, we can assign it to a variable. With the assignment, my_numbers variable points to the generator object. Note that we are not iterating the generator with the assignment. To manually iterate the generator, we can use next(my_number) notation.

In every iteration step, the value of k, and the inner state of the my_simple_generator are remembered by Python.

Photo by Sean O. on Unsplash

Defining Generators with a Loop

To simplify the generator definition we can define and iterate the generators with the help of for loops.

# Defining Generators with Loop

def my_generator_with_loop(my_str):

length = len(my_str)

for k in range(length):

yield my_str[k]my_text = my_generator_with_loop("Coding")for char in my_text:

print(char)#Output

C

o

d

i

n

g

Defining Generators With Generator Expressions

We can further simplify the definition of generators with generator expressions. The syntax for creating a generator expression is very similar to the one used for list comprehension. Round brackets are used to create generator expressions, instead of square brackets used in the list comprehension.

As list comprehension returns and stores the whole content in the memory, generator expressions just returns and stores the portion of data when it is demanded.

# Defining Generators with Expressions

my_generator_expression = (number**2 for number in range(4))

print (sum(my_generator_expression))#Output

14

Defining Generator Pipelines

To create complex pipelines, multiple generators can be stacked easily as shown in the below example.

# Defining Generator Pipeline

my_generator_01 = (number**2 for number in range(40))

my_generator_02 = (number-5 for number in my_generator_01)print(sum(my_generator_02))#Output

20340

Photo by Ethan Robertson on Unsplash

Key Takeaways

Generators are memory-friendly as they return and store the portion of data only when it is demanded.

Generators simplify the iterator definition process

We can define generators with generators expressions or generator functions.

We can develop memory-efficient data pipelines by using multiple generators.

Conclusion

In this post, I explained the basics of generators in Python.

The code in this post is available in my GitHub repository.

I hope you found this post useful.

Thank you for reading!