An Introduction to Python Generator Functions
By: Mark A. Brennan
Tags: S3, Design Patterns, Python

Mechanics

Python generator functions are a powerful concept, but unlike the elaborate scaffolding of function decorators, they are achieved or expressed with a remarkably simple mechanism — the “yield” statement (with yield being a new keyword added to Python under PEP 255).

As a transitive verb, yield denotes production; as an intransitive verb, it denotes giving way or relinquishing. Both senses of the word come into play in Python’s generator functions.

Traditionally we think of functions producing results through their return statement when they return single values, multiple values in the form of lists or dictionaries, or user-defined objects. We think of the return statement as a function’s way of ending control and ceding both control and results back to its caller. After the return statement, the runtime environment (interpreter) pops the given function’s stack frame off of the call stack, and the given function’s “environment”, such as it is, ceases to exist (until the next time the function is invoked).

Python’s yield statement utterly changes this behavior. Let’s look at a remarkably simple and contrived example of a generator — with some additional code to demonstrate its use (code is taken from an iPython interpreter interactive session):

This function does nothing other than “yield” the value that is passed as an argument. However, simply invoking the function like a “normal” function does not produce a returned value. The generator function is instantiated, if you will, with an argument and saved in the variable g . Now, as the next() call on the generator object makes clear, the generator must be iterated on in order to produce a value. And, once it yields its (single) value, the generator is exhausted - subsequent invocations of next() cause a “StopIteration” exception to be thrown. If we had iterated over this generator function in a for loop, the underlying iteration mechanics embodied in the for would have handled the StopIteration exception gracefully.

Most Python texts introduce generators using a loop statement, similar to the following code:

But I’ve found this can cause confusion as to the flow and transfer of control. What is imperative to understand is that the generator does not yield any values until they are requested from the client, while iterating within the for loop. Implicitly within the for loop, Python is invoking next() on the iterator it has obtained from the generator object. That is, within the for loop, Python is implicitly doing this:

Of course, next() can be called explicitly on the generator function’s iterator, and it is helpful to manually force iteration on a generator function at a Python interpreter console.

The diagram below may also help explain these steps.

In this way, and as with closures, Python’s generator functions retain state across successive calls. Or, as PEP 255 puts it:

If a yield statement is encountered, the state of the function is frozen, and the value of expression_list is returned to .next()’s caller. By “frozen” we mean that all local state is retained, including the current bindings of local variables, the instruction pointer, and the internal evaluation stack: enough information is saved so that the next time .next() is invoked, the function can proceed exactly as if the yield statement were just another external call.

This kind of state retention coupled with a lazy production of values is hard to conceptualize with such a small and trivial example, so I have tried to make it more concrete by coding a generator function I think may be useful.