How do Python Generators Work? | Towards Data Science

In this article, I’ll attempt to provide an in-depth understanding of Python generators, and how they operate from the inside. It will include the how and why. Sections will be self-contained, and you are welcome to skip at your will. Without further ado, let us begin our journey. ⛵️
Added since Python 2.2, Generators are unique functions that can be paused, resumed and iterated over. Let’s take a look over an example:
By defining a function that contains the yield keyword, the function is marked as a generator. We then run the generator in the following order:
Initialize the generator and put it in g. At this point, we have a new generator object that didn’t run yet.Advance the generator by calling next() on it. This causes the generator to advance to the first yield keyword, printing “Hello” on the way.Advance the generator again. It prints “Goodbye”, and since it reached the end of the function, it raises a StopIteration exception and finishes.
Initialize the generator and put it in g. At this point, we have a new generator object that didn’t run yet.
Advance the generator by calling next() on it. This causes the generator to advance to the first yield keyword, printing “Hello” on the way.
Advance the generator again. It prints “Goodbye”, and since it reached the end of the function, it raises a StopIteration exception and finishes.
In order to understand these three simple steps, we first need to start from the very basics of how a function in Python runs.
Running a function involves two main steps: Initialization, and Evaluation.
Under the hood, CPython runs a function inside a frame object.
A frame object, sometimes called a stack-frame, consists of the following fields:
code object, which includes the function’s actual compiled code.
free stack space, called value-stack, for running the interpreter.
execution pointer for the current line of code being executed.
other fields like mappings of local variables.
A list of all frame fields can be seen in the code while a shorter list can be seen in the documentation.
When we initialize a function, a frame is created. In pseudo-code:
f = Frame()
The arguments we passed to the function are then assigned inside the frame:
f.f_locals["arg1"] = arg1f.f_locals["arg2"] = arg2
The execution point of the frame, is set to the first line of code in the function:
f.lasti = 0  # Last Instructionf.lineno = 0 # Function line number on file
At this point our function starts running.
Almost the entire Python interpreter can be summarized into one C-level function: PyEval_EvalFrameEx.
This function is the interpreter loop. Consisting of 3k lines of code, its job is to evaluate a frame, or in other words, run it. Every function in Python, every bytecode or opcode that the interpreter runs, goes through that function.
During “PyEval”, Python operates on the frame’s unique value-stack. This bit of information will be crucial later on.
In summary, when you enter a function, and run it, the interpreter creates a frame, and enters the interpreter loop.
A site called PythonTutor has a very nice interactive example of this mechanism.
Much like functions, running a generator involves the initialization step, and the evaluation step.
At this step we are creating the generator object. The generator object consists of the frame object and the code object. If you’re wondering, yes, the frame object also contains the code object inside it. In pseudo-code:
g = Generator()f = Frame() # Initialized as beforeg.gi_frame = fg.gi_code = f.f_code = Code()  # The compiled function
At this point, our generator g is ready to run.
Every time we call next(g), the frame is evaluated using the same PyEval_EvalFrameEx.
The difference is, when running a generator, we can reach the yield keyword, at which point the frame evaluation stops, and the generator “pauses”.
A pointer in the frame is saved, marking our current stack position, in order to know exactly what was the generator state upon stopping.
It is important to remember, that unlike functions, when the evaluation stops the frame is not destroyed as it’s still attached to the generator object. This will allow us to load it later on.
PyEval returns and our calling function resumes.
When we advance the generator once more, the frame is loaded from gi_frame, and PyEval is called again on the same frame. It checks where we stopped, goes to our execution pointer, and uses the value-stack stored on the frame.
Since we’re using the same stack, and the frame also contains all of the local variables, we can continue from the exact same spot, as if the function never stopped.
When we reach the end of the generator, or return statement, StopIteration gets thrown, and the frame is deleted. gi_frame is then set to None.
Now that we know how a generator internally pauses and resumes, let’s dive into a few more features that make generators truly remarkable.
Generators can yield values, allowing them to be iterated over and return results in a lazy manner. For example:
Internally, yielding values is much like getting the result of a function. When PyEval reaches the YIELD_VALUE opcode, it pops the top value of the stack, and returns it. Quite simple, isn’t it?
Generators can use the return keyword to return a result:
As you can see, the return statement sets the StopIteration exception. Exceptions can have arguments, and here, the first argument sent to the StopIteration exception is the return value.
The return value can be retrieved by catching the exception, and getting its first argument:
We can also communicate with generators using .send() and .throw():
As you can see, we sent numbers to the generator using .send(), and they were the return value of the yield keyword.
Internally, .send() works by putting the value at the top of the generator’s stack. It then evaluates the frame, and pops the top value of the stack, putting it in our local variable.
Similarly, .throw() works by sending a throwflag to PyEval stating an exception occurred. It then handles the exception normally. If the generator did not catch it, the exception is propagated outwards, just like a normal function.
Generators can be nested (or “delegated”) using the yield from keyword:
As you can see, using yield from creates a way to communicate between the innermost generator, all the way out.
Internally, it works by using the gi_yieldfrom field of the generator object. It points to the inner generator, and when you use .send() it will go all the way in.
When the inner generator returns, it goes up the chain, and sets yield from's return value accordingly.
In this article, I provided some insight into the inner workings of the CPython generator implementation. While the article is not beginner material, it only scratches the surface of this topic, and there is much more to understanding how the interpreter works. The full implementation is of course open source, so you are welcome to explore at your will. Enjoy 😉
Some rights reserved