How to Create Decorators in Python That You Can Actually Use | Towards Data Science

One of the magical things you can do with Python and many other languages is decorating functions. Decorators can modify the function’s inputs, its output, and the very behavior of the function itself. And the best part is you can do all of those with only one line of code and without modifying the function syntax at all!
To learn how decorators work and how you can create one for yourself, you need to know some of the concepts of Python.
Therefore, before we get to decorators, we are gonna go knee-deep into learning some of the internals of Python such as scope and closures. If you are familiar with these concepts, feel free to skip them and go to section 5 where all the fun starts!
IntroductionFunctions are objectsScopeClosuresDecoratorsReal-world Examples With DecoratorsDecorators That Take ArgumentsPreserving the Decorated Function’s MetadataConclusion
Introduction
Functions are objects
Scope
Closures
Decorators
Real-world Examples With Decorators
Decorators That Take Arguments
Preserving the Decorated Function’s Metadata
Conclusion
One of the many things you will love about Python is its ability to represent anything as objects and functions are no exception. For people who are first reading this, passing a function as an argument to another may seem strange but it is completely legal to do so:
Important note to be useful in later sections: Using the function with parentheses, my_func(), is called 'calling' the function while writing without, my_func, is called referencing. As you have seen, print(my_func) prints the function's index in memory.
As objects go, functions are absolutely the same as:
str
int, float
pandas.DataFrame
list, tuple, dict
modules: os, datatime, numpy
You may assign functions to a new variable and use it to call the function:
>>> new_func = my_func>>> new_func()Printing the function's argument
Now this variable also contains the function’s attributes:
You can also store each function in other objects such as lists, dictionaries, and call them:
Consider this conversation between Bob and Job:
Bob: ‘Jon, why did not you come to the lesson yesterday?’
Jon: ‘I had a flu...’
Not the best of stories but when Bob asks the reason for Jon’s absence in yesterday’s class, we know he is referring to the Jon standing next to him not some random Jon in another country. As humans, it is not difficult to notice this but programming languages use something called scope to tell which name we are referring to in our programs.
In Python, names can be variables, function names, module names, etc.
Consider these two variables:
>>> a = 24>>> b = 42>>> print(a)24
Here, print had no trouble to tell that we are referring to the a we just defined. Now consider this:
>>> def foo():...     a = 100...     print(a)
What do you think will happen if we run foo? Will it print 24 or 100?
>>> foo()100
How did Python differentiate between the a we defined in the beginning or in the function? This is where scope gets interesting, because we are introducing layers of scope:
The above image shows the scope for this little script:
The global scope is the overall scope of your script/program. Variables, functions, etc. with the same indentation level as a and b defined in the beginning will be in the global scope. For example, foo function is in the global scope but its variable a is in the scope that is local to foo.
In one global scope, there can be many local scopes. For example, each temporary variables in for loops and list comprehensions, return values of context managers will be local inside their code block and cannot be accessed from the global scope.
So, a rule of thumb is that Python interpreter will not be able to access a name defined in a smaller scope than the current one.
There is also a bigger level of scope outside global:
The built-in scope contains all the modules and packages you installed with Python, pip or conda.
Now, let’s explore another case. In our foo function, we want to modify the value of global a. We want it to be a string but if we write a = 'some text' inside the foo, Python will just create a new variable without modifying the global a.
Python provides us with a keyword that lets us specify we are referring to names in the global scope:
Writing global <name> will let us modify the values of names in the global scope.
BTW, bad news, I left out one level of scope in the above image. Between global and local, there is one level we did not cover:
nonlocal scope comes into play when we have, for example, nested functions:
In the nested function outer, we first create a variable called my_var and assign it to the string Python. Then we decide to create a new inner function and want to assign my_var a new value, Data Science and print it. But if we run it, we see that my_var is still assigned to ‘Python’. We cannot use global keyword since my_var is not in the global scope.
For such cases, you can use nonlocal keyword which gives access to all the names in the scope of the outer function (nonlocal) but not the global:
In conclusion, scope tells the Python interpreter where to look for names in our program. There can be four levels of scope in a single script/program:
Built-in: all the package names installed with Python, pip and conda
Global: general scope, all names that have no indentation in the script
Local: contains local variables in code blocks such as functions, loops, list comprehensions, etc.
Nonlocal: an extra level of scope between global and local in the case of nested functions
Before I explain how decorators work, we need to talk about closures too. Let’s start with an example:
We create a nested function bar inside foo and return it. bar tries to print the value of x:
When we write var = foo(), we are assigning the bar function to var. Now var can be used to call bar. When we call it, it prints out 42.
>>> var()42
But wait a minute, how does var know anything about x? x is defined in foo's scope not bar's. You would think that x would be accessible outside the scope of foo. That's where closures come in.
Closure is a built-in memory of a function that contains all the nonlocal names (in a tuple) the function needs to run!
So, when foo returned bar, it attached all the nonlocal variables bar needs to run outside of the foo's scope. The closure of a function can be accessed with .__closure__ attribute:
Once you access the closure of a function as a tuple, it will contain elements called cells with the value of a single nonlocal argument. There can be as many cells inside closure as the function needs:
In this example, the variables x, y, z are nonlocal variables to child so they get added to the function's closure. Any other names such as value and outside are not in the closure because they are not in the nonlocal scope.
Now, consider this trickier example:
We create a parent function which takes a single argument and a nested function child which prints whatever value passed to parent. We call parent with var ('dummy') and assign the result to func. If we call it:
>>> func()dummy
As expected, it prints out ‘dummy’. Now let’s delete var and call func again:
>>> # Delete 'var'>>> del var>>> # call func again>>> func()dummy
It still prints out ‘dummy’. Why?
You guessed it, it got added to the closure! So, when a value from outer levels of scope gets added to closure, it will stay there unchanged even if we delete the original value!
>>> func.__closure__[0].cell_contents‘dummy’
If we did not delete var and changed its value, the closure would still contain its old value:
This concept is going to be important when we talk about decorators in the next section.
Let’s go over some of the concepts to make sure you understand:
The closure is an internal memory of a nested function. It contains all the nonlocal variables stored in a tuple which are essential for the function to run.
Once a value is stored in a closure, it can be accessed but cannot be overridden if the original value gets deleted or modified
A nested function is a function defined in another and follows this general pattern:
>>> def parent(arg):    ...     def child():...        print(arg)...    ...    return child
Decorators are functions that modify another function. They can change the function’s inputs, its output or even its behavior.
You may have seen decorators when you were creating custom context managers or when you were first introduced to Flask (remember @app.route?)
Below, we created a function that squares whatever argument is passed and we are decorating it with add_one. add_one adds 1 to the argument of the passed function:
To use a function as a decorator, just put @ symbol followed by the decorating function's name right above the function definition. When we passed 5 to the decorated square function, instead of returning 25, it returns 36 because add_one takes the argument of square, which is 5, and adds one to it and inserts back into our function:
>>> square(10)121
In this section, we will build add_one together.
First, let’s start with add_one that only returns whatever function passed to it:
For our decorator to return a modified function, it is usually helpful to define a nested function to return:
Our decorator is still doing nothing. Inside add_one we defined a nested child function. child only takes one argument and calls whatever function passed to add_one. Then, add_one returns child.
In this case of nested child function, we are assuming func passed to add_one takes exactly the same number of arguments as child.
Now, we can make all the magic happen inside the child function. Instead of simply calling the func, we want to modify its arguments by adding 1 to them:
Notice func(a + 1)? It is calling whatever passed to add_one with 1 added to the argument. This time, instead of creating a new variable to store child, we will override square:
>>> square = add_one(square)>>> square(5)36
Now it is returning 36 instead of 25 when we pass 5.
How can it use square function even when we override it? Good thing we learned closures because the old square is now inside the closure of child:
>>> square.__closure__[0].cell_contents<function __main__.square(a)>
At this point, our add_one function is ready to be used as a decorator. We can just put @add_one right above the definition of square and see the magic happen:
I think it would be a shame if I did not show you how to create a timer decorator:
This time, notice how we are using *args and **kwargs. They are used when we don't know the exact number of positional and keyword arguments in the function which is perfect in this case since we may use timer on any kind of function.
Now, you can use this decorator before any function to find out how long it runs. No repeated code!
The next very useful decorator would be a caching decorator. Caching decorators are great for computation-heavy functions which may be called with the same arguments many times. Caching the results of each function call in a closure will let us immediately return the result if the decorated function gets called with known values:
In the main, cache function, we want to create a dictionary which stores all arguments in tuples as keys and their results. The caching dictionary would look like this:
cache = {    (arg1, arg2, arg3): func(arg1, arg2, arg3)}
We can use tuples of arguments as keys because tuples are immutable objects.
Now, let’s see what happens if we decorate our sleeping function with both cache and timer:
First, let’s try to sleep for 10 seconds:
>>> sleep(10)sleep took 10.0001 seconds to run!
As expected, it took 10 seconds to run. Now, what do you think will happen if we run sleep with 10 as an argument again:
>>> sleep(10)sleep took 0.0 seconds to run!
It took 0 seconds! Our caching decorator works!
So far, our knowledge of decorators is pretty solid. However, the real power of decorators comes when you enable them to take arguments.
Consider this decorator which checks if the function’s result is of type str:
We call it on a dummy function to check that it works:
It is working. However, wouldn’t be cool if we had a way to check the function’s return type for any data type? Here, check this out:
With this type of decorator, you could write data type checks for all your functions. Let’s build it together from scratch.
First, let’s just create a simple decorator that calls whatever function passed to it:
How do we tweak this piece of code so that it also accepts a custom data type and performs a check for func's results? We cannot add an extra argument to decorator because decorators should only take a function as an argument.
The way to get around this problem is to define an even bigger parent function that returns a decorator. That way, we can pass any argument to the parent function which, in turn, can be used in the decorator:
Note how we just wrapped our decorator in a bigger parent function? All it does is take a data type as an argument and pass it to our decorator and return it. In wrapper, we wrote type(result) == dtype which evaluates to True or False whether data types match or not. Now you can use this function to perform type checks for any function:
Up until this point, we never checked one thing — is the decorated function preserved in all ways? For example, let’s go back to our sleep function which was decorated with timer:
Let’s call it and check its metadata:
We check 3 metadata attributes of the function. The first two returned None but they should have returned something. I mean, sleep had a long docstring and a default argument which was equal to 5. Where did they go? We got the answer when we called __name__ and got wrapper for the function name.
If we examine the definition of timer:
We can see that we are not actually returning the passed function but returning inside wrapper. Obviously wrapper does not have a docstring or any default arguments, that was the reason we got None above.
To solve this problem, Python provides us with a helpful function from functools module:
Using wraps on the wrapper function lets us keep all the metadata attached to func. Notice how we are passing func to wraps above the function definition.
If we use this modified version of timer, we will see that it works as expected:
Using wraps(func) is a good practice for writing decorators, so go and add it to all the decorators we defined today!
After reading this post, you have a pretty strong knowledge of creating decorators. More importantly, you know how they work and the way they work.
As a final point, I suggest using decorators whenever you have repeated code that performs similar tasks on your functions. Decorators can be another step to making your code DRY (Don’t Repeat Yourself).
Read more articles related to the topic: