Function Timeouts using Wrapt Timeout Decorator | Towards Data Science

In this short guide, I will show how to use a handy timeout decorator for Python function to limit the execution times. This guide is based on an example that utilizes Python’s multiprocessing package.
Lately, I’ve been building a script using Beautiful Soup that scrapes similarly structured websites and identifies its chapters. These were 10-K annual reports on the SEC’s website which had to be split according to their 21 sections. To make the script as robust as possible, I’ve used the simple, yet effective, python-constraint module to make sure that the sections were identified in the correct order. This approach worked perfectly in most cases and took less than a second to execute. Unfortunately, for some cases, the constraint function caused the script to run for days until it went through all iterations. By analyzing the execution times of the constraint function, I came to the following conclusion:
Execution time = < 1 second, if there is a solution for the constraint function.
Execution time = hours or days, if there is no solution for the constraint function.
A fix for this issue was simple thanks to the wrapt_timeout_decorator module. The great thing about this module is that it’s easy to use, works great with Python’s multiprocessing module, and has no problem running on Windows.
You can install the wrapt_timeout_decorator module from PyPI using pip.
pip install wrapt_timeout_decorator
If you’re using Conda, I suggest downloading the source files from GitHub (https://github.com/bitranox/wrapt_timeout_decorator) and installing the module manually by running
python setup.py install
within your active environment. You will need Python 3.5 or later for the module to work.
The following example is intended for Windows users. Linux users will have it easier using the wrapt_timeout_decorator because it’s not necessary to put the timeout function in a separate module. For more information on this, see the official documentation here. I haven’t tried this module in a Jupyter notebook, but I assume it will require a workaround similar to the multiprocessing package.
The example below shows how to apply a timeout on a function that is parallelized on multiple threads using the multiprocessing package.
Here is our main program:
# main_program.pyfrom multiprocessing import Poolimport my_functionmy_list = [1,2,3,4,4,3,2,1]if __name__ == ‘__main__’:    num_processors = 4    p = Pool(processes = num_processors)    p.map_async(my_function.sleep_function, my_list).get()
And here is the function my_function which is parallelized across 4 threads:
# my_function.pyfrom time import sleepdef sleep_function(num):    sleep(num)    print(f'The program slept for {num} seconds.')
The output of this program will be:
The program slept for 1 seconds.The program slept for 2 seconds.The program slept for 3 seconds.The program slept for 4 seconds.The program slept for 4 seconds.The program slept for 3 seconds.The program slept for 2 seconds.The program slept for 1 seconds.
The first for 4 lines will appear 1, 2, 3, and 4 seconds after executing the program respectively. Because we’re using the map_async method, the remaining 4 lines will appear all at once after another 4 seconds have passed. The first thread waits 1 second and then takes the next available element from my_list, which is 4, and waits 4 more seconds (in total 5 seconds). The second thread waits 2 seconds, then takes the next available element from my_list, which is 3, and waits 3 more seconds (again 5 seconds in total) and so on.
Imagine we want to limit the time for which the sleep_function is allowed to run. Let’s say we set this timeout to 2.5 seconds. To do this, we have to make three changes to our program.
Apply a timeout decorator to the sleep_function.Carve out the timed-out sleep_function into a separate module.Change the my_function.py to execute this carved-out module.
Apply a timeout decorator to the sleep_function.
Carve out the timed-out sleep_function into a separate module.
Change the my_function.py to execute this carved-out module.
Here is how the carved-out module of the timed-out function will look like:
# my_timeout_function.pyfrom time import sleepfrom wrapt_timeout_decorator import *@timeout(2.5)def timeout_sleep_function(num):    sleep(num)    print(f'The program slept for {num} seconds.')
Using the @timeout decorator, we’ve set the limit for how long the function can run to 2.5 seconds. Apart from that, the code of the function remains unchanged.
Now we have to go back to our my_function.py module and call the newly created timeout_sleep_function from there:
# my_function.pyimport my_timeout_functiondef sleep_function(num):    try:        my_timeout_function.timeout_sleep_function(num)    except:        print('Timeout of 2.5 seconds reached!')
Because we have set a limit of 2.5 seconds for the timeout_sleep_function, the output of our main_program.py will now be:
The program slept for 1 seconds.The program slept for 2 seconds.Timeout of 2.5 seconds reached!Timeout of 2.5 seconds reached!The program slept for 1 seconds.Timeout of 2.5 seconds reached!The program slept for 2 seconds.Timeout of 2.5 seconds reached!
The timeout wrapper has changed the structure of our code as follows:
Main program main_program.py. This initializes the multiprocessing and calls the function sleep_function from the “in-between” module my_function.py. No actual changes to the code were made.The “in-between” module my_function.py. This calls the timed-out function timeout_sleep_function from the module my_timeout_function.py. Having this “in-between” module is necessary for Windows users.The module my_timeout_function.py that has the actual timed-out function timeout_sleep_function.
Main program main_program.py. This initializes the multiprocessing and calls the function sleep_function from the “in-between” module my_function.py. No actual changes to the code were made.
The “in-between” module my_function.py. This calls the timed-out function timeout_sleep_function from the module my_timeout_function.py. Having this “in-between” module is necessary for Windows users.
The module my_timeout_function.py that has the actual timed-out function timeout_sleep_function.
As you can see, using the wrapt_timeout_decorator to limit execution times of functions is relatively simple. I was able to add a timeout to my 10-K annual report section separator quickly without a lot of changes to the original code. This small addition, however, caused a major reduction in execution times.
Here is a final wrap-up of the example using the wrapt_timeout_decorator module:
#----------------------------# main_program.pyfrom multiprocessing import Poolimport my_function_timeoutmy_list = [1,2,3,4,4,3,2,1]if __name__ ==  '__main__':    num_processors = 4    p = Pool(processes = num_processors)    p.map_async(my_function.sleep_function, my_list).get()#----------------------------# my_function.pyimport my_timeout_functiondef sleep_function(num):    try:        my_timeout_function.timeout_sleep_function(num)    except:        print('Timeout of 2 seconds reached!')#----------------------------# my_timeout_function.pyfrom time import sleepfrom wrapt_timeout_decorator import *@timeout(2.5)def timeout_sleep_function(num):    sleep(num)    print(f'The program slept for {num} seconds.')
I hope this little guide will help you apply timeouts to functions in Python.