Package: perftester
perftester: Lightweight performance testing of Python functionsInstallationInstall usingpip:pipinstallperftesterThe package has three external dependencies:memory_profiler(repo),easycheck(repo), androunder(repo).perftesteris still under heavy testing. If you find anything that does not work as intended, please let me know via nyggus<at>gmail.com.Pre-introduction: TL;DRAt the most basic level, usingperftesteris simple. It offers you two functions for benchmarking (one for execution time and one for memory), and two functions for performance testing (likewise). Read below for a very short introduction of them. If you want to learn more, however, do not stop there, but read on.BenchmarkingYou havetime_benchmark()andmemory_benchmark()functions:importperftesterasptdeffoo(x,n):return[x]*npt.time_benchmark(foo,x=129,n=100)and this will print the results of the time benchmark, with raw results similar to those thattimeit.repeat()returns, but unlike it,pt.time_benchmark()returns mean raw time per function run, not overall; in additional, you will see some summaries of the results.The above call did actually runtimeit.repeat()function, with the default configuration ofNumber=100_000andRepeat=5. If you want to change any of these, you can use argumentsNumberandRepeat, correspondigly:pt.time_benchmark(foo,x=129,n=100,Number=1000)pt.time_benchmark(foo,x=129,n=100,Repeat=2)pt.time_benchmark(foo,x=129,n=100,Number=1000,Repeat=2)These calls do not change the default settings so you use the arguments' values on the fly. Later you will learn how to change the default settings and the settings for a particular function.Some of you may wonder why theNumberandRepeatarguments violate what we can call the Pythonic style, by using a capital first letter for function arguments. The reason is simple: I wanted to minimize a risk of conflicts that would happen when benchmarking (or testing) a function with any of the argumentsNumberorRepeat(or both). A chance that a Python function will have aNumberor aRepeatargument is rather small. If that happens, however, you can usefunctools.partial()to overcome the problem:fromfunctoolsimportpartialdefbar(Number,Repeat):return[Number]*Repeatbar_p=partial(bar,Number=129,Repeat=100)pt.time_benchmark(bar_p,Number=100,Repeat=2)Benchmarking RAM usage is similar:pt.memory_usage_benchmark(foo,x=129,n=100)It uses thememory_profiler.memory_usage()function, which runs the function just once to measure its memory use. Almost always, there is no need to repeat it, unless there is great randomness in memory usage by the function. If you have good reasons to change this behavior (e.g., in the case of such randomness), you can request several calls of the function, using theRepeatargument:pt.memory_usage_benchmark(foo,x=129,n=100,Repeat=100)You can learn more in the detailed description of the package below.TestingThe API ofperftestertestinf functions is similar to that of benchmarking functions, the only difference being limits you need to provide. You can determine those limits using the above benchmark functions. Here are examples of several performance tests usingperftester:>>>importperftesteraspt>>>deffoo(x,n):return[x]*n# A raw test>>>pt.time_test(foo,raw_limit=9.e-07,x=129,n=100)# A relative test>>>pt.time_test(foo,relative_limit=7,x=129,n=100)# A raw test>>>pt.memory_usage_test(foo,raw_limit=25,x=129,n=100)# A relative test>>>pt.memory_usage_test(foo,relative_limit=1.2,x=129,n=100)You can, certainly, useRepeatandNumber:>>>pt.time_test(foo,relative_limit=7,x=129,n=100,Repeat=3,Number=1000)Raw tests work with raw executation time. Relative tests work with relative time against a call of an empty function; that way, the test should be more or less independent of the machine you run the test on; so, a quick machine should provide more or less similar relative results as a slow machine.Relative results, however, can differ between different operating systems.You can use these testing functions inpytest, or in dedicateddoctestfiles. You can, however, useperftesteras a separate performance testing framework. Read on to learn more about that. What's more,perftesteroffers more functionalities, and aconfigobject that offers you much more control of testing.That's all in this short introduction. If you're interested in more advanced use ofperftester, read on to read a far more detailed introduction. In addition, files in thedocsfolder explain in detail particular functionalities thatperftesteroffers.Introductionperftesteris a lightweight package for simple performance testing in Python. Here, performance refers to execution time and memory usage, so performance testing means testing if a function performs quickly enough and does not use too much RAM. In addition, the module offers you simple functions for straightforward benchmarking, in terms of both execution time and memory.Under the hood,perftesteris a wrapper around two functions from other modules:perftester.time_benchmark()andperftester.time_test()usetimeit.repeat()perftester.memory_usage_benchmark()andperftester.memory_usage_test()usememory_profiler.memory_usage()Whatperftesteroffers is a testing framework with as simple syntax as possible.You can useperftesterin three main ways:in an interactive session, for simple benchmarking of functions;as part of another testing framework, likedoctestorpytests; andas an independent testing framework.The first way is a different type of use from the other two. I use it to learn the behavior of functions (interms of execution time and memory use) I am working on right now, so not for actual testing.When it comes to actual testing, it's difficult to say which of the last two ways is better or more convinient: it may depend on how many performance tests you have, and how much time they take. If the tests do not take more than a couple of seconds, then you can combine them with unit tests. But if they take much time, you should likely make them independent of unit tests, and run them from time to time.UsingperftesterUse it as a separate testing frameworkTo useperftesterthat way,Collect tests in Python modules whose names start with "perftester_"; for instance, "perftester_module1.py", perftester_module2.py" and the like.Inside these modules, collect testing functions that start with "perftester_"; for instance,def perftester_func_1(),def perftester_func_2(), and the like (note how similar this approach is to that whichpytestuses);You can create a config_perftester.py file, in which you can change any configuration you want, using theperftester.configobject. The file should be located in the folder from which you will run the CLI commandperftester. If this file is not there,perftesterwill use its default configuration. Note that cofig_perftester.py is a Python module,soperftesterconfiguration is done in actual Python code.Now you can run performance tests usingperftesterin your shell. You can do it in three ways:$ perftesterrecursively collects allperftestermodules from the directory in which the command was run, and from all its subdirectories; then it runs all the collectedperftestertests;$ perftester path_to_dirrecursively collects allperftestermodules from path_to_dir/ and runs all perftesters located in them.$ perftester path_to_file.pyruns all perftesters from the module given in the path.Read more about using perftester that wayhere.Itdoesmake a difference how you do that. When you run theperftestercommand with each testing file independently, each file will be tested in a separated session, so with a new instance of thept.configobject. When you run the command for a directory, all the functions will be tested in one session. And when you run a bareperftestercommand, all your tests will be run in one session.There is no best approach, but remember to choose one that suits your needs.UseperftesterinsidepytestThis is a very simple approach, perhaps the simplest one: When you usepytest, you can simply addperftestertesting functions topytesttesting functions, and that way both frameworks will be combined, or rather thepytestframework will runperftestertests. The amount of additional work is minimal.For instance, you can write the following test function:importperftesterasptfrommy_moduleimportf1# assume that f1 takes two arguments, a string (x) and a float (y)deftest_performance_of_f1():pt.time_test(f1,raw_limit=10,relative_limit=None,x="whatever string",y=10.002)This will use either the settings for this particular function (if you set them inpt.config) or the default settings (also frompt.config). However, you can also useNumberandRepeatarguments, in order to overwrite these settings (passed totimeit.repeat()asnumberandrepeat, respectively) for this particular function call:importperftesterasptfrommy_moduleimportf1# assume that f1 takes two arguments, a string (x) and a float (y)deftest_performance_of_f1():pt.time_test(f1,raw_limit=10,relative_limit=None,x="whatever string",y=10.002Number=1_000_000,Repeat=20)If you now runpytestand the test passes, nothing will happen — just like with a regularpytesttest. If the test fails, however, aperftester.TimeTestErrorexception will be thrown, with some additional information.perftester's default behavior is to significantly shorten traceback, but only during testing (so when you runpt.time_test()andpt.memory_usage_test()). You can extend this behavior to other situations, with just one command:pt.config.cut_traceback(); to reverse, usept.config.full_traceback()— but do remember that this willnotmean the full traceback will be used during perftesting.This is the easiest way to useperftester. Its only drawback is that if the performance tests take much time,pytestwill also take much time, something usually to be avoided. You can then do somepytesttricks to not runperftestertests, and run them only when you want — or you can simply use the above-described command-lineperftesterframework for performance testing.UseperftesterinsidedoctestIn the same way, you can useperftesterindoctest. You will find plenty of examples in the documentation here, and in thetests/ folder.A great fan ofdoctesting, I donotrecommend usingperftesterin docstrings. For me,doctests in docstrings should clarify things and explain how functions work, and adding a performance test to a function's docstring would decrease readability.The best way, thus, is to write performance tests as separatedoctestfiles, dedicated to performance testing. You can collect such files in a shell script that runs performance tests.Basic use ofperftesterSimple benchmarkingTo create a performance test for a function, you likely need to know how it behaves. You can run two simple benchmarking functions,pt.memory_usage_benchmark()andpt.time_benchmark(), which will run time and memory benchmarks, respectively. First, we will decreasenumber(passed totimeit.repeat), in order to shorten the benchmarks (which here serve asdoctests):>>>importperftesteraspt>>>deff(n):returnsum(map(lambdai:i**0.5,range(n)))>>>pt.config.set(f,"time",Number=1000)>>>b_100_time=pt.time_benchmark(f,n=100)>>>b_100_memory=pt.memory_usage_benchmark(f,n=100)>>>b_1000_time=pt.time_benchmark(f,n=1000)>>>b_1000_memory=pt.memory_usage_benchmark(f,n=1000)Remember also about the possibility of overwriting (for this single benchmark) the settings frompt.config.settings, which you can do usingNumber(only for time testing) andRepeat(for both):pt.time_benchmark(f, n=100, Number=1_000_000, Repeat=20)andpt.memory_usage_benchmark(f, n=1000, Repeat=10).And this is it. You can usept.pp()function to pretty-print the results. In my machine, I got the following results (here, forb_100):# pt.pp(b_100_time){'max':16.66,'max_relative':1.004,'max_result_per_run':[16.66],'max_result_per_run_relative':[1.004],'mean':16.66,'mean_result_per_run':[16.66],'raw_results':[[16.66,16.66,16.66]],'relative_results':[[1.004,1.004,1.004]]}# pt.pp(b_100_memory){'max':1.389e-05,'mean':1.303e-05,'min':1.168e-05,'min_relative':129.5,'raw_times':[1.168e-05,1.263e-05,1.349e-05,1.346e-05,1.389e-05],'raw_times_relative':[129.5,140.0,149.5,149.2,154.0]}For memory testing, the main result ismaxwhile for time testing, it ismin. For relative testing, we would look atmax_relativeandmin_relative, respectively.Surely, we should expect that the function withn=100be quicker than withn=1000:>>>b_100_time["min"]<b_1000_time["min"]Truebut memory use will be more or less the same:>>>importmath>>>math.isclose(b_100_memory["max"],b_1000_memory["max"],rel_tol=.01)TrueTime testingFor time tests, we have thept.time_test()function. First, a raw time test:>>>pt.time_test(f,raw_limit=2e-05,n=100)raw_limit,relative_limit,NumberandRepeatare keyword-only arguments.Like before, we can useNumberandRepeatarguments:>>>pt.time_test(func=f,raw_limit=3e-05,n=100,Number=10)Now, let's define a relative time test:>>>pt.time_test(f,relative_limit=230,n=100)We also can combine both:>>>pt.time_test(f,raw_limit=2e-05,relative_limit=230,n=100)You can read about relative testing below,in section.Memory testingMemory tests usept.memory_usage_test()function, which is used in the same way aspt.time_test():>>>pt.memory_usage_test(f,raw_limit=27,n=100)# test on raw memory>>>pt.memory_usage_test(f,relative_limit=1.2,n=100)# relative time test>>>pt.memory_usage_test(f,raw_limit=27,relative_limit=1.2,n=100)# bothIn a memory usage test, a function is called only once. You can change that — but do that only if you have solid reasons — using, for example,pt.config.set(f, "time", "repeat", 2), which will set this setting for the function in the configuration (so it will be used for all next calls for functionf()). You can also do it just once (so, without saving the setting inpt.config.settings), using theRepeatargument:>>>pt.memory_usage_test(f,raw_limit=27,relative_limit=1.2,n=100,Repeat=100)(There is little sence in repeating this particular function, as you will get almost the same results in each repetition.)Of course, memory tests do not have to be very useful for functions that do not have to allocate too much memory, but as you will see in other documentation files inperftester, some function do use a lot of memory, and such tests do make quite a lot sense for them.Configuration:pt.configThe whole configuration is stored in thept.configobject, which you can easily change. Here's a short example of how you can use it:>>>deff(n):returnlist(range(n))>>>pt.config.set(f,"time",Number=10_000,Repeat=1)but you can change much more using it.You can read in detail about usingpt.confighere.When you useperftesteras a command-line tool, you can modifypt.configin thesettings_perftester.pymodule, for instance:# settings_perftester.pyimportperftesteraspt# shorten the testspt.config.set_defaults("time",Number=10_000,Repeat=3)# log the results to file (they will be printed in the console anyway)pt.config.log_to_file=Truept.config.log_file="./perftester.log"# increase the digits for printing floating numberspt.config.digits_for_printing=7# Use regular tracebackpt.config.full_traceback()and so on. You can also change settings in each testing file itself, preferably inperftester_functions.When you useperftesterin an interactive session, you updatept.configin a normal way, in the session. And when you useperftesterinsidepytest, you can do it in conftest.py and in each testing function.OutputIf a test fails, you will see something like this:# for time testTimeTestErrorinperftester_for_testing.perftester_f
Timetestnotpassedforfunctionf:raw_limit=0.011
minimumruntime=0.1007# for memory testMemoryTestErrorinperftester_for_testing.perftester_f2_time_and_memory
Memorytestnotpassedforfunctionf2:memory_limit=20maximummemoryusage=20.04Let's analyze what we see in this output:Whether it's an error from a time test (TimeTestError) or a memory test (MemoryTestError).perftester_for_testing.perftester_fprovides the testing module (perftester_for_testing) and the perftester function (perftester_f2_time_and_memory).Memory test not passed for function f2:: Here you see for which tested (notperftester_) function the test failed (here,f2()).raw_limitandmemory_limit: these are the raw limits you provided; these could be alsorelative_limitandrelative_memory_limit, for relative tests.minimum run timeandmaximum memory usageare the actual results from testing, and they were too high (higher than the limits set inside the testing function).You can locate where a particular test failed, using the module,perftester_function, and the tested function. If aperftester_function combines more tests, then you can find the failed test using the limits.Like inpytest, a recommended approach is to use one performance test perperftester_function. This can save you some time and trouble, but also this will ensure that all tests will be run.Summary outputAt the end, you will see a simple summary of the results, something like this:Outof8tests,5haspassedand3hasfailed.

Passedtests:
perftester_for_testing.perftester_f2
perftester_for_testing.perftester_f2_2
perftester_for_testing.perftester_f2_3
perftester_for_testing.perftester_f3
perftester_for_testing_2.perftester_f

Failedtests:
perftester_for_testing.perftester_f
perftester_for_testing.perftester_f2_time_and_memory
perftester_for_testing.perftester_f_2Relative tests against another functionIn the basic use, when you choose a relative benchmark, you compare the performance of your function with that of a built-in (empty) functionpt.config.benchmark_function(). In most cases, this is what you need. Sometimes, however, you may wish to benchmark against another function. For instance, you may want to build your own function that does the same thing as a Python built-in function, and you want to test (and show) that your function performs better. There are two ways of achieving this:you can use a simple trick;see here;you can overwrite the built-in benchmark functions;see here.Raw and relative performance testingSurely, any performance tests are strongly environment-dependent, so you need to remember that when writing and conducting any performance tests.perftester, however, offers a solution to this: You can define tests based onraw values: raw execution time and raw memory usage, andrelative values: relative execution time and relative memory usageAbove,relativemeans benchmarking against a built-in (intoperftester) simple function, which is actually an empty function (so it represents the overhead of running a function). Thus, you can, for instance, test whether your function is two times slower than this function. The benchmarking function itself does not matter, as it is just a benchmark. What matters is that, usually, your function shouldrelatively to this benchmarking functionbehave the same way between different machines. So, if it works two times slower than the benchmarking function on your machine, then it should work in a similar way on another machine, even if this other machine is much faster than yours. Of course, this assumes linearity (so, two times slower here means two times slower everywhere), which does not have to be always true. Anyway, such tests will almost always be more representative, and more precise, than those based on raw times.This does not mean, however, that raw tests are useless. In fact, in a production environment, you may wish to use raw tests. Imagine a client expects that an app never takes longer than an hour to perform a particular task (note that this strongly depends on what other processes are run in the production environment). You can create an automated test for that usingperftester, in a very simple way - just several lines of code.You can of course combine both types of tests, and you can do it in a very simple way. Then, the test is run once, but the results are checked with raw limits and relative limits.Warning! Relative results can be different between operating systems.Other toolsOf course, Python comes with various powerful tools for profiling, benchmarking and testing. Here are some of them:cProfileandprofile, the built-in powerful tools for deterministic profilingthe built-intimeitmodule, for benchmarkingmemory_profiler, a powerful memory profiler (memory_profileris utilized byperftester)In fact,perftesteris just a simple wrapper aroundtimeitandmemory_profiler, sinceperftesteritself does not come with its own solutions. It simply uses these functions and offers an easy-to-use API to benchmark and test memory and time performance.Manipulating the tracebackThe default behavior ofperftesteris tonotinclude the full traceback when a test does not pass. This is because when running performance tests, you're not interested in finding bugs, and this is what traceback is for. Instead, you want to see which test did not pass and how.This behavior will not affect any other function than the twoperftestertesting functions:pt.time_test()andpt.memory_usage_test(). If you want to use this behavior for other functions, too, you can usept.config.cut_traceback(); to reverse, usept.config.full_traceback().Tracing full memory usage → Moved totracememSince the0.5.*versions,perftestercontained a beta version of a memory tracer that could be used to trace full memory usage of a Python session.Sinceperftesterrequires some memory to load, it over-measured session memory. In order to avoid this, this feature was moved to a separate Python package, calledtracemem. You can install it fromPyPi, and you will find its Git repositoryhere.Caveatsperftesterdoes not work with multiple threads or processes.perftesteris still in a beta version and so is still under testing.Watch out when you're running the same test in different operating systems. Even relative tests can differ from OS to OS.Operating systemsThe package is developed in Linux (actually, under WSL) and checked in Windows 10, so it works in both these environments.SupportAny contribution will be welcome. You can submit an issue in therepository. You can also create your own pull requests.