Package: defbench
defbench.pya simple python3 module for easily benchmarking the elapsed time and memory usage of python functionsquick startpipinstalldefbenchimport defbenchdefine a function to testusedefbench.run(func[, repeat][, name])importdefbenchdefsearch_list():"two"in["one","two","three"]results=defbench.run(search_list,repeat=1000)print(results)outputs<TestRun 'search_list'
  runs:     1,000
  avg time: 0.0002124s
  avg mem:  0.0Mib>defbench.run()this is the easiest way to use this tool. it generates aTestRunobject, returns it, and adds it todefbench.historyfor later review or analysis.run(func:Callable,repeat:int=10,name:str=None)->TestRunpass in the function to be tested. optionally, specify the number of times to repeat the function and the name to use. if a name is not provided, it defaults to the function name ("<lambda>" for lambda functions)TestRunaTestRunobject is returned by eitherdefbench.run()ordefbench.Test.run(). it contains all of the results from benchmarking a function.attributesattributetypedescription_mem_rawList[float]raw slices of theentireprogram's memory usage (MiB) over the course of every repeated call as returned bymemory-profiler_memList[float]_mem_raw "normalized" (initial program memory usage subtracted from all memory usage values)funcCallablethe benchmarked functionnamestrthe function nameorthe name passed with.run(func, name="foobar")timefloatthe total time (seconds) of all runsavg_timefloatthe average time (seconds) of each runmemoryfloatthe peak memory usage (MiB)stdoutstroutput is suppressed by.run()and stored here for later retrievalrepeatintnumber of tests runinitialization__init__(self,func:Callable,name:str=None,repeat:int=1,mem:List[float]=[],time:float=0.0,stdout:str="")aTestRunobject isnotmeant to be initialized on its own! it is meant to be generated by eitherdefbench.Test.run()ordefbench.run()measuring memoryi want to explain how memory measurement withmemory-profiler(and thus defbench) works since it took me a minute to understand. essentially, it captures theentireprogram's memory usage at specified intervals (0.01s for now, although i might tweak that and/or let the interval be set). when profiling a specific function, it still grabs theentireprogram's memory usage; it just does so for the duration that the function is running. i.e.:your python script starts up -- 1MiB -- 1MiB totalyou create some lists and do some stuff -- 3MiB -- 4MiB totalyou usedefbench.run(my_function)-- 2MiB -- 6MiB totalso whendefbench.run(my_function)is called,memory-profilerwill report4.0as the initial memory usage slice and6.0as the peak memory slice (e.g.:[4.0, 4.3, 4.9, 5.5, 6.0]). this is what's stored inTestRun._mem_raw. however, we don't really care about the rest of the program, so we subtract the initial value from all subsequent memory usage slices (e.g.:[0.0, 0.3, 0.9, 1.5, 2.0]). this is what's stored inTestRun._mem. but since all most usersreallycare about is the peak usage, that's what's returned byTestRun.memory(2.0in our example).and just to be totally clear,max()is used to find the peak memory usage slice. so even if some objects get released from memory throughout your function, and the last memory usage slice is lower than the peak (e.g.:[0.0, 0.5, 1.0, 0.8]), the maximum value is still returned byTestRun.memory(e.g.:1.0).stdoutduring a.run()call, all output tosys.stdout(e.g.print()statements) is temporarily redirected so that output can be captured. you can access it later usingTestRun.stdoutTestaTestobject is initialiezd with a function name and some default values. callingTest.run()will add a newTestRunobject toTest.historyanddefbench.historyand then return it.attributesattributetypedescription_funcCallablethe benchmarked function_repeatintthe default number of times to run the function_runningboolboolean representing if thisTestis currently runningnamestrthe default name to use for testshistoryList[TestRun]allTestRuns generated byTest.run()methodsrun(repeat:int=100,name:str=None)->TestRunreturns a newTestRunand appends it toTest.history. optionally, set the number of times to run repeat the test and the name to use for thisTestRun.default values prioritypriority for name/repeat values used are:value passed in toTest.run()ordefbench.run()value passed to__init__()(orTest()) during initialization(for name only) function name if neither above provideddefault value (10 for repeat and "<function>" for function names, but i'm not sure a function can evennothave a name 99% of the time... still, it's there just in case someone does some weird code compiling voodoo)historythere's a history object at the module level (defbench.history) that contains a history of everyTestRunobject (each instance is added inTestRun.__init__).attributesattributetypedescription_historyList[TestRun]allTestRunobjectsmethodsaverage_time(filter:Callable=None)->floatreturn the average time of allTestRuns. optionally, pass a function to be used as a filter, e.g.:history.average_time(lambda x: x.name == "str_list_test")average_memory(filter:Callable=None)->floatget the average memory usage inMiBof allTestRuns. optionally, pass a function to be used as a filter, e.g.:history.average_memory(lambda x: x.stdout.contains("hello world")add(run:TestRun)->Noneadd aTestRunto the module level historyget(filter:Callable=None)->List[TestRun]return allTestRunobjects in history. optionally, pass a function to be used as a filter, e.g.:history.get(lambda x: x.time > 30)todocreate a generic_historyclass to be used at the module level and by instances ofTestadd more analysis options for items inhistorycapturestderrinTestRun.run()?add to pipadd unit tests