Package: codeprofile
Python Code ProfilerFor measuring how much time different functions or pieces of code take to execute.InstallingPyPipip install codeprofileFor more details, check thePython Package Index project.Source CodeHosted onGithub. At this time, practically just one file.UsageAdd the provided decorators to the pieces of code / functions you want to profile.This document uses the termtrace point.
It refers to part of code that is profiled as a single block.
If you profile execution time of a specific function, that function is the trace point.
If you profile a specific block of code inside a function, that block is the trace point.FunctionsTo trace / profile a function:fromcodeprofileimportprofilerimporttime@profiler.profile_funcdefmy_func():print("hello world")time.sleep(0.1)x=some_other_func()returnint(x)In the above, every time the functionmy_funcis executed, its execution time is recorded.
It becomes a trace point namedmy_func.Code snippetsTo trace the performance of a code snippet inside a function / module:fromcodeprofileimportprofilerforxinrange(100):withprofiler.profile("block read"):block=read_block(x)withprofiler.profile("db insert"):insert_into_database(block)In the above, there are two trace points nameblock readanddb insert.
The code above executes each trace point 100 times,
resulting in 100 performance measurements for each trace point.
The profiler will then provice access to cumulative, average, min, max, and count statistics
for each trace point.
If configured to keep the raw measurements, those and the median are also available per trace point.Imagine the aboveblock readas an example of reading a block of data over the network (e.g., scanning a blockchain).
And writing this to a database indb insert. Maybe it seems slow.
With the above profiling, you can look at your program and wonder, where does the time go?
Now there is a question we all would like to know..AsyncIOAsyncIOis the Python way of making
use of inherent processing delays in IO-intensive operations to execute code in parallel.
For example, the system might be waiting for some disk or network IO to proceed.
At such points, the CPU is just idle.
So AsyncIO is intended as a way to execute other code in parallel while waiting on IO.
Since Python has no true multithreading (the multi-processing module is not quite the same),
this can be a nice feature.AsyncIO uses special keywords (such asasync) and code-structures to manage all this.
Functions used with AsyncIO thus need to be defined with theasynckeyword.Because of this, the approach to profile regular functions with@profiler.profile_funcdoes not work with AsyncIO, as theprofile_funcdecorator is notasync.
A different decorator using theasyncdefinition is provided for this purpose:fromcodeprofileimportprofilerimportasyncio@profiler.profile_async_funcasyncdefa_blocker(self):block=read_block()# <- assume this function uses asyncio to access network / diskawaitasyncio.sleep(1)insert_into_database(block)# <- again, assume this call uses asyncio access to a databaseIn the above example,a_blockerbecomes a trace point measuring the function execution time for AsyncIO.For code blocks inside AsyncIO methods / functions, the same approach as for other code blocks should work.fromcodeprofileimportprofilerasyncdefhundred_blocks():forxinrange(100):withprofiler.profile("block read"):block=read_block()withprofiler.profile("db insert"):insert_into_database(block)Configurationignore_sleep: If true, use a performance counter that ignores time spent in sleep mode. Defaults to false.collect_raw: If true, keep the raw measurement data for eachtrace point. Takes more memory, but gives access to more detailed profiling information. Defaults to true.Setting them:fromcodeprofileimportprofilerprofiler.collect_raw=False#by default this is trueprofiler.ignore_sleep=True#by default this is falseData AnalysisThe performance data collected is stored and available as part of theprofilermodule.Access raw resultsThe following variables are available as part of theprofilermodule:cumulative_times: sum of all recorded execution times for a trace point.max_times: highest time per trace pointmin_times: minimum time per trace pointcounts: number of times a trace point execution has been recorded.raw_times: list of all recorded execution times per trace point.The following function can be used ifcollect_rawis enabled:median_times: median time per trace point.Like so:fromcodeprofileimportprofilercumulative_block_time=profiler.cumulative_times["block read"]max_block_time=profiler.max_times["block read"]min_block_time=profiler.min_times["block read"]all_block_times=profiler.raw_times["block read"]median_block_time=profiler.median("block read")If you want to reset the statistics while running:fromcodeprofileimportprofilerprofiler.reset_stats()Summary Printoutsprint_run_stats(*names, file=sys.stdout)Thenamesparameter forprint_run_statsis optional.
It defaults to all names.
A name is simply a reference to name of a trace point.Thefileparameter allows writing the results to a file, a string, or elsewhere.
By default, it uses the system output, writing the summary to console.
You can also use an actual filesystem file as target, or build a string usingio.StringIO,
or whatever else the Python filesystem can do.Export to CSV and Pandasprint_csv(*names, file=sys.stdout)The result ofprint_csvis a CSV file, where each column represents a trace-point.
The rows are not synchronized in any way, since only the person who implements the trace-points knows if they are related.
So each column is just a list of traced values (performance times) for that trace-point.
Each column has equal number of values, which is the largest number of points recorded for any trace-point.
The ones with fewer values have the last rows left empty (nan in Pandas).fromcodeprofileimportprofilerimportpandasaspdwithopen("output.csv","wb")asf:profiler.print_csv(file=f)df=pd.read_csv("output.csv")Hierarchical profilingYou can also nest the trace profiling calls.
For example:fromcodeprofileimportprofilerwithprofiler.profile("loop"):forxinrange(100):withprofiler.profile("block read"):block=read_block(x)withprofiler.profile("db insert"):insert_into_database(block)With the above, you would have thelooptrace point collecting stats for the read and write operations together,
and theblock readanddb inserttrace points for the specific operations.LicenseMIT