Package: microbench
MicrobenchMicrobench is a small Python package for benchmarking Python functions, and
optionally capturing extra runtime/environment information. It is most useful in
clustered/distributed environments, where the same function runs under different
environments, and is designed to be extensible with new
functionality. In addition to benchmarking, this can help reproducibility by
e.g. logging the versions of key Python packages, or even all packages loaded
into the global environment. Other captured metadata can include CPU and RAM
usage, environment variables, and hardware specifications.RequirementsMicrobench by default has no dependencies outside of the Python standard
library, althoughpandasis recommended to
examine results. However, some mixins (extensions) have specific requirements:Theline_profilerpackage needs to be installed for line-by-line code benchmarking.MBInstalledPackagesrequiressetuptools, which is not a part of the
standard library, but is usually available.The CPU cores, total RAM, and telemetry extensions requirepsutil.The NVIDIA GPU plugin requires thenvidia-smiutility, which usually ships with the NVIDIA graphics card drivers. It needs
to be on yourPATH.InstallationTo install usingpip:pip install microbenchUsageMicrobench is designed for benchmarking Python functions. These examples will
assume you have already defined a Python functionmyfunctionthat you wish to
benchmark:defmyfunction(arg1,arg2,...):...Minimal exampleFirst, create a benchmark suite, which specifies the configuration and
information to capture.Here's a minimal, complete example:frommicrobenchimportMicroBenchbasic_bench=MicroBench()To attach the benchmark to your function, simply usebasic_benchas a
decorator, like this:@basic_benchdefmyfunction(arg1,arg2,...):...That's it! Whenmyfunction()is called, metadata will be captured
into aio.StringIO()buffer, which can be read as follows
(using thepandaslibrary):importpandasaspdresults=pd.read_json(basic_bench.outfile.getvalue(),lines=True)The above example captures the fieldsstart_time,finish_timeandfunction_name. Microbench can capture many other types of metadata
from the environment, resource usage, and hardware,
which are covered below.Extended examplesHere's a more complete example using mixins (theMBprefixed class
names) to extend functionality. Note that keyword arguments can be supplied
to the constructor (in this casesome_info=123) to specify additional
information to capture. This example also specifies theoutfileoption,
which appends metadata to a file on disk.frommicrobenchimport*importnumpy,pandasclassMyBench(MicroBench,MBFunctionCall,MBPythonVersion,MBHostInfo):outfile='/home/user/my-benchmarks'capture_versions=(numpy,pandas)# Or use MBGlobalPackages/MBInstalledPackagesenv_vars=('SLURM_ARRAY_TASK_ID',)benchmark=MyBench(some_info=123)Theenv_varsoption from the example above specifies a list of environment
variables to capture asenv_<variable name>. In this example,
theslurmarray task ID will be stored asenv_SLURM_ARRAY_TASK_ID. Where the environment variable is not set, the
value will benull.To capture package versions, you can either specify them individually (as
above), or you can capture the versions of every package in the global
environment. In the following example, we would capture the versions ofmicrobench,numpy, andpandasautomatically.frommicrobenchimport*importnumpy,pandasclassBench2(MicroBench,MBGlobalPackages):outfile='/home/user/bench2'bench2=Bench2()If you want to go even further, and capture the version of every package
available for import, there's a mixin for that:frommicrobenchimport*classBench3(MicroBench,MBInstalledPackages):passbench3=Bench3()MixinFields captured(default)start_timefinish_timefunction_nameMBGlobalPackagespackage_versions, with entry for every package in the global environmentMBInstalledPackagespackage_versions, with entry for every package available for importMBCondaPackagesconda_versions, with entry for every conda package in the environmentMBFunctionCallargs(positional arguments)kwargs(keyword arguments)MBReturnValueWrapped function's return valueMBPythonVersionpython_version(e.g. 3.6.0) andpython_executable(e.g./usr/bin/python, which should indicate any active virtual environment)MBHostInfohostnameoperating_systemMBHostCpuCorescpu_cores_logical(number of cores, requirespsutil)MBHostRamTotalram_total(total RAM in bytes, requirespsutil)MBNvidiaSmiVarious NVIDIA GPU fields, detailed in a later sectionMBLineProfilerline_profilercontaining line-by-line profile (see section below)Examine resultsEach result is aJSONobject. When using
theoutfileoption, a JSON object for each@benchmarkcall is stored on a
separate line in the file. The output from the minimal example above for a
single run will look similar to the following:{"start_time":"2018-08-06T10:28:24.806493","finish_time":"2018-08-06T10:28:24.867456","function_name":"my_function"}The simplest way to examine results in detail is to load them into apandasdataframe:importpandasresults=pandas.read_json('/home/user/my-benchmarks',lines=True)Pandas has powerful data manipulation capabilities. For example, to calculate
the average runtime by Python version:# Calculate runtime for each runresults['runtime']=results['finish_time']-results['start_time']# Average runtime by Python versionresults.groupby('python_version')['runtime'].mean()Many more advanced operations are available. Thepandas tutorialis recommended.Line profiler supportMicrobench also has support forline_profiler, which shows the execution time
of each line of Python code. Note that this will slow down your code, so only use it if needed, but it's useful for
discovering bottlenecks within a function. Requires theline_profilerpackage to be installed
(e.g.pip install line_profiler).frommicrobenchimportMicroBench,MBLineProfilerimportpandas# Create our benchmark suite using the MBLineProfiler mixinclassLineProfilerBench(MicroBench,MBLineProfiler):passlpbench=LineProfilerBench()# Decorate our function with the benchmark suite@lpbenchdefmy_function():""" Inefficient function for line profiler """acc=0foriinrange(1000000):acc+=ireturnacc# Call the function as normalmy_function()# Read the results into a Pandas DataFrameresults=pandas.read_json(lpbench.outfile.getvalue(),lines=True)# Get the line profiler report as an objectlp=MBLineProfiler.decode_line_profile(results['line_profiler'][0])# Print the line profiler reportMBLineProfiler.print_line_profile(results['line_profiler'][0])The last line of the previous example will print the line profiler report, showing the execution time of each line of
code. Example:Timer unit: 1e-06 s

Total time: 0.476723 s
File: /home/user/my_test.py
Function: my_function at line 12

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    12                                               @lpbench
    13                                               def my_function():
    14                                                   """ Inefficient function for line profiler """
    15         1          2.0      2.0      0.0          acc = 0
    16   1000001     217874.0      0.2     45.7          for i in range(1000000):
    17   1000000     258846.0      0.3     54.3              acc += i
    18
    19         1          1.0      1.0      0.0          return accNVIDIA GPU supportAttributes about NVIDIA GPUs can be captured using theMBNvidiaSmiplugin.
This requires thenvidia-smiutility to be available in the currentPATH.By default, thegpu_name(model number) andmemory.totalattributes are
captured. Extra attributes can be specified using the class or object-level
variablenvidia_attributes. To see which attributes are available, runnvidia-smi --help-query-gpu.By default, all installed GPUs will be polled. To limit to a specific GPU,
specify thenvidia_gpusattribute as a tuple of GPU IDs, which can be
zero-based GPU indexes (can change between reboots, not recommended),
GPU UUIDs, or PCI bus IDs. You can find out GPU UUIDs by runningnvidia-smi -L.Here's an example specifying the optionalnvidia_attributesandnvidia_gpusfields:frommicrobenchimportMicroBench,MBNvidiaSmiclassGpuBench(MicroBench,MBNvidiaSmi):outfile='/home/user/gpu-benchmarks'nvidia_attributes=('gpu_name','memory.total','pcie.link.width.max')nvidia_gpus=(0,)# Usually better to specify GPU UUIDs here insteadgpu_bench=GpuBench()Telemetry supportWe use the term "telemetry" to refer to metadata which is captured periodically
during the execution of a function by a thread which runs in parallel. For
example, this may be useful to see how memory usage changes over time.Telemetry support requires thepsutillibrary.Microbench launches and cleans up the monitoring thread automatically.
The end user only needs to define atelemetrystatic method, which accepts
apsutil.Processobject and returns the telemetry data as a dictionary.The default telemetry collection interval is every 60 seconds, which can be
customized if needed using thetelemetry_intervalclass variable.A minimal example to capture memory usage every 90 seconds is shown below:frommicrobenchimportMicroBenchclassTelemBench(MicroBench):telemetry_interval=90@staticmethoddeftelemetry(process):returnprocess.memory_full_info()._asdict()telem_bench=TelemBench()Extending microbenchMicrobench includes a few mixins for basic functionality as described in the
extended example, above.You can also add functions to your benchmark suite to capture
extra information at runtime. These functions must be prefixed withcapture_for them to run automatically before the function starts. They take
a single argument,bm_data, a dictionary to be extended with extra data.
Care should be taken to avoid overwriting existing key names.Here's an example to capture the machine type (i386,x86_64etc.):frommicrobenchimportMicroBenchimportplatformclassBench(MicroBench):outfile='/home/user/my-benchmarks'defcapture_machine_platform(self,bm_data):bm_data['platform']=platform.machine()benchmark=Bench()Extending JSONEncoderMicrobench encodes data in JSON, but sometimes Microbench will
encounter data types (like custom objects or classes)
that are not encodable as JSON by default (usually meaning they
don't have a way to be represented as a string, list, or
dictionary). For example, when using theMBFunctionCallandMBReturnValue, a warning will be shown if any argument or
return value (respectively) is not encodable as JSON, and the
value will be replaced with a placeholder to allow the metadata
capture to continue, and a warning will be shown.If you wish to actually capture those values, you will need to
specify a way to convert the object to JSON. This is done using
by extendingmicrobench.JSONEncoderwith a test for the object
type and implementing a conversion to a string, list, or dict.For example, to capture aGraphobject from theigraphpackage usingstr(graph)as the representation, we could
do the following (note that we could use any representation
we want, e.g. if we wanted to capture the object in a more
or less detailed way):import microbench as mb
from igraph import Graph

# Extend the JSONEncoder to encode Graph objects
class CustomJSONEncoder(mb.JSONEncoder):
    def default(self, o):
        # Encode igraph.Graph objects as strings
        if isinstance(o, Graph):
            return str(o)

        # Add further isinstance(o, ...) cases here
        # if needed

        # Make sure to call super() to handle
        # default cases
        return super().default(o)

# Define your benchmark class as normal
class Bench(mb.MicroBench, mb.MBReturnValue):
    pass

# Create a benchmark suite with the custom JSON
# encoder from above
bench = Bench(json_encoder=CustomJSONEncoder)

# Attach the benchmark suite to our function
@bench
def return_a_graph():
    return Graph(2, ((0, 1), (0, 2)))

# This should now work without warnings or errors
return_a_graph()Redis supportBy default, microbench appends output to a file, but output can be directed
elsewhere, e.g.redis- an in-memory, networked data source.
This option is useful when a shared filesystem is not available.Redis support requiresredis-py.To use this feature, inherit fromMicroBenchRedisinstead ofMicroBench,
and specify the redis connection and key name as in the following example:frommicrobenchimportMicroBenchRedisclassRedisBench(MicroBenchRedis):# redis_connection contains arguments for redis.StrictClient()redis_connection={'host':'localhost','port':6379}redis_key='microbench:mykey'benchmark=RedisBench()To retrieve results, theredispackage can be used directly:importredisimportpandas# Establish the connection to redisrconn=redis.StrictRedis(host=...,port=...)# Read the redis data from 'myrediskey' into a list of byte arraysredis_data=redis.lrange('myrediskey',0,-1)# Convert the list into a single stringjson_data='\n'.join(r.decode('utf8')forrinredis_data)# Read the string into a pandas dataframeresults=pandas.read_json(json_data,lines=True)Runtime impact considerationsThe runtime impact varies depending on what information is captured and by platform.
Broadly, capturing environment variables, Python package versions, and timing
information for a function has a negligible impact. Capturing telemetry and
invoking external programs (likenvidia-smifor GPU information) has a larger impact,
although the latter is a one-off per invocation and typically less than one second.
Telemetry capture intervals should be kept relatively infrequent (e.g., every minute
or two, rather than every second) to avoid significant runtime impacts.FeedbackPlease note this is a recently created, experimental package. Please let me know
your feedback or feature requests in Github issues.