Package: benchit
Tools to benchmark Python solutions on runtime performance and visualize. Based ontimeit, it primarily aims to functionally simulate thetimeitbehaviour and hence the name! This facilitates benchmarking on multiple datasets and solutions.DocumentationInstallationLatest PyPI stable release :pipinstallbenchitPull latest development release on GitHub and install in the current directory :pipinstall-egit+https://github.com/droyed/benchit.git@master#egg=benchitGetting startedConsider a setup to compare NumPy ufuncs -sum,prod,maxon arrays varying in their sizes. To keep it simple, let’s consider1Darrays. Thus, we would have :>>>importnumpyasnp>>>funcs=[np.sum,np.prod,np.max]>>>inputs=[np.random.rand(i)foriin10**np.arange(5)]>>>importbenchit>>>t=benchit.timings(funcs,inputs)It’s adataframe-likeobject and as such we can plot it. It automatically adds in specs into the title area to convey all of available benchmarking info :>>>t.plot(logy=True,logx=True)Multiple argumentsLet’s consider a setup where functions accept more than one argument. Let’s take the case of computingeuclidean distancesbetween two2Darrays. We will feed in arrays with varying number of rows and 3 columns to represent data in3DCartesian coordinate system and benchmark two commonly used functions in Python.>>>fromsklearn.metrics.pairwiseimportpairwise_distances>>>fromscipy.spatial.distanceimportcdist>>>fns=[cdist,pairwise_distances]>>>importnumpyasnp>>>in_={n:[np.random.rand(n,3),np.random.rand(n,3)]fornin[10,100,500,1000,4000]}>>>t=benchit.timings(fns,in_,multivar=True,input_name='Array-length')>>>t.plot(logx=True)Multiple arguments with groupingsWe will extend the previous example to make the second argument a variable too and study the trend as we vary the number of columns, resulting in subplots.# Get benchmarking object (dataframe-like) and plot results>>>R=np.random.rand>>>in_={(n,W):[R(n,W),R(n,W)]fornin[10,100,500,1000]forWin[3,20,50,100]}>>>t=benchit.timings(fns,in_,multivar=True,input_name=['nrows','ncols'])>>>t.plot(logx=True,sp_ncols=2,sp_argID=0,sp_sharey='g')For plotting, we are using number of rows as the x-axis base.Usesp_argID=1to switch-over to use number of cols as the x-axis base instead.Single argument with groupingsLet’s manufacture a simple forward-filling scheme based on indices ofTruevalues in a boolean-array :# Functionsdefrepeat(b):idx=np.flatnonzero(np.r_[b,True])returnnp.repeat(idx[:-1],np.diff(idx))defmaxaccum(b):returnnp.maximum.accumulate(np.where(b,np.arange(len(b)),0))in_={(n,sf):np.random.rand(n)<(100-sf)/100.fornin[100,1000,10000,100000,1000000]forsfin[20,40,60,80,90,95]}t=benchit.timings([repeat,maxaccum],in_,input_name=['Array-length','Sparseness %'])t.plot(logx=True,sp_ncols=2,save='singlegrp_id0_ffillmask_timings.png')Quick Tips1. Plotting on notebooks?Usebenchit.setparams(environ='notebook')before plotting. Check outsample notebook run.2. Get a quick glance into the benchmarking trend before the actual oneUsebenchit.setparams(rep=1)before plotting. Then, usebenchit.setparams()for a proper benchmarking.3. Get a quicker glance into plot layout and vague benchmarking trend before the actual oneUsebenchit.setparams(timeout=1e-5,rep=1)before plotting. Then, usebenchit.setparams()for a proper benchmarking.4.  Working with multi-variable datasets to study trend w.r.t. each argument?Use nested loops to set-up input datasets as shown earlier. More information is available in documentation.As a general rule, it’s advisable to work on Python3.6or newer for better plotting experience.