Package: benchmarkit
BenchmarkitBenchmark and analyze functions' time execution and results over the course of development.FeaturesNo boilerplate codeSaves history and additional infoSaves function output and parameters to benchmark data science tasksEasy to analyze resultsDisables garbage collector during benchmarkingMotivationI need to benchmark execution time of my functionI don't want to memorize and write boilerplate codeI want to compare results with previous runs before some changes were introducedI don't want to manually write down results somewhereI want to know exact commits of my previous runs months agoI want to benchmark accuracy, precision, recall of my models and keep track of hyperparametersInstallationpip install benchmarkitUsageBenchmark execution timesPut@benchmarkdecorator over function with piece of code that should be timedfrombenchmarkitimportbenchmark,benchmark_runN=10000seq_list=list(range(N))seq_set=set(range(N))SAVE_PATH='/tmp/benchmark_time.jsonl'@benchmark(num_iters=100,save_params=True,save_output=False)defsearch_in_list(num_items=N):returnnum_items-1inseq_list@benchmark(num_iters=100,save_params=True,save_output=False)defsearch_in_set(num_items=N):returnnum_items-1inseq_setnum_iters- how many times to repeat benchmarked function. Default1save_params- save parameters passed to the benchmarked function in the file with benchmark results. In the example abovenum_itemswill be saved. DefaultFalsesave_output- save benchmarked function output. Should return dict{'name': value}. DefaultFalse. See example how to benchmark model results.Run benchmark:benchmark_results=benchmark_run([search_in_list,search_in_set],SAVE_PATH,comment='initial benchmark search',rows_limit=10,extra_fields=['num_items'],metric='mean_time',bigger_is_better=False,)functions- function or list of functions withbenchmarkdecoratorsave_file- path to file where to save resultscomment- comment to save alongside the resultsrows_limit- limit table rows in console output. Default10extra_fields- extra fields to include in console outputmetric- metric which is used for comparison. Defaultmean_timebigger_is_better- whether bigger value of metric indicates that result is better. For time benchmarks should beFalse, for model accuracy should beTrue. DefaultFalsePrints to terminal and returns list of dictionaries with data for the last run.ChangeN=1000000and rerunThe same can be run from command line:benchmark_run test_data/time/benchmark_functions.py --save_dir /tmp/ --comment "million items" --extra_fields num_itemsBenchmark model resultsfrombenchmarkitimportbenchmark,benchmark_runfromsklearn.datasetsimportload_irisfromsklearn.linear_modelimportLogisticRegressionMODEL_BENCHMARK_SAVE_FILE='/tmp/benchmark_model.jsonl'x,y=load_iris(return_X_y=True)@benchmark(save_params=True,save_output=True)deflog_regression(C=1.0,fit_intercept=True):clf=LogisticRegression(random_state=0,solver='lbfgs',multi_class='multinomial',C=C,fit_intercept=fit_intercept,)clf.fit(x,y)score=clf.score(x,y)return{'score':score}model_benchmark_results=benchmark_run(log_regression,MODEL_BENCHMARK_SAVE_FILE,comment='baseline model',extra_fields=['C','fit_intercept'],metric='score',bigger_is_better=True,)Change hyperparameterC=0.5and rerun. Output:The same can be run from command line:benchmark_run file_with_benchmark.py --save_dir /tmp/ --comment "stronger regularization" --extra_fields C fit_intercept --metric score --bigger_is_betterAnalyze results from the filefrombenchmarkitimportbenchmark_analyzeSAVE_PATH='/tmp/benchmark_time.jsonl'benchmark_df=benchmark_analyze(SAVE_PATH,func_name=None,rows_limit=10,metric='mean_time',bigger_is_better=False,extra_fields=['num_items'],)input_path- path to.jsonlfile or directory with.jsonlfiles with benchmark resultsfunc_name- display statistics for particular function. IfNonethen all functions, stored in file, are displayed. DefaultNonerows_limit- limit table rows in console output. Default10metric- metric which is used for comparison. Defaultmean_timebigger_is_better- whether bigger value of metric indicates that result is better. For time benchmarks should beFalse, for model accuracy should beTrue. DefaultFalseextra_fields- extra fields to include in console outputPrints to terminal and returns pandasDataFrame.The same can be run from command line:benchmark_analyze /tmp/benchmark_time.jsonl --extra_fields num_itemsOther examples