Package: perf-tool
perf_toolAnothertime performancetool for python.
It is the right support tool if you need to profile an entire program, some piece of code,
a module, a class, a function or a single row. The idea here is to profile only the codeyou wantto profile.Installationperf_tool is tested on python 2.7, 3.6, 3.7, 3.8> pip install perf_toolAfter installation you can try to run a little example:$ curl https://raw.githubusercontent.com/glaucouri/perf_tool/master/example.py > perf_tool_sample.py
$ python perf_tool_sample.py 
================== PerfTool ==================
task                     |aver(s) |sum(s)  |count   |std     
main                     |   0.065|   0.065|       1|   0.000
  +-call                 |   0.050|   0.050|       1|   0.000
    +-body               |   0.050|   0.050|       1|   0.000
  +-row                  |   0.001|   0.014|      10|   0.000

overall                  |    0.04|    0.06|      13|-DescriptionI have often been faced with the problem of performance on very challanging scope.In many cases difficult if given by:intrinsic complexity of algorithms, in this cases, piece of codes can't be explained with a right asymptotic
representations akaBig O notationdomain not fully known, yes! if you don't know exacty how and why the developer has done some choices each row of code
can be interpreted as an error candidate.coupling with data, sometimes algorithms is too much
intricated with data, the result is an unreadable spaghetti code.use of external framework like Pandas, Scipy, Numpy. In these cases is not simple get a correct idea of where is spent
computational time.In these casescomputational analysiscan be misleading,
on the other handspython profileris too dispersive so juggling
with performance tools give to us very counterintuitive reports.How it worksWhen you profile a program your need is to construct a metric that can tell you where the time is spent.
So at the beginning you must putsentinelsin the entry point in order to create main metrics.
Gradually you put othersentinelsin the most relevant section of codes, increasing the capillarity
until individual lines of code are identified.perf_toolhas 2 sentinel:> from perf_tool import PerfTool, perf_toolPerfToolIs a context managerperf_toolIs a function / method decoratoreachSentinelwill cover a piece of code, and for that, records how many time is spent on it.With a function we can print a report containing 4 columns:task: task label or name, it is set when you put the sentinel, i suggest to use mnemonic labelaver(s): mean of time spent ove the piece of code.sum(s): sum of time spent ove the piece of code.count: how many times that sentinel was calledstd: standard deviationSentinelscan be nested in order to represent the hierarchy of codeExamplesFirst step is to put root sentinels:@perf_tool('main')
def main():
  <some code>Second step is to place sentinel in subset of code.@perf_tool('main')
def main():
  with PerfTool('preparation'):
    <some code part 1>

  with PerfTool('calculus'):
    <some code part 2>

  with PerfTool('output'):
    <some code part 3>And so on@perf_tool('main')
def main():
  with PerfTool('preparation'):
    <some code part 1>

  with PerfTool('calculus'):
    <some code part 2>

  with PerfTool('output'):
    <some code part 3a>
    for each row:
    with PerfTool('row write'):
      <some code part 3b>In the last example output is (times are mock)task                     |aver(s) |sum(s)  |count   |std     
main                     |   0.209|   0.209|       1|   0.000
  +-preparation          |   0.013|   0.013|       1|   0.000
  +-calculus             |   0.030|   0.030|       1|   0.000
  +-output               |   0.166|   0.166|       1|   0.000
    +-row write          |   0.001|   0.143|     100|   0.000

overall                  |    0.08|    0.21|     104|-Here a production use.task                     |aver(s) |sum(s)  |count   |std     
Definition set value     |   0.006| 111.786|   17199|   0.013
  +-Comp3 call           |   0.003|   0.043|      13|   0.000
  +-Comp2 call           |   0.009|  37.078|    4056|   0.009
    +-calculate cache    |   0.018|  36.941|    2016|   0.002
      +-aggregate        |   0.012|  23.772|    2016|   0.002
  +-Comp3   call         |   0.083|  30.259|     364|   0.031
    +-aggregate          |   0.091|  16.431|     180|   0.011
    +-agg+pivot          |   0.092|  13.200|     144|   0.011
  +-Comp1 call           |   0.001|   1.607|    2626|   0.002
  +-Comp4  call          |   0.005|   0.061|      13|   0.000
Comp2 set value          |   1.054|  41.113|      39|   1.497
  +-Comp2 compute meas   |   3.163|  41.113|      13|   0.244
    +-Comp2 tmx          |   0.301|  15.675|      52|   0.014
      +-Broker call      |   0.005|   9.377|    1872|   0.001
        +-Comp3 call     |   0.003|   6.372|    1872|   0.001
    +-Comp2 Comp3        |   0.097|   4.650|      48|   0.124
      +-Broker call      |   0.029|   4.220|     144|   0.041
        +-Comp3   call   |   0.028|   3.985|     144|   0.041
          +-aggregate    |   0.086|   3.851|      45|   0.010
    +-Comp2 Comp1        |   0.056|   2.670|      48|   0.008
      +-Broker call      |   0.003|   1.215|     432|   0.002
        +-Comp1 call     |   0.001|   0.522|     432|   0.002
Comp4  set value         |   0.665|  17.300|      26|   0.719
  +-Comp4  compute all   |   1.441|  17.296|      12|   0.037
    +-call transform     |   0.415|  14.935|      36|   0.019
      +-Broker call      |   0.010|  11.894|    1152|   0.009
        +-Comp2 call     |   0.009|  10.519|    1152|   0.009
          +-calculate cac|   0.018|  10.470|     576|   0.003
            +-agg all    |   0.012|   7.000|     576|   0.002
    +-retrieve data      |   0.003|   0.448|     144|   0.000