Package: yappi
YappiA tracing profiler that ismultithreading, asyncio and geventaware.HighlightsFast: Yappi is fast. It is completely written in C and lots of love and care went into making it fast.Unique: Yappi supports multithreaded,asyncioandgeventprofiling. Tagging/filtering multiple profiler results has interestinguse cases.Intuitive: Profiler can be started/stopped and results can be obtained from any time and any thread.Standards Compliant: Profiler results can be saved incallgrindorpstatformats.Rich in Feature set: Profiler results can show eitherWall Timeor actualCPU Timeand can be aggregated from different sessions. Various flags are defined for filtering and sorting profiler results.Robust: Yappi has been around for years.MotivationCPython standard distribution comes with three deterministic profilers.cProfile,Profileandhotshot.cProfileis implemented as a C module based onlsprof,Profileis in pure Python andhotshotcan be seen as a small subset of a cProfile. The major issue is that all of these profilers lack support for multi-threaded programs and CPU time.If you want to profile a  multi-threaded application, you must give an entry point to these profilers and then maybe merge the outputs. None of these profilers are designed to work on long-running multi-threaded applications. It is also not possible to profile an application that start/stop/retrieve traces on the fly with these profilers.Now fast forwarding to 2019: With the latest improvements onasynciolibrary and asynchronous frameworks, most of the current profilers lacks the ability to show correct wall/cpu time or even call count information per-coroutine. Thus we need a different kind of approach to profile asynchronous code. Yappi, with v1.2 introduces the concept ofcoroutine profiling. Withcoroutine-profiling, you should be able to profile correct wall/cpu time and call count of your coroutine. (including the time spent in context switches, too). You can see detailshere.InstallationCan be installed via PyPI$ pip install yappiOR from the source directly.$ pip install git+https://github.com/sumerc/yappi#egg=yappiExamplesA simple example:importyappidefa():for_inrange(10000000):# do something CPU heavypassyappi.set_clock_type("cpu")# Use set_clock_type("wall") for wall timeyappi.start()a()yappi.get_func_stats().print_all()yappi.get_thread_stats().print_all()'''Clock type: CPUOrdered by: totaltime, descname                                  ncall  tsub      ttot      tavgdoc.py:5 a                            1      0.117907  0.117907  0.117907name           id     tid              ttot      scnt_MainThread    0      139867147315008  0.118297  1'''Profile a multithreaded application:You can profile a multithreaded application via Yappi and can easily retrieve
per-thread profile information by filtering onctx_idwithget_func_statsAPI.importyappiimporttimeimportthreading_NTHREAD=3def_work(n):time.sleep(n*0.1)yappi.start()threads=[]# generate _NTHREAD threadsforiinrange(_NTHREAD):t=threading.Thread(target=_work,args=(i+1,))t.start()threads.append(t)# wait all threads to finishfortinthreads:t.join()yappi.stop()# retrieve thread stats by their thread id (given by yappi)threads=yappi.get_thread_stats()forthreadinthreads:print("Function stats for (%s) (%d)"%(thread.name,thread.id))# it is the Thread.__class__.__name__yappi.get_func_stats(ctx_id=thread.id).print_all()'''Function stats for (Thread) (3)name                                  ncall  tsub      ttot      tavg..hon3.7/threading.py:859 Thread.run  1      0.000017  0.000062  0.000062doc3.py:8 _work                       1      0.000012  0.000045  0.000045Function stats for (Thread) (2)name                                  ncall  tsub      ttot      tavg..hon3.7/threading.py:859 Thread.run  1      0.000017  0.000065  0.000065doc3.py:8 _work                       1      0.000010  0.000048  0.000048Function stats for (Thread) (1)name                                  ncall  tsub      ttot      tavg..hon3.7/threading.py:859 Thread.run  1      0.000010  0.000043  0.000043doc3.py:8 _work                       1      0.000006  0.000033  0.000033'''Different ways to filter/sort stats:You can usefilter_callbackonget_func_statsAPI to filter on functions, modules
or whatever available inYFuncStatobject.importpackage_aimportyappiimportsysdefa():passdefb():passyappi.start()a()b()package_a.a()yappi.stop()# filter by module objectcurrent_module=sys.modules[__name__]stats=yappi.get_func_stats(filter_callback=lambdax:yappi.module_matches(x,[current_module]))# x is a yappi.YFuncStat objectstats.sort("name","desc").print_all()'''Clock type: CPUOrdered by: name, descname                                  ncall  tsub      ttot      tavgdoc2.py:10 b                          1      0.000001  0.000001  0.000001doc2.py:6 a                           1      0.000001  0.000001  0.000001'''# filter by function objectstats=yappi.get_func_stats(filter_callback=lambdax:yappi.func_matches(x,[a,b])).print_all()'''name                                  ncall  tsub      ttot      tavgdoc2.py:6 a                           1      0.000001  0.000001  0.000001doc2.py:10 b                          1      0.000001  0.000001  0.000001'''# filter by module namestats=yappi.get_func_stats(filter_callback=lambdax:'package_a'inx.module).print_all()'''name                                  ncall  tsub      ttot      tavgpackage_a/__init__.py:1 a             1      0.000001  0.000001  0.000001'''# filter by function namestats=yappi.get_func_stats(filter_callback=lambdax:'a'inx.name).print_all()'''name                                  ncall  tsub      ttot      tavgdoc2.py:6 a                           1      0.000001  0.000001  0.000001package_a/__init__.py:1 a             1      0.000001  0.000001  0.000001'''Profile an asyncio application:You can see that coroutine wall-time's are correctly profiled.importasyncioimportyappiasyncdeffoo():awaitasyncio.sleep(1.0)awaitbaz()awaitasyncio.sleep(0.5)asyncdefbar():awaitasyncio.sleep(2.0)asyncdefbaz():awaitasyncio.sleep(1.0)yappi.set_clock_type("WALL")withyappi.run():asyncio.run(foo())asyncio.run(bar())yappi.get_func_stats().print_all()'''Clock type: WALLOrdered by: totaltime, descname                                  ncall  tsub      ttot      tavgdoc4.py:5 foo                         1      0.000030  2.503808  2.503808doc4.py:11 bar                        1      0.000012  2.002492  2.002492doc4.py:15 baz                        1      0.000013  1.001397  1.001397'''Profile a gevent application:You can use yappi to profile greenlet applications now!importyappifromgreenletimportgreenletimporttimeclassGreenletA(greenlet):defrun(self):time.sleep(1)yappi.set_context_backend("greenlet")yappi.set_clock_type("wall")yappi.start(builtins=True)a=GreenletA()a.switch()yappi.stop()yappi.get_func_stats().print_all()'''name                                  ncall  tsub      ttot      tavgtests/test_random.py:6 GreenletA.run  1      0.000007  1.000494  1.000494time.sleep                            1      1.000487  1.000487  1.000487'''DocumentationIntroductionClock TypesAPICoroutine Profiling(new in 1.2)Greenlet Profiling(new in 1.3)Note: Yes. I know I should be moving docs to readthedocs.io. Stay tuned!Related TalksSpecial thanks to A.Jesse Jiryu Davis:Python Performance Profiling: The Guts And The Glory (PyCon 2015)PyCharm IntegrationYappi is the default profiler inPyCharm. If you have Yappi installed,PyCharmwill use it. Seethe officialdocumentation for more details.