Package: waterfalls
WaterfallsVisualizes duration and CPU usage of blocks of code to let you see bottlenecks in your applications.To use Waterfalls, you mark the blocks of code in your application that you want to time and then run the application as you normally do.
Waterfalls runs in the background and collects the data.
When your application exits, Waterfalls automatically generates and saves a report.To see the report, you use thewaterfallscommand which shows the timings as a waterfall diagram.
It is much easier to understand the application flow when youseeit, rather than trying to decipher timestamps printed to the console.Waterfalls handles also applications that use threading and multiprocessing.The tool is very lightweight and has a negligible impact on the main program.
All slower tasks (e.g., saving reports to disk) are delayed untilafterthe main program finishes.Basic exampleimporttarfileimporturllib.requestfromwaterfallsimportTimerURLS=["https://www.python.org/","https://pypi.org/","https://github.com/",]fori,urlinenumerate(URLS):withTimer("Download site"):withurllib.request.urlopen(url)asconn:html=conn.read()withTimer("Save as HTML"):withopen(f"{i}.html","wb")asfile:file.write(html)withTimer("Save as TAR"):withtarfile.open(f"{i}-xz.tar","w:xz")astar:tar.add(f"{i}.html")The program downloads sites from a list of URLs, saves each to an HTML file, and finally compresses the HTML file into a TAR file.
We usewaterfalls.Timerto define each of the 3 logical steps.When we run the program, Waterfalls automatically saves awaterfalls.jsonfile to the current working directory.We can now type thewaterfallscommand which will display it as an interactive diagram.Looking at the diagram, we immediatelyseethat the program spends a lot of time downloading each site.
During these periods, nothing else is happening, the CPU usage is very low and the program is generally wasting time.
We can certainly improve on that.
In the sectionAdvanced examples, we will refactor the program to take advantage of multiple threads and use Waterfalls to analyze the refactored program.InstallationWaterfalls requires Python 3.8+ to run.pipinstallwaterfallsUsageStep 1/2: generating recordsMark the sections of the code you want to measure.First, create one or moretimers. Atimertypically represents one logical activity, for example,Download site.
Thetimercorresponds to one row in the diagram. It gives the row its name.Onetimercan have one or moreblocks, for example, one for each site that is being downloaded.
Eachblockmust be started (start()) and stopped (stop()).
Allblockswill be placed on thetimerrow from left to right, showing exactly when they were started and stopped.fromwaterfallsimportTimert=Timer("Download site")t.start()# Download site At.stop()t.start()# Download site Bt.stop()For convenience,Timercan also be used as a context manager or as a decorator.withTimer("Download site"):# Download site C@Timer("Download site")defdownload_function():# Download site DAdd informative text to timing blocksSometimes it can be useful to add customtextto the timingblocks, for example, to record the concrete value that is being processed.
Thistextwill then be displayed in the diagram on top of its correspondingblock.Thetextcan be defined when calling thestart()method.t=Timer("Download site")t.start(text="www.python.org")# Download www.python.orgt.stop()t.start(text="pypi.org")# Download pypi.orgt.stop()It can also be defined when usingTimeras a context manager or decorator.withTimer("Download site",text="www.python.org"):# Download www.python.orgwithTimer("Download site",text="pypi.org"):# Download pypi.org@Timer("Download site",text="www.python.org")defdownload_function():# Download www.python.org@Timer("Download site",text="pypi.org")defdownload_function():# Download pypi.orgSometimes thetextisn't known at the moment of theblock'sstart. For example, it may be obtained during theblock'sexecution.
In this case, you can define it in thestop()method.t=Timer("Download site")t.start()# Download github.com# Save response size to `bytes_len`t.stop(text=bytes_len)When usingTimeras a context manager,textcan also be set after initiation - using an instance attribute.withTimer("Download site")ast:# Download github.com# Save response size to `bytes_len`t.text=bytes_lenNote that whiletextcan be set at nearly every step of theblock'slifecycle, only the last value will be saved in the report.Change the reports directoryBy default, the report file will be saved to the current working directory.
To choose a different directory, you can setTimer.directoryanywhere in the code.Timer.directory="/path/to/reports/"You can also use theWATERFALLS_DIRECTORYenvironment variable.exportWATERFALLS_DIRECTORY=/path/to/reports/Step 2/2: viewing diagramsTo see the diagram, use thewaterfallscommand. By default, it will look for report files in the current working directory.
If you want to search in another location, you can specify the directory.waterfalls"/path/to/reports/"The command supports the following arguments.Positional arguments:directory: Directory containing report file(s) generated by waterfalls.Timer. By default, the current working directory is used.Optional arguments:-u,--unit{nsec,usec,msec,sec,min,hour}: Specifies time unit. By default, the time unit is determined automatically.-t,--thread_id: Shows thread ID next to each timer. By default, thread ID is hidden and only shown to distinguish overlapping blocks belonging to the same timer.-l,--lines: Shows horizontal lines in the graph.-i,--image: Saves diagram as an image (waterfalls.svg) to the reports directory instead of launching interactive window.Advanced examplesMultiple threadsWhen we timed the program in the sectionBasic example, we noticed that it spends a lot of time downloading the websites and the CPU usage is very low.
This type of program should benefit from using multiple threads so we will refactor it and useThreadPoolExecutor.We will instruct Waterfalls to save the report file to the./thread_pool_records/directory.We will also pass theDownload sitetimeratextcontaining the domain that is being downloaded so the diagram becomes even clearer.importconcurrent.futuresimporttarfilefromurllib.parseimporturlparseimporturllib.requestfromwaterfallsimportTimerURLS=["https://www.python.org/","https://pypi.org/","https://github.com/",]Timer.directory="./thread_pool_records/"defsave_site(i,url):netloc=urlparse(url).netlocwithTimer("Download site",text=netloc):withurllib.request.urlopen(url)asconn:html=conn.read()withTimer("Save as HTML"):withopen(f"{i}.html","wb")asfile:file.write(html)withTimer("Save as TAR"):withtarfile.open(f"{i}-xz.tar","w:xz")astar:tar.add(f"{i}.html")withconcurrent.futures.ThreadPoolExecutor(max_workers=len(URLS))asexecutor:executor.map(save_site,range(len(URLS)),URLS)After we run the program, there will be a new directory (thread_pool_records) in our current working directory.
We can now launch the Waterfalls viewer. We will also use the--thread_idflag to tell Waterfalls to show the thread ID belonging to eachtimer.waterfalls"./thread_pool_records/"--thread_idMain observations:We canseethat all sites are being downloaded concurrently, reducing the total time needed to run the program.It takes around 350 ms to download, save and compress all 3 sites. (It was over 520 ms for the single-threaded version.)pypi.orgis the quickest site to be downloaded,github.comis the slowest.Creating the TAR files is very CPU-intensive. However, multiprocessing would not help here since the TARs are created at different times and do not compete for the CPU.Multiple processesWe have a program that determines if a number from a list is prime or not. We will addwaterfalls.TimercalledDetermine primeto time the program.importmathfromwaterfallsimportTimerNUMBERS=[112272535095293,112582705942171,115280095190773,115797848077099,1099726899285419]Timer.directory="./prime_records/"fornumberinNUMBERS:withTimer("Determine prime")ast:not_prime=Falseifnumber>1:foriinrange(2,math.floor(math.sqrt(number)+1)):if(number%i)==0:not_prime=Truebreakt.text=f"{number}\nnot prime"ifnot_primeelsef"{number}\nis prime"After the program finishes, we will launch the viewer.waterfalls"./prime_records/"We can immediatelyseethat determining whether a number is prime or not uses almost 100% CPU.
But the computer has multiple CPU cores so we can take advantage of multiprocessing to speed up the program.
We will use theProcessPoolExecutor.
We will also change the reports directory to./prime_multiprocessing_records/.importconcurrent.futuresimportmathfromwaterfallsimportTimerNUMBERS=[112272535095293,112582705942171,115280095190773,115797848077099,1099726899285419]Timer.directory="./prime_multiprocessing_records/"defdetermine_prime(number):withTimer("Determine prime")ast:not_prime=Falseifnumber>1:foriinrange(2,math.floor(math.sqrt(number)+1)):if(number%i)==0:not_prime=Truebreakt.text=f"{number}\nnot prime"ifnot_primeelsef"{number}\nis prime"if__name__=="__main__":withconcurrent.futures.ProcessPoolExecutor(max_workers=2)asexecutor:executor.map(determine_prime,NUMBERS)After the program finishes, we can open the./prime_multiprocessing_records/directory where we will find 2 report files - one for each process.
They are differentiated by their process ID:waterfalls.<process_id>.json.
We use the viewer the same way we did before - the viewer automatically combines all record files in the directory into a unified view.waterfalls"./prime_multiprocessing_records/"Main observations:We canseethat the refactored program can analyze 2 numbers in parallel.It takes around 4.7 s to finish the analysis. (It was over 11 s for the version using a single process.)LicenseMIT license