Package: progress-tracker
progress_trackeris an easy and flexible way to print custom progress messages while processing streams of events on the CLI.It was originally developed atexactEarth Ltd. Seethis presentationtoDevHouse Waterloofor the original motivation.Built and tested with Python 3.6+ContentsQuick StartUsageExamplesPrint after every n records are processedPrint after every x percent of records are processedPrint every n records OR every n seconds during processingCombining trigger conditionsReport Creation InvariantsCustomizing the report formatting / InternationalizationReport values availableCustomizing the print behaviourevery_n_seconds_idleDifference betweenevery_n_secondsandevery_n_seconds_idleAccessing tracker after processingOther ResourcesQuick Start%pipinstallprogress_tracker>>>fromprogress_trackerimporttrack_progress>>>for_intrack_progress(list(range(1000)),every_n_records=100):...continue...100/1000(10.0%)in0:00:00.000114(Timeleft:0:00:00.001026)200/1000(20.0%)in0:00:00.000274(Timeleft:0:00:00.001096)300/1000(30.0%)in0:00:00.000374(Timeleft:0:00:00.000873)400/1000(40.0%)in0:00:00.000473(Timeleft:0:00:00.000710)500/1000(50.0%)in0:00:00.000572(Timeleft:0:00:00.000572)600/1000(60.0%)in0:00:00.000671(Timeleft:0:00:00.000447)700/1000(70.0%)in0:00:00.000770(Timeleft:0:00:00.000330)800/1000(80.0%)in0:00:00.000868(Timeleft:0:00:00.000217)900/1000(90.0%)in0:00:00.000979(Timeleft:0:00:00.000109)1000in0:00:00.001086Usageprogress_trackeris very customizable to fit your desires, but tries to have sensible defaults.The core ofprogress_trackeris a method calledtrack_progress.
By changing the parameters passed totrack_progress, you can customize how frequently (and with what messages) the tracker will report.deftrack_progress(iterable:Iterable[T],# The iterable to iterate overtotal:Optional[int]=None,# Override for the total message count, defaults to len(iterable)callback:Callable[[str],Any]=print,# A function (f(str) -> None) that gets called each time a condition matchesformat_callback:Callable[[Dict[str,Any],Set[str]],str]=default_format_callback,# A function (f(str) -> str) that formats the progress values into a string.every_n_percent:Optional[float]=None,# Reports after every n percentevery_n_records:Optional[int]=None,# Reports every n recordsevery_n_seconds:Optional[float]=None,# Reports every n secondsevery_n_seconds_idle:Optional[float]=None,# Report if there has not been a record processed in the past n seconds. Useful for infinite streams.every_n_seconds_since_report:Optional[float]=None,# Report if there hasn’t been any report in the past n seconds.report_first_record:bool=False,# Report after the first recordreport_last_record:bool=False# Report after the last record)->NoneExamplesPrint after every n records are processedTheevery_n_recordsparameter will trigger a report after every nth record is processed.>>>fromprogress_trackerimporttrack_progress>>>>>>for_intrack_progress(list(range(1000)),every_n_records=100):...continue...100/1000(10.0%)in0:00:00.000114(Timeleft:0:00:00.001026)200/1000(20.0%)in0:00:00.000274(Timeleft:0:00:00.001096)300/1000(30.0%)in0:00:00.000374(Timeleft:0:00:00.000873)400/1000(40.0%)in0:00:00.000473(Timeleft:0:00:00.000710)500/1000(50.0%)in0:00:00.000572(Timeleft:0:00:00.000572)600/1000(60.0%)in0:00:00.000671(Timeleft:0:00:00.000447)700/1000(70.0%)in0:00:00.000770(Timeleft:0:00:00.000330)800/1000(80.0%)in0:00:00.000868(Timeleft:0:00:00.000217)900/1000(90.0%)in0:00:00.000979(Timeleft:0:00:00.000109)1000in0:00:00.001086Print after every x percent of records are processedTheevery_n_percentparameter will trigger a report after every nth percent of records are processed.>>>fromprogress_trackerimporttrack_progress>>>for_intrack_progress(list(range(1000)),every_n_percent=10):...continue...100/1000(10.0%)in0:00:00.000114(Timeleft:0:00:00.001026)200/1000(20.0%)in0:00:00.000274(Timeleft:0:00:00.001096)300/1000(30.0%)in0:00:00.000374(Timeleft:0:00:00.000873)400/1000(40.0%)in0:00:00.000473(Timeleft:0:00:00.000710)500/1000(50.0%)in0:00:00.000572(Timeleft:0:00:00.000572)600/1000(60.0%)in0:00:00.000671(Timeleft:0:00:00.000447)700/1000(70.0%)in0:00:00.000770(Timeleft:0:00:00.000330)800/1000(80.0%)in0:00:00.000868(Timeleft:0:00:00.000217)900/1000(90.0%)in0:00:00.000979(Timeleft:0:00:00.000109)1000in0:00:00.001086every_n_percentonly works for bounded iterables. For unbounded iterables (ex. streams), usingevery_n_percentwill report aRuntimeWarning.At most a single report is generated per processed record. Even if processing of a single record would meet the conditions multiple times
(ex. ifevery_n_percent=10, but there are only 2 records, then processing each record causes 50%, or 5 * 10%, progress), only a single report is created (containing the latest values).Print every n records OR every n seconds during processingThis is especially useful when you have highly variable processing times (ex. most records take 2 seconds to process, but some take 20 seconds to process).
You can use theevery_n_secondsparameter to get reports between the expensive records.importtimefromprogress_trackerimporttrack_progressdefsimulated_processing(item):ifitem=='hard':time.sleep(10)variable_stream_simulation=(['easy']*15)+(['hard']*5)+(['easy']*15)foritemintrack_progress(variable_stream_simulation,every_n_records=5,every_n_seconds=10):simulated_processing(item)...5/35(14.285714285714285%)in0:00:00.000014(Timeleft:0:00:00.000084)10/35(28.57142857142857%)in0:00:00.000095(Timeleft:0:00:00.000238)15/35(42.857142857142854%)in0:00:00.000120(Timeleft:0:00:00.000160)16/35(45.714285714285715%)in0:00:10.011364(Timeleft:0:00:11.888495)17/35(48.57142857142857%)in0:00:20.022107(Timeleft:0:00:21.199878)18/35(51.42857142857142%)in0:00:30.031801(Timeleft:0:00:28.363368)19/35(54.285714285714285%)in0:00:40.041754(Timeleft:0:00:33.719372)20/35(57.14285714285714%)in0:00:50.073991(Timeleft:0:00:37.555493)25/35(71.42857142857143%)in0:00:50.074246(Timeleft:0:00:20.029698)30/35(85.71428571428571%)in0:00:50.074286(Timeleft:0:00:08.345714)35in0:00:50.074319During the processing of the slow records,track_progressreported after every record.Note: Because the default “Time left” calculation is just a simple linear extrapolation, it is not as useful in the face of such variability in processing times.Combining trigger conditionsAs seen in the previous example, you can combine multiple conditions together to dictate when a report is created.Each of the conditions are combined using an OR operator, meaning that if any condition is met, a report is created.Even if multiple conditions are met simultaneously, only a single report will be created.Report Creation InvariantsReport creation observes two invariants:At most a single report is created per processed record.Reports are only created in response to a record being processed.Customizing the report formatting / InternationalizationBy default,progress_trackerformats the report into an English language string.
This can be overriden by supplying a different function as theformat_callbackparameter totrack_progress.This can be used to perform advanced formatting, or to add internationalization/localization.defformat_en_francais(report:Dict[str,Any],reasons:Set[str]):i=report["i"]total=report["total"]iftotalisNoneori==total:format_string="{i}messages traités en{time_taken}"else:format_string="{i}/{total}messages traités en{time_taken}(temps restant:{estimated_time_remaining})"returnformat_string.format(**report)forposteintrack_progress(postes,every_n_records=100,format_callback=format_en_francais):traité(poste)(Veuillez excuser toute erreur en français. C’est le résultat de Google Translate.)Simple cases can also be done using a lambda:>>>fromprogress_trackerimporttrack_progress>>>>>>for_intrack_progress(list(range(5)),every_n_records=1,format_callback=lambda**kwargs:"Got one!"):...continue...Gotone!Gotone!Gotone!Gotone!Gotone!Report values availableThe following values are available in every report for use in theformat_callback:ValueTypeMeaning{records_seen}intThe number of records processed so far.{total}Optional[int]The total of records in the iterable, if known. ElseNone{percent_complete}Optional[float]The percentage of records processed so far.Noneif{total}isNoneorrecords_seen= 0{time_taken}timedeltaThe amount of time that processing has taken thus far.{estimated_time_remaining}Optional[timedelta]The estimated amount of time needed in order to process the rest of the records (simple linear estimate).Noneif total isNone{items_per_second}Optional[float]The number of records processed so far / the number of seconds elapsed.Noneif no time have elapsed.{idle_time}timedeltaThe amount of idle time between the previous record’s processing and this record’s arrival.Customizing the print behaviourBy default,progress_trackercalls Python’sprintfunction with the formatted report.
This can be overriden by supplying a different function as thecallbackparameter totrack_progress.every_n_seconds_idleevery_n_seconds_idleallows you to trigger a report if there is ever more thannseconds when no records were processed.Note: If processing of a single record takes longer thanevery_n_seconds_idle, then it will be triggered after every record.Difference betweenevery_n_secondsandevery_n_seconds_idleevery_n_secondstriggers a report anytime it has been more than n seconds sinceevery_n_secondslast triggered a report.every_n_seconds_idletriggers a report anytime there has not been a record processed in the past n seconds (ie. the processing has been idle).For example:After# of records processed in intervalCummulative records processedevery_n_seconds=3every_n_seconds_idle=30 seconds001 second112 seconds123 seconds13Triggered, since it is the first record T >= 3s (T >= 0s + 3s)4 seconds145 seconds156 seconds16Triggered, since it is the first record T >= 6s (T >= 3s + 3s)7 seconds168 seconds069 seconds0610 seconds0611 seconds17Triggered, since it is the first record T >= 9s (T >= 6s + 3s)Triggered, since it is the first record processed in the past 3 seconds (T >= 6s + 3s)12 seconds1813 seconds1914 seconds110Triggered, since it is the first record T >= 14s (T >= 11s + 3s)15 seconds111Note thatevery_n_secondsreports at 3 seconds and 6 seconds, as one would expect. Then it reports at 11 seconds, since that is the first time a record was processed after the 9 seconds mark.
Then note that instead of next reporting at 12 seconds (9s + 3s), it reports next at 14 seconds (11s + 3s).every_n_seconds_idleonly reported at 11 seconds, since that was the only time that a record was processed without other records being processed during the previous 3 seconds.Accessing tracker after processingBy default,track_progresshides the internalProgressTrackerobject underneath. However, in some cases you might want to be able to access the internals of the object after iteration.
In these cases, you can usetrack_progressan explicit context manager:withtrack_progress(range(0,101),every_n_percent=5)astracker:foritemintracker:process(item)final_report=tracker.create_report()print(f"Processing took{final_report['time_taken']}and processed{final_report['records_seen']}records.")Other ResourcesThis presentationtoDevHouse Waterloo.