Package: pypbars
pypbarsThepypbarsmodule provides a convenient way to display progress bars for concurrentasyncioormultiprocessing Poolprocesses. Thepypbarsclass is a subclass oflist2termthat displays a list to the terminal, and usesprogress1barto render the progress bar.Installationpipinstallpypbarsexample1 - ProgressBars with asyncioCreateProgressBarsusing a lookup list containing unique values, these identifiers will be used to get the index of the appropriateProgressBarto be updated. The convention is for the function to includelogger.writecalls containing the identifier and a message for when and how the respective progress bar should be updated. In this example the defaultregexdict is used but the caller can specify their own, so long as it contains regular expressions for how to detect whentotal,countand optionalaliasare set.CodeimportasyncioimportrandomfromfakerimportFakerfrompypbarsimportProgressBarsasyncdefdo_work(worker,logger=None):logger.write(f'{worker}->worker is{worker}')total=random.randint(10,65)logger.write(f'{worker}->processing total of{total}items')forcountinrange(total):# mimic an IO-bound processawaitasyncio.sleep(.1)logger.write(f'{worker}->processed{count}')returntotalasyncdefrun(workers):withProgressBars(lookup=workers,show_prefix=False,show_fraction=False)aslogger:doers=(do_work(worker,logger=logger)forworkerinworkers)returnawaitasyncio.gather(*doers)defmain():workers=[Faker().user_name()for_inrange(10)]print(f'Total of{len(workers)}workers working concurrently')results=asyncio.run(run(workers))print(f'The{len(workers)}workers processed a total of{sum(results)}items')if__name__=='__main__':main()example2 - ProgressBars with multiprocessing PoolThis example demonstrates howpypbarscan be used to display progress bars from processes executing in amultiprocessing Pool. Thelist2term.multiprocessingmodule contains apool_mapmethod that fully abstracts the required multiprocessing constructs, you simply pass it the function to execute, an iterable containing the arguments to pass each process, and an instance ofProgressBars. The method will execute the functions asynchronously, update the progress bars accordingly and return a multiprocessing.pool.AsyncResult object. Each progress bar in the terminal represents a background worker process.If you do not wish to use the abstraction, thelist2term.multiprocessingmodule contains helper classes that facilitate communication between the worker processes and the main process; theQueueManagerprovide a way to create aLinesQueuequeue which can be shared between different processes. Refer toexample2bfor how the helper methods can be used.Notethe function being executed must accept aLinesQueueobject that is used to write messages via itswritemethod, this is the mechanism for how messages are sent from the worker processes to the main process, it is the main process that is displaying the messages to the terminal. The messages must be written using the format{identifier}->{message}, where {identifier} is a string that uniquely identifies a process, defined via the lookup argument toProgressBars.CodeimporttimefrompypbarsimportProgressBarsfromlist2term.multiprocessingimportpool_mapfromlist2term.multiprocessingimportCONCURRENCYdefis_prime(num):ifnum==1:returnFalseforiinrange(2,num):if(num%i)==0:returnFalseelse:returnTruedefcount_primes(start,stop,logger):workerid=f'{start}:{stop}'logger.write(f'{workerid}->worker is{workerid}')logger.write(f'{workerid}->processing total of{stop-start}items')primes=0fornumberinrange(start,stop):ifis_prime(number):primes+=1logger.write(f'{workerid}->processed{number}')logger.write(f'{workerid}->{workerid}processing complete')returnprimesdefmain(number):step=int(number/CONCURRENCY)iterable=[(index,index+step)forindexinrange(0,number,step)]lookup=[':'.join(map(str,item))foriteminiterable]progress_bars=ProgressBars(lookup=lookup,show_prefix=False,show_fraction=False,use_color=True)# print to screen with progress bars contextresults=pool_map(count_primes,iterable,context=progress_bars)# print to screen without progress bars context# results = pool_map(count_primes, iterable)# do not print to screen# results = pool_map(count_primes, iterable, print_status=False)returnsum(results.get())if__name__=='__main__':start=time.perf_counter()number=50_000result=main(number)stop=time.perf_counter()print(f"Finished in{round(stop-start,2)}seconds\nTotal number of primes between 0-{number}:{result}")example3 - resettable ProgressBars with multiprocessing Pool and QueueThis example demonstrates howpypbarscan be used to display progress bars from a small set processes executing in amultiprocessing Poolworking on large amount of data defined in a shared work Queue. The workers will pop off the work from work queue and process it until there is no more work left in the work Queue. Since the workers are working on multiple sets the ProgressBar is reset everytime a worker begins work on a new set.  The ProgressBar maintains the number of iterations it has completed.Codeimporttime,random,loggingfrommultiprocessingimportQueuefromqueueimportEmptyimportnamesfromfakerimportFakerfrommultiprocessingimportPoolfrommultiprocessingimportget_contextfrommultiprocessingimportcpu_countfromlist2term.multiprocessingimportLinesQueuefromlist2term.multiprocessingimportQueueManagerfromqueueimportEmptyfrompypbarsimportProgressBarslogger=logging.getLogger(__name__)defprepare_queue(queue):for_inrange(55):queue.put({'total':random.randint(100,150)})defdo_work(worker_id,total,logger):logger.write(f'{worker_id}->worker is{names.get_full_name()}')logger.write(f'{worker_id}->processing total of{total}items')forindexinrange(total):# simulate work by sleepingtime.sleep(random.choice([.001,.003,.008]))logger.write(f'{worker_id}->processed{index}')returntotaldefrun_q(worker_id,queue,logger):result=0whileTrue:try:total=queue.get(timeout=1)['total']result+=do_work(worker_id,total,logger)logger.write(f'{worker_id}->reset')exceptEmpty:breakreturnresultdefmain(processes):QueueManager.register('LinesQueue',LinesQueue)QueueManager.register('Queue',Queue)withQueueManager()asmanager:queue=manager.LinesQueue(ctx=get_context())data_queue=manager.Queue()prepare_queue(data_queue)withPool(processes)aspool:print(f">> Adding{data_queue.qsize()}sets into a data queue that{processes}workers will work from until empty")process_data=[(Faker().name(),data_queue,queue)forindexinrange(processes)]results=pool.starmap_async(run_q,process_data)lookup=[f'{data[0]}'fordatainprocess_data]withProgressBars(lookup=lookup,show_prefix=False,show_fraction=False,use_color=True,show_duration=True,clear_alias=True)aslines:whileTrue:try:item=queue.get(timeout=.1)ifitem.endswith('->reset'):index,message=lines.get_index_message(item)lines[index].reset(clear_alias=False)else:lines.write(item)exceptEmpty:ifresults.ready():forindex,_inenumerate(lines):lines[index].complete=Truebreakreturnsum(results.get())if__name__=='__main__':processes=3results=main(processes)print(f">>{processes}workers processed a total of{results}items")DevelopmentClone the repository and ensure the latest version of Docker is installed on your development server.Build the Docker image:dockerimagebuild\-t\pypbars:latest.Run the Docker container:dockercontainerrun\--rm\-it\-v$PWD:/code\pypbars:latest\bashExecute the build:pyb-X