Package: progress-executor
Known IssuesWe believe the current version of tqdm has a bug when the number of instances decreases.
If you are using future.add_tqdm_callback for progress, you should modify the _decr_instances method in tqdm/std.py line 690 of tqdm to the following (as described inthis issue):@classmethoddef_decr_instances(cls,instance):"""Remove from list and reposition another unfixed barto fill the new gap.This means that by default (where all nested bars are unfixed),order is not maintained but screen flicker/blank space is minimised.(tqdm<=4.44.1 moved ALL subsequent unfixed bars up.)"""withcls._lock:try:cls._instances.remove(instance)exceptKeyError:# if not instance.gui:  # pragma: no cover#     raisepass# py2: maybe magically removed already# else:ifnotinstance.gui:#CUSTOM Additionforinstincls._instances:pos=getattr(inst,"pos",0)adjust=1ifpos<0else-1ifposandabs(pos)>abs(instance.pos):inst.pos+=adjust#END of CUSTOM Additionlast=(instance.nrowsor20)-1# find unfixed (`pos >= 0`) overflow (`pos >= nrows - 1`)instances=list(filter(lambdai:hasattr(i,"pos")andlast<=i.pos,cls._instances))# set first found to current `pos`ifinstances:inst=min(instances,key=lambdai:i.pos)inst.clear(nolock=True)inst.pos=abs(instance.pos)MotivationWhile Concurrent.futures enables one to seemlessly launch with different executors (threadpool and processpool) the same code,
it is non trivial to add the two following features:Handling cancellation of a task currently executingHandling progress of a task from elsewhere (i.e. one can print a tqdm bar within the task, but what if one wishes to aggregate progress from different tasks?)Furthermore, we additionally add a new executor that behaves in "sync" (no threads, no processes) for testing purposesA full example is provided at the bottom of this page.ProblemCorrectly handling cancellation is more complicated than can be expected:
a thread cannot be cancelled from the outside without harm.As for progress, as soon as one wishes to do computation on the progress of different tasks one needs to retrieve the data from the different processes.
The synchronization aspects are not extremelly difficult but are still error prone.SolutionThe big pictureWhile cancellation of processes could probably (see alternatives) be handled by a reimplementation of theconcurrent.futures.ProcessPoolExecutor, we do not see a good way to do so for the ThreadPoolExecutor. As we wish to have a uniform API with similar semantics, we opted for another solution.This solution has the drawback of requiring the programmer to give regurlar progress information, but we believe this is a relatively low drawback as the programmer should already be doing so.
We then use the regurlar progress calls to check for cancellation and cancel from within the task if necessary.A limitation for progress callsAs we wish progress updates to happen in the main thread, and yet not block whatever else the main thread is doing, we implemented it as a async coroutine.
This means that the main thread should be running as asyncio loop (this is probably already the case if you wanted to use concurrent.futures) where fairness is respected.APIUsing ProgressExecutorsOur executors inherit for the concurrent.futures.Executor. We provide three executors that can be created (for example) with the following lines:fromprogress_executorimportThreadPoolProgressExecutor,ProcessPoolProgressExecutor,SyncProgressExecutortp=ThreadPoolProgressExecutor(max_workers=3)pp=ProcessPoolProgressExecutor(max_workers=3)se=SyncProgressExecutor()To create tasks, one should use the submit function of concurrent.futures.Executor, and we highly recommand creating a task from that future (as suggested by modern asyncio).
The use of asyncio.TaskGroup in an async with block is highly recommanded (see asyncio docs)withexecutor:asyncwithasyncio.TaskGroup()astg:future1=executor.submit(f1,*args,**kwargs)task1=tg.create_task(future1)future2=executor.submit(f2,*args,**kwargs)task2=tg.create_task(future2)The following lines runs the task, but without any changes compared to using the executors of concurrent.futures (except for cancellation). This is because we have not taken advantage that the futures returned by a the submit method of ProgressExecutor are ProgressFuture.ProgressFuture MethodTo correctly use ProgressFuture, there three modifications that are required:Usetg.create_task(future.check_for_progress())instead oftg.create_task(future). This adds the progress callbacks to the asyncio loop.Add callbacks on progress. You can either usefuture.add_progress_callback(old_state, new_state), where state is a dictionary with itemsn, total, status,
or usefuture.add_tqdm_callback(tqdm_cls=tqdm.tqdm, init_kwargs = {}, trigger: Set[Literal["now", "running", "cancelled"]] = {"now"} )wheretqdm_clsshould be a class/function (not instance)
that has similar API to tqdm.tqdm,init_kwargsis the initializer arguments fortqdm_clsandtriggerstates whentqdm_cls(**init_kwargs)is called.Add a progress argument to your function.Full Exampleimportlogging,beautifulloggerimportsys,time,asyncio,tqdmfromprogress_executorimport*logger=logging.getLogger(__name__)deflong_compute(n):tot=17foriinrange(int(n*25000000)):tot=tot//2iftot%2==0else3*tot+1returntotdeff(n,progress:Updater):progress.total=nforiinprogress(range(2*n)):#you can use progress directly on an iteratorifi%2==0:long_compute(0.1)else:time.sleep(0.1)returnntp=ThreadPoolProgressExecutor(max_workers=3)pp=ProcessPoolProgressExecutor(max_workers=3)se=SyncProgressExecutor()executor=tp#Change here to see the differencesasyncdefmain():vals=[30,40,35,60,20,50,38,27]*2try:withexecutor:asyncwithasyncio.TaskGroup()astg:tasks=[]fori,valinenumerate(vals):fut=executor.submit(f,val)fut.add_tqdm_callback(tqdm.tqdm,dict(desc=f"Task{i}"),triggers=["now","running","cancelled"])tasks.append(tg.create_task(fut.check_for_progress()))#See what happends if you uncomment these two lines# await asyncio.sleep(2)# tasks[0].cancel()finally:print("FINISHED")fori,(val,task)inenumerate(zip(vals,tasks)):print(f"Task{i}with val={val}has result{'cancelled'iftask.cancelled()elsetask.result()}")if__name__=="__main__":#Necessary due to multiprocessingbeautifullogger.setup(warning_level_modules=["asyncio"])#Just for pretty printinglogger.info("Running start")asyncio.run(main())logger.info("Running end")