Package: meanwhile
meanwhile - Very Easy MultithreadingIf you want to call the same function on many inputs, in a multithreaded
fashion,meanwhilemakes it easy.It can make your code significantly faster, especially if the function requires
I/O operations, like file access or HTTP(S) queries.Installationpip3installmeanwhileSimple Usage ExampleSuppose you have a function namedtest_url, that gets a URL, downloads
its content, and tests whether it contains the word "meanwhile". Also,
suppose you have a file, namedurls.txt, where each line contains a URL
you would like to applytest_urlto.You can do the following:>>>frommeanwhileimportJob>>>job=Job(test_url,10)# at most 10 threads will be used concurrently.>>>urls=open("urls.txt").read().splitlines()>>>job.add_many(urls)>>>job.wait()>>>results=job.get_results()The target function (in this case:test_url) should get one argument, and
this argument should be hashable.Note that if your function prints output, you probably want to usemeanwhile.print()instead of Python's built-inprint()function.
This function prevents conflicts both with other threads, and with the progress
updates shown by thewaitmethod.In More DetailsTheJobobject holds a queue of inputs to process. It automatically
spawns and kills threads, as needed (up to the maximal number of concurrent
threads set by the user).The methodsaddandadd_manycan be used to add inputs to the queue.The methodwaitstops until the queue is empty. By default, it shows the
job's progress, like this:pending: 19	 running: 20	 finished: 11	 failed: 0In order towaitwithout showing the progress, one can set the keyword
argumentshow_statusto beFalse. Also, it's possible to show the
job's progress withoutwait-ing, using the methodprint_status.waitalso supports a keyword argument,timeout. If it is set, the
method will unconditionally return aftertimeoutseconds.waitcan
also be stopped safely by aKeyboardInterrupt.The return values of the target function can be inspected using the methodsget_result,has_resultandget_results.Fix Mistakes On The Flymeanwhilealso makes it easy to debug your code and fix it while the job
is already in progress.First, almost everything can be changed at any time. For example:You can always add new inputs to the queue usingaddandadd_many(even after all previous inputs were already processed, and all threads were
killed);You can always change the maximal number of threads allowed to run
concurrently using the methodset_n_threads;You can always change the target function using the methodset_target(this will apply only to inputs that weren't successfully processed yet).Also, you can inspect the exceptions raised by the target function using the
methodget_exceptions(and alsohas_exceptionandget_exception).After you fix the cause for the exceptions, you can use the methodsretry,retry_manyandretry_allto return inputs that raised exceptions
into the job's queue.In case your target function sometimes randomly fails (i.e. raises exception),
you can also use the methodset_n_attempts, to make the job retry inputs
automatically for a limited number of attempts (and note that theJobclass'es constructor also can take the keyword argumentn_attempts).Finally, note the methodspause,resumeandkill, that also
can be useful when you debug a job in progress (don't be afraid ofkill:
it is just equivalent toset_n_threads(0)).Resource ReuseSometimes it's useful to have a thread reusing resources whlie sequentially
processing inputs. For example, if your target function makes an HTTPS request,
you may want to reuse the same session, in order to save the TLS handshake time.That's whymeanwhileallows you to provide a targetfactoryinstead of a
target function. The factory is used to create a different target function for
each thread spawned.A target factory can be either a function that does not take arguments, and
returns a target function, or a callable class (that is initialized without
arguments, and its__call__method is used as the target function.In order to provide a target factory instead of target function, one must set
the keyword argumentfactoryto beTrue(this is true for both theJobclass constructor and theset_targetmethod).Module ReferenceFor the full module reference, seehelp(meanwhile).