Package: interleave
GitHub|PyPI|Issues|ChangelogTheinterleavepackage provides a function of the same name that takes a
number of iterators, runs them in separate threads, and yields the values
produced as soon as each one is available.Installationinterleaverequires Python 3.7 or higher.  Just usepipfor Python 3 (You have pip, right?) to installinterleaveand its dependencies:python3 -m pip install interleaveExample>>> from time import sleep, strftime
>>> from interleave import interleave
>>>
>>> def sleeper(idno, delays):
...     for i, d in enumerate(delays):
...         sleep(d)
...         yield (idno, i)
...
>>> with interleave(
...     [
...         sleeper(0, [0, 1, 2]),
...         sleeper(1, [2, 2, 2]),
...         sleeper(2, [5, 2, 1]),
...     ]
... ) as it:
...     for x in it:
...         print(strftime("%H:%M:%S"), x)
...
22:08:39 (0, 0)
22:08:40 (0, 1)
22:08:41 (1, 0)
22:08:42 (0, 2)
22:08:43 (1, 1)
22:08:44 (2, 0)
22:08:45 (1, 2)
22:08:46 (2, 1)
22:08:47 (2, 2)APIinterleave.interleave(iterators:Iterable[Iterator[T]],*,max_workers:Optional[int]=None,thread_name_prefix:str="",queue_size:Optional[int]=None,onerror:interleave.OnError=interleave.STOP,)->interleave.Interleaver[T]interleave()runs the given iterators in separate threads and returns an
iterator that yields the values yielded by them as they become available.  (See
below for details on theInterleaverclass.)Themax_workersandthread_name_prefixparameters are passed through to
the underlyingconcurrent.futures.ThreadPoolExecutor(q.v.).max_workersdetermines the
maximum number of iterators to run at one time.Thequeue_sizeparameter sets the maximum size of the queue used internally
to pipe values yielded by the iterators; when the queue is full, any iterator
with a value to yield will block waiting for the next value to be dequeued by a
call to the interleaver’s__next__.  Whenqueue_sizeisNone(the
default),interleave()uses aqueue.SimpleQueue, which has no maximum
size.  Whenqueue_sizeis non-None(including zero, signifying no
maximum size),interleave()uses aqueue.Queue, whoseget()method
is uninterruptible (including byKeyboardInterrupt) on Windows.Theonerrorparameter is an enum that determines howinterleave()should behave if one of the iterators raises an exception.  The possible values
are:STOP(default)Stop iterating over all iterators, cancel any outstanding
iterators that haven’t been started yet, wait for all running threads to
finish, and reraise the exception.  Note that, due to the inability to stop
an iterator between yields, the “waiting” step involves waiting for each
currently-running iterator to yield its next value before stopping.  This
can deadlock if the queue fills up in the interim.DRAINLikeSTOP, but any remaining values yielded by the iterators before
they finish are yielded by the interleaver before raising the exceptionFINISH_ALLContinue running as normal and reraise the exception once all iterators
have finishedFINISH_CURRENTLikeFINISH_ALL, except that only currently-running iterators are run
to completion; any iterators whose threads haven’t yet been started when
the exception is raised will have their jobs cancelledRegardless of the value ofonerror, any later exceptions raised by
iterators after the initial exception are discarded.classInterleaver(Generic[T]):def__init__(self,max_workers:Optional[int]=None,thread_name_prefix:str="",queue_size:Optional[int]=None,onerror:OnError=STOP,)An iterator and context manager.  As an iterator, it yields the values
generated by the iterators passed to the correspondinginterleave()call as
they become available.  As a context manager, it returns itself on entry and,
on exit, cleans up any unfinished threads by calling theshutdown(wait=True)method (see below).AnInterleavercan be instantiated either by callinginterleave()or by
calling the constructor directly.  The constructor takes the same arguments asinterleave(), minusiterators, and produces a newInterleaverthat
is not yet running any iterators.  Iterators are submitted to a newInterleavervia thesubmit()method; once all desired iterators have
been submitted, thefinalize()methodmustbe called so that theInterleavercan tell when everything’s finished.AnInterleaverwill shut down itsThreadPoolExecutorand wait for the
threads to finish after yielding its final value (specifically, when a call is
made to__next__/get()that would result inStopIterationor
another exception being raised).  In the event that anInterleaveris
abandoned before iteration completes, the associated resources may not be
properly cleaned up, and threads may continue running indefinitely.  For this
reason, it is strongly recommended that you wrap any iteration over anInterleaverin the context manager in order to handle a premature end to
iteration (including from aKeyboardInterrupt).Besides the iterator and context manager APIs, anInterleaverhas the
following public methods:Interleaver.submit(it:Iterator[T])->NoneNew in version 0.2.0Add an iterator to theInterleaver.If theInterleaverwas returned frominterleave()or has already hadfinalize()called on it, callingsubmit()will result in aValueError.Interleave.finalize()->NoneNew in version 0.2.0Notify theInterleaverthat all iterators have been registered.  This
method must be called in order for theInterleaverto detect the end of
iteration; if this method has not been called and all submitted iterators have
finished & had their values retrieved, then a subsequent call tonext(it)will end up hanging indefinitely.Interleaver.get(block:bool=True,timeout:Optional[float]=None)->TNew in version 0.2.0Fetch the next value generated by the iterators.  If all iterators have
finished and all values have been retrieved, raisesinterleaver.EndOfInputError.  IfblockisFalseand no values are
immediately available, raisesqueue.Empty.  IfblockisTrue, waits
up totimeoutseconds (or indefinitely, iftimeoutisNone) for the
next value to become available or for all iterators to end; if nothing happens
before the timeout expires, raisesqueue.Empty.it.get(block=True, timeout=None)is equivalent tonext(it), except that
the latter converts anEndOfInputErrortoStopIteration.Note:Whenonerror=STOPand a timeout is set, if an iterator raises an
exception, the timeout may be exceeded as theInterleaverwaits for all
remaining threads to shut down.Interleaver.shutdown(wait:bool=True)->NoneCallfinalize()if it hasn’t been called yet, tell all running iterators to
stop iterating, cancel any outstanding iterators that haven’t been started yet,
and shut down theThreadPoolExecutor.  Thewaitparameter is passed
through to the call toThreadPoolExecutor.shutdown().TheInterleavercan continue to be iterated over after callingshutdown()and will yield any remaining values produced by the iterators
before they stopped completely.