Package: cs.seq
Stuff to do with counters, sequences and iterables.Latest release 20221118:
Small doc improvement.Note that any function accepting an iterable
will consume some or all of the derived iterator
in the course of its function.Functioncommon_prefix_length(*seqs)Return the length of the common prefix of sequencesseqs.Functioncommon_suffix_length(*seqs)Return the length of the common suffix of sequencesseqs.Functionfirst(iterable)Return the first item from an iterable; raise IndexError on empty iterables.Functionget0(iterable, default=None)Return first element of an iterable, or the default.Functiongreedy(g=None, queue_depth=0)A decorator or function for greedy computation of iterables.Ifgis omitted or callable
this is a decorator for a generator function
causing it to compute greedily,
capacity limited byqueue_depth.Ifgis iterable
this function dispatches it in aThreadto compute greedily,
capacity limited byqueue_depth.Example with an iterable:for packet in greedy(parse_data_stream(stream)):
    ... process packet ...which does some readahead of the stream.Example as a function decorator:@greedy
def g(n):
    for item in range(n):
        yield nThis can also be used directly on an existing iterable:for item in greedy(range(n)):
    yield nNormally a generator runs on demand.
This function dispatches aThreadto run the iterable
(typically a generator)
putting yielded values to a queue
and returns a new generator yielding from the queue.Thequeue_depthparameter specifies the depth of the queue
and therefore how many values the original generator can compute
before blocking at the queue's capacity.The defaultqueue_depthis0which creates aChannelas the queue - a zero storage buffer - which lets the generator
compute only a single value ahead of time.A largerqueue_depthallocates aQueuewith that much storage
allowing the generator to compute as many asqueue_depth+1values
ahead of time.Here's a comparison of the behaviour:Example without@greedywhere the "yield 1" step does not occur until after the "got 0":>>> from time import sleep
>>> def g():
...   for i in range(2):
...     print("yield", i)
...     yield i
...   print("g done")
...
>>> G = g(); sleep(0.1)
>>> for i in G:
...   print("got", i)
...   sleep(0.1)
...
yield 0
got 0
yield 1
got 1
g doneExample with@greedywhere the "yield 1" step computes before the "got 0":>>> from time import sleep
>>> @greedy
... def g():
...   for i in range(2):
...     print("yield", i)
...     yield i
...   print("g done")
...
>>> G = g(); sleep(0.1)
yield 0
>>> for i in G:
...   print("got", repr(i))
...   sleep(0.1)
...
yield 1
got 0
g done
got 1Example with@greedy(queue_depth=1)where the "yield 1" step computes before the "got 0":>>> from cs.x import X
>>> from time import sleep
>>> @greedy
... def g():
...   for i in range(3):
...     X("Y")
...     print("yield", i)
...     yield i
...   print("g done")
...
>>> G = g(); sleep(2)
yield 0
yield 1
>>> for i in G:
...   print("got", repr(i))
...   sleep(0.1)
...
yield 2
got 0
yield 3
got 1
g done
got 2Functionimerge(*iters, **kw)Merge an iterable of ordered iterables in order.Parameters:iters: an iterable of iteratorsreverse: keyword parameter: if true, yield items in reverse order.
This requires the iterables themselves to also be in
reversed order.This function relies on the source iterables being ordered
and their elements being comparable, through slightly misordered
iterables (for example, as extracted from web server logs)
will produce only slightly misordered results, as the merging
is done on the basis of the front elements of each iterable.Functionisordered(items, reverse=False, strict=False)Test whether an iterable is ordered.
Note that the iterable is iterated, so this is a destructive
test for nonsequences.Functionlast(iterable)Return the last item from an iterable; raise IndexError on empty iterables.Functiononetomany(func)A decorator for a method of a sequence to merge the results of
passing every element of the sequence to the function, expecting
multiple values back.Example:class X(list):
        @onetomany
        def chars(self, item):
              return item
  strs = X(['Abc', 'Def'])
  all_chars = X.chars()Functiononetoone(func)A decorator for a method of a sequence to merge the results of
passing every element of the sequence to the function, expecting a
single value back.Example:class X(list):
        @onetoone
        def lower(self, item):
              return item.lower()
  strs = X(['Abc', 'Def'])
  lower_strs = X.lower()ClassSeqA numeric sequence implemented as a thread safe wrapper foritertools.count().ASeqis iterable and both iterating and calling it return
the next number in the sequence.Functionseq()Return a new sequential value.Functionsplitoff(sq, *sizes)Split a sequence into (usually short) prefixes and a tail,
for example to construct subdirectory trees based on a UUID.Example:>>> from uuid import UUID
>>> uuid = 'd6d9c510-785c-468c-9aa4-b7bda343fb79'
>>> uu = UUID(uuid).hex
>>> uu
'd6d9c510785c468c9aa4b7bda343fb79'
>>> splitoff(uu, 2, 2)
['d6', 'd9', 'c510785c468c9aa4b7bda343fb79']ClassStatefulIteratorA trivial iterator which wraps another iterator to expose some tracking state.This has 2 attributes:.it: the internal iterator which should yield(item,new_state).state: the last state value from the internal iteratorThe originating use case is resuse of an iterator by independent
calls that are typically sequential, specificly the .read
method of file like objects. Naive sequential reads require
the underlying storage to locate the data on every call, even
though the previous call has just performed this task for the
previous read. Saving the iterator used from the preceeding
call allows the iterator to pick up directly if the file
offset hasn't been fiddled in the meantime.Functiontee(iterable, *Qs)A generator yielding the items from an iterable
which also copies those items to a series of queues.Parameters:iterable: the iterable to copyQs: the queues, objects accepting a.putmethod.Note: the item is.putonto every queue
before being yielded from this generator.Functionthe(iterable, context=None)Returns the first element of an iterable, but requires there to be
exactly one.ClassTrackingCounterA wrapper for a counter which can be incremented and decremented.A facility is provided to wait for the counter to reach a specific value.
The .inc and .dec methods also accept atagargument to keep
individual counts based on the tag to aid debugging.TODO: addstrictoption to error and abort if any counter tries
to go below zero.MethodTrackingCounter.__init__(self, value=0, name=None, lock=None):
Initialise the counter tovalue(default 0) with the optionalname.Functionunrepeated(it, seen=None, signature=None)A generator yielding items from the iterableitwith no repetitions.Parameters:it: the iterable to processseen: an optional setlike container supportinginand.add()signature: an optional signature function for items fromitwhich produces the value to compare to recognise repeated items;
its values are stored in theseensetThe defaultsignaturefunction is equality;
the items are stored nseenand compared.
This requires the items to be hashable and support equality tests.
The same applies to whatever values thesignaturefunction produces.Another common signature is identity:id, useful for
traversing a graph which may have cycles.Sinceseenaccrues all the signature values for yielded items
generally it will grow monotonicly as iteration proceeeds.
If the items are complex or large it is well worth providing a signature
function even if the items themselves can be used in a set.Release LogRelease 20221118:
Small doc improvement.Release 20220530:
Seq: calling a Seq is like next(seq).Release 20210924:
New greedy(iterable) or @greedy(generator_function) to let generators precompute.Release 20210913:
New unrepeated() generator removing duplicates from an iterable.Release 20201025:
New splitoff() function to split a sequence into (usually short) prefixes and a tail.Release 20200914:
New common_prefix_length and common_suffix_length for comparing prefixes and suffixes of sequences.Release 20190103:
Documentation update.Release 20190101:New and UNTESTED class StatefulIterator to associate some externally visible state with an iterator.Seq: accept optionallockparameter.Release 20171231:Python 2 backport for imerge().New tee function to duplicate an iterable to queues.Function isordered() is now a test instead of an assertion.Drop NamedTuple, NamedTupleClassFactory (unused).Release 20160918:New function isordered() to test ordering of a sequence.imerge: accept newreverseparameter for merging reversed iterables.Release 20160828:
Modify DISTINFO to say "install_requires", fixes pypi requirements.Release 20160827:
TrackingCounter: accept presupplied lock object. Python 3 exec fix.Release 20150118:
metadata updateRelease 20150111:
Initial PyPI release.