Package: excitertools
excitertoolsitertoolsreimagined as afluent interface.In software engineering, a fluent interface is an object-oriented API whose design
relies extensively on method chaining. Its goal is to increase code legibility by
creating a domain-specific language (DSL). The term was coined in 2005 by Eric
Evans and Martin Fowler.*Wikipedia - Fluent Interface*Note that nearly all of themore-itertoolsextension library is included.Demo>>>range(10).map(lambdax:x*7).filter(lambdax:x%3==0).collect()[0,21,42,63]>>>range(10).map(lambdax:x*7).filter(lambdax:x>0andx%3==0).collect()[21,42,63]When the lines get long, parens can be used to split up each instruction:>>>(...range(10)....map(lambdax:x*7)....filter(lambdax:x%3==0)....collect()...)[0,21,42,63]What’s also interesting about that is how lambda’s can easily contain these
processing chains, since an entire chain is a single expression. For
example:>>>names=['caleb','james','gina']>>>Iter(names).map(...lambdaname:(...Iter(name)....map(lambdac:c.upper()ifcin'aeiouy'elsec)....collect(str)...)...).collect()['cAlEb','jAmEs','gInA']Something I’ve noticed is thatreduceseems easier to use and reason
about with this fluent interface, as compared to the conventional usage
as a standalone function; also, the operator module makesreducequite
useful for simple cases:>>>fromoperatorimportadd,mul>>>(...range(10)....map(lambdax:x*7)....filter(lambdax:x>0andx%3==0)....reduce(add)...)126>>>(...range(10)....map(lambdax:x*7)....filter(lambdax:x>0andx%3==0)....reduce(mul)...)55566ContentsexcitertoolsHow to Understand the API DocumentationModule-level Replacements for BuiltinsTheIterClassExperiments and Provisional IdeasRelated projectsDev InstructionsHow to Understand the API DocumentationSeveral symbols are used to indicate things about parts of the API:🎤 This function is asource, meaning that it produces data
that will be processed in an iterator chain.🎧 This function is asink, meaning that it consumes data that
was processed in an iterator chain.♾ This function returns an infinite iterable⚠ Warning - pay attention🛠 This API is still in flux, and might be changed or
removed in the future✨ Noteworthy; could be especially useful in many situations.The API is arranged roughly with the module-level functions first, and
thereafter theIterclass itself. It is theIterclass that does
the work to allow these iterators to be chained together. However, the
module-level functions are more likely to be used directly and that’s
why they’re presented first.The API includes wrappers for the stdlibitertoolsmodule, including
the “recipes” given in theitertoolsdocs, as well as wrappers for
the iterators from themore-itertools3rd-party package.Module-level Replacements for BuiltinsThe following module-level functions, likerange,zipand so on, are
intended to be used as replacements for their homonymous builtins. The
only difference between these and the builtin versions is that these
return instances of theIterclass. Note that becauseIteris itself
iterable, it means that the functions here can be used as drop-in
replacements.Once you have anIterinstance, all of its methods become available
via function call chaining, so these toplevel functions are really only
a convenience to “get started” using the chaining syntax with minimal
upfront cost in your own code.🎤range(*args)->"Iter[int]"zip(*iterables:Any)->"Iter[Tuple[T,...]]"enumerate(iterable)->"Iter[Tuple[int,T]]"map(func:Union[Callable[...,C], str], iterable)->"Iter[C]"filter(function:"Callable[[Any],bool]",iterable:"Iterable[T]")->"Iter[T]"🎤count(start=0, step: int = 1)->"Iter[int]"cycle(iterable)->"Iter[T]"🎤repeat(object: C, times=None)->"Iter[C]"accumulate(iterable, func=None, *, initial=None)chain(*iterables:Iterable[T])->"Iter[T]"chain_from_iterable(iterable)->"Iter[T]"compress(data, selectors)dropwhile(pred, iterable)filterfalse(pred, iterable)groupby(iterable, key=None)islice(iterable, *args)->"Iter"starmap(func, iterable)takewhile(pred, iterable)tee(iterable, n=2)zip_longest(*iterables,fillvalue=None)finditer_regex(pat:"re.Pattern[AnyStr]",s: AnyStr, flags: Union[int, re.RegexFlag] = 0)->"Iter[AnyStr]"splititer_regex(pat:"re.Pattern[AnyStr]",s: AnyStr, flags: Union[int, re.RegexFlag] = 0)->"Iter[AnyStr]"concat(iterable: Iterable[AnyStr], glue: AnyStr)->"AnyStr"✨ 🎤from_queue(q: queue.Queue, timeout=None, sentinel=None)->"Iter"🎤range(*args)->"Iter[int]"Replacement for the builtinrangefunction.  This version returns
an instance ofIterto allow further iterable chaining.All the same calling variations work because this function merely wraps
the original function.>>>range(3).collect()[0,1,2]>>>range(1,4).collect()[1,2,3]>>>range(1,6,2).collect()[1,3,5]>>>range(1,101,3).filter(lambdax:x%7==0).collect()[7,28,49,70,91]This example multiples, element by element, the series[0:5]with the
series[1:6]. Two things to note: Firstly,Iter.zipis used to emit
the tuples from each series. Secondly,Iter.starmapis used to receive
those tuples into separate arguments in thelambda.>>>range(5).zip(range(1,6)).starmap(lambdax,y:x*y).collect()[0,2,6,12,20]When written in a single line as above, it can get difficult to follow
the chain of logic if there are many processing steps. Parentheses in
Python allow grouping such that expressions can be spread over multiple
lines.This is the same example as the prior one, but formatted to be spread
over several lines. This is much clearer:>>># Written out differently>>>(...range(5)....zip(range(1,6))....starmap(lambdax,y:x*y)....collect()...)[0,2,6,12,20]If you wanted the sum instead, it isn’t necessary to do the collection
at all:>>>(...range(5)....zip(range(1,6))....starmap(lambdax,y:x*y)....sum()...)40zip(*iterables:Any)->"Iter[Tuple[T,...]]"Replacement for the builtinzipfunction.  This version returns
an instance ofIterto allow further iterable chaining.enumerate(iterable)->"Iter[Tuple[int,T]]"Replacement for the builtinenumeratefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>enumerate(string.ascii_lowercase).take(3).collect()[(0,'a'),(1,'b'),(2,'c')]map(func:Union[Callable[...,C], str], iterable)->"Iter[C]"Replacement for the builtinmapfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>result=map(lambdax:(x,ord(x)),'caleb').dict()>>>assertresult=={'a':97,'b':98,'c':99,'e':101,'l':108}>>>result=map('x, ord(x)','caleb').dict()>>>assertresult=={'a':97,'b':98,'c':99,'e':101,'l':108}filter(function:"Callable[[Any],bool]",iterable:"Iterable[T]")->"Iter[T]"Replacement for the builtinfilterfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>filter(lambdax:x%3==0,range(10)).collect()[0,3,6,9]🎤count(start=0, step: int = 1)->"Iter[int]"Replacement for the itertoolscountfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>count().take(5).collect()[0,1,2,3,4]>>>count(0).take(0).collect()[]>>>count(10).take(0).collect()[]>>>count(10).take(5).collect()[10,11,12,13,14]>>>count(1).filter(lambdax:x>10).take(5).collect()[11,12,13,14,15]cycle(iterable)->"Iter[T]"Replacement for the itertoolscountfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>cycle(range(3)).take(6).collect()[0,1,2,0,1,2]>>>cycle([]).take(6).collect()[]>>>cycle(range(3)).take(0).collect()[]🎤repeat(object: C, times=None)->"Iter[C]"Replacement for the itertoolscountfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>repeat('a').take(3).collect()['a','a','a']>>>repeat([1,2]).take(3).collect()[[1,2],[1,2],[1,2]]>>>repeat([1,2]).take(3).collapse().collect()[1,2,1,2,1,2]>>>repeat([1,2]).collapse().take(3).collect()[1,2,1]>>>repeat('a',times=3).collect()['a','a','a']This next set of functions return iterators that terminate on the shortest
input sequence.accumulate(iterable, func=None, *, initial=None)Replacement for the itertoolsaccumulatefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>accumulate([1,2,3,4,5]).collect()[1,3,6,10,15]>>>ifsys.version_info>=(3,8):...output=accumulate([1,2,3,4,5],initial=100).collect()...assertoutput==[100,101,103,106,110,115]>>>accumulate([1,2,3,4,5],operator.mul).collect()[1,2,6,24,120]>>>accumulate([]).collect()[]>>>accumulate('abc').collect()['a','ab','abc']>>>accumulate(b'abc').collect()[97,195,294]>>>accumulate(bytearray(b'abc')).collect()[97,195,294]chain(*iterables:Iterable[T])->"Iter[T]"Replacement for the itertoolschainfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>chain('ABC','DEF').collect()['A','B','C','D','E','F']>>>chain().collect()[]chain_from_iterable(iterable)->"Iter[T]"Replacement for the itertoolschain.from_iterablemethod.
This version returns an instance ofIterto allow
further iterable chaining.>>>chain_from_iterable(['ABC','DEF']).collect()['A','B','C','D','E','F']>>>chain_from_iterable([]).collect()[]compress(data, selectors)Replacement for the itertoolscompressfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>compress('ABCDEF',[1,0,1,0,1,1]).collect()['A','C','E','F']dropwhile(pred, iterable)Replacement for the itertoolsdropwhilefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>dropwhile(lambdax:x<4,range(6)).collect()[4,5]filterfalse(pred, iterable)Replacement for the itertoolsfilterfalsefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>filterfalse(None,[2,0,3,None,4,0]).collect()[0,None,0]groupby(iterable, key=None)Replacement for the itertoolsgroupbyfunction.  This version returns
an instance ofIterto allow further iterable chaining.groupbyreturns an iterator of a key and “grouper” iterable. In the
example below, we useIter.starmapto collect each grouper iterable
into a list, as this makes it neater for display here in the docstring.>>>(...groupby(['john','jill','anne','jack'],key=lambdax:x[0])....starmap(lambdak,g:(k,list(g)))....collect()...)[('j',['john','jill']),('a',['anne']),('j',['jack'])]islice(iterable, *args)->"Iter"Replacement for the itertoolsislicefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>islice('ABCDEFG',2).collect()['A','B']>>>islice('ABCDEFG',2,4).collect()['C','D']>>>islice('ABCDEFG',2,None).collect()['C','D','E','F','G']>>>islice('ABCDEFG',0,None,2).collect()['A','C','E','G']starmap(func, iterable)Replacement for the itertoolsstarmapfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>starmap(pow,[(2,5),(3,2),(10,3)]).collect()[32,9,1000]takewhile(pred, iterable)Replacement for the itertoolstakewhilefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>takewhile(lambdax:x<5,[1,4,6,4,1]).collect()[1,4]tee(iterable, n=2)Replacement for the itertoolsteefunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>a,b=tee(range(5))>>>a.collect()[0,1,2,3,4]>>>b.sum()10It is also possible to operate on the returned iterators in the chain
but it gets quite difficult to understand:>>>tee(range(5)).map(lambdait:it.sum()).collect()[10,10]In the example above we passed inrange, but withexcitertoolsit’s usually more natural to push data sources further left:>>>range(5).tee().map(lambdait:it.sum()).collect()[10,10]Pay close attention to the above. Themapis acting on each of the
copied iterators.zip_longest(*iterables,fillvalue=None)Replacement for the itertoolszip_longestfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>zip_longest('ABCD','xy',fillvalue='-').collect()[('A','x'),('B','y'),('C','-'),('D','-')]>>>(...zip_longest('ABCD','xy',fillvalue='-')....map(lambdatup:concat(tup,''))....collect()...)['Ax','By','C-','D-']>>>(...zip_longest('ABCD','xy',fillvalue='-')....starmap(operator.add)....collect()...)['Ax','By','C-','D-']finditer_regex(pat:"re.Pattern[AnyStr]",s: AnyStr, flags: Union[int, re.RegexFlag] = 0)->"Iter[AnyStr]"Wrapper forre.finditer. Returns an instance ofIterto allow
chaining.>>>pat=r"\w+">>>text="Well hello there! How ya doin!">>>finditer_regex(pat,text).map(str.lower).filter(lambdaw:'o'inw).collect()['hello','how','doin']>>>finditer_regex(r"[A-Za-z']+","A programmer's RegEx test.").collect()['A',"programmer's",'RegEx','test']>>>finditer_regex(r"[A-Za-z']+","").collect()[]>>>finditer_regex("","").collect()['']>>>finditer_regex("","").filter(None).collect()[]splititer_regex(pat:"re.Pattern[AnyStr]",s: AnyStr, flags: Union[int, re.RegexFlag] = 0)->"Iter[AnyStr]"Lazy string splitting using regular expressions.Most of the time you wantstr.split. Really! That will almost
always be fastest. You might think thatstr.splitis inefficient
because it always has to build a list, but it can do this very, very
quickly.The lazy splitting shown here is more about supporting a particular
kind of programming model, rather than performance.See more discussionhere.>>>splititer_regex(r"\s","A programmer's RegEx test.").collect()['A',"programmer's",'RegEx','test.']Note that splitting at a single whitespace character will return blanks
for each found. This is different to howstr.split()works.>>>splititer_regex(r"\s","aaa     bbb\nccc\nddd\teee").collect()['aaa','','','','','bbb','','','','','ccc','ddd','eee']To matchstr.split(), specify a sequence of whitespace as the
regex pattern.>>>splititer_regex(r"\s+","aaa     bbb\nccc\nddd\teee").collect()['aaa','bbb','ccc','ddd','eee']Counting the whitespace>>>splititer_regex(r"\s","aaa     bbb\nccc\nddd\teee").collect(Counter)Counter({'':8,'aaa':1,'bbb':1,'ccc':1,'ddd':1,'eee':1})Lazy splitting at newlines>>>splititer_regex(r"\n","aaa     bbb\nccc\nddd\teee").collect()['aaa     bbb  ','  ccc','ddd\teee']A few more examples:>>>splititer_regex(r"","aaa").collect()['','a','a','a','']>>>splititer_regex(r"","").collect()['','']>>>splititer_regex(r"\s","").collect()['']>>>splititer_regex(r"a","").collect()['']>>>splititer_regex(r"\s","aaa").collect()['aaa']concat(iterable: Iterable[AnyStr], glue: AnyStr)->"AnyStr"Concatenate strings, bytes and bytearrays. It is careful to avoid the
problem with single bytes becoming integers, and it looks at the value
ofglueto know whether to handle bytes or strings.This function can raiseValueErrorif called with something
other thanbytes,bytearrayorstr.✨ 🎤from_queue(q: queue.Queue, timeout=None, sentinel=None)->"Iter"Wrap a queue with an iterator interface. This allows it to participate
in chaining operations. The iterator will block while waiting for
new values to appear on the queue. This is useful: it allows you
to easily and safely pass data between threads or processes, and
feed the incoming data into a pipeline.The sentinel value, defaultNone, will terminate the iterator.>>>q=queue.Queue()>>># This line puts stuff onto a queue>>>range(10).chain([None]).map(q.put).consume()>>>from_queue(q).filter(lambdax:2<x<9).collect()[3,4,5,6,7,8]This can be used in the same way you would normally use a queue, in
that it will block while waiting for future input. This makes it
convenient to run in a thread and wait for work. Below is a rough
sketch of how one might cobble together a thread pool using this
feature. Note the use ofIter.side_effectto calltask_done()on the queue.importqueuefromthreadingimportThreadimportloggingfromexcitertoolsimportfrom_queuelogger=logging.getLogger(__name__)defprocess_job(job):result=...returnresultdefworker(inputs:Queue,results:Queue):(from_queue(inputs).side_effect(lambdajob:logger.info(f"Received job{job}").map(process_job).side_effect(lambdaresult:logger.info(f"Got result{job}").into_queue(results)# Note that we only mark the task as done after the result# is added to the results queue..side_effect(lambda_:inputs.task_done())defcreate_job_pool(n:int)->Tuple[Queue,Queue,Callable]:"""Returns two queues, and a pool shutdown method. The
    shutdown function can be called to shut down the pool and
    the ``inputs`` queue. Caller is responsible for draining
    the ``results`` queue."""# Independent control of the sizes of the input and output# queues is interesting: it lets you decide how to bias# backpressure depending on the details of your workload.inputs,results=Queue(maxsize=100),Queue(maxsize=3)kwargs=dict(target=worker,args=(inputs,results),daemon=True)threads=repeat(Thread).map(lambdaT:T(**kwargs)).take(n).collect()defshutdown():# Shut down each threadrepeat(None).map(inputs.put).take(n).consume()inputs.join()Iter(threads).map(lambdat:t.join()).consume()returninputs,results,shutdownNow the two queuesinputsandresultscan be used in various
other threads to supply and consume data.TheIterClass✨class Iter(Generic[T])@classmethod Iter.register(cls, *func)🎧Iter.collect(self, container=list)->"List[T]"✨ 🎤@classmethod Iter.open(cls, file,mode="r",buffering=-1,encoding=None, errors=None, newline=None, closefd=True, opener=None, )->"Iter"🎤@classmethod Iter.read_lines(cls, stream: IO[str], rewind=True)🎤@classmethod Iter.read_bytes(cls, stream: IO[bytes], size:Union[Callable[[],int], int] =-1,rewind=True)🎧Iter.write_text_to_stream(self, stream: IO[str], insert_newlines=True, flush=True)🎧Iter.write_bytes_to_stream(self, stream: IO[bytes], flush=True)✨ 🎧Iter.write_to_file(self, file,mode="w",buffering=-1,encoding=None, errors=None, newline=None, closefd=True, opener=None, )🎤@classmethod Iter.range(cls, *args)->"Iter[int]"Iter.zip(self, *iterables: Any)->"Iter[Tuple[T,...]]"🎧Iter.any(self)->"bool"🎧Iter.all(self)->"bool"Iter.enumerate(self)->"Iter[Tuple[int,T]]"Iter.dict(self)->"Dict"Iter.map(self, func:Union[Callable[...,C], str])->"Iter[C]"Iter.filter(self, function:"Optional[Callable[[T],bool]]"= None)->"Iter[T]"✨Iter.starfilter(self, function:"Optional[Callable[[T,...],bool]]"= None)->"Iter[T]"Iter.filter_gt(self, value)->"Iter[T]"Iter.filter_ge(self, value)->"Iter[T]"Iter.filter_lt(self, value)->"Iter[T]"Iter.filter_le(self, value)->"Iter[T]"Iter.filter_eq(self, value)->"Iter[T]"Iter.filter_ne(self, value)->"Iter[T]"Iter.filter_in(self, value: Sized)->"Iter[T]"Iter.filter_ni(self, value)->"Iter[T]"🎧Iter.reduce(self, func:Callable[...,T], *args)->"T"🎧Iter.starreduce(self, function:Callable[...,T], initializer=0)->"T"🎧Iter.sum(self)🎧Iter.concat(self, glue: AnyStr = '')->"AnyStr"Iter.insert(self, glue: C)->"Iter[Union[C,T]]"🎤@classmethod Iter.count(cls, *args)->"Iter[int]"♾Iter.cycle(self)->"Iter[T]"🎤 ♾@classmethod Iter.repeat(cls, elem: C, times=None)->"Iter[C]"Iter.accumulate(self, func=None, *, initial=None)Iter.chain(self, *iterables: Iterable[T])->"Iter[T]"Iter.chain_from_iterable(self)->"Iter[T]"Iter.compress(self, selectors)Iter.dropwhile(self, pred)Iter.filterfalse(self, pred)Iter.groupby(self, key=None)Iter.islice(self, *args)->"Iter"Iter.starmap(self, func)Iter.takewhile(self, pred)Iter.tee(self, n=2)Iter.zip_longest(self, *iterables, fillvalue=None)Iter.chunked(self, n: int)->"Iter"Iter.ichunked(self, n: int)->"Iter"@classmethod Iter.sliced(cls, seq: Sequence, n: int)->"Iter"Iter.distribute(self, n: int)->"Iter"Iter.divide(self, n: int)->"Iter"Iter.split_at(self, pred)Iter.split_before(self, pred)Iter.split_after(self, pred)Iter.split_into(self, sizes)Iter.split_when(self, pred)Iter.bucket(self, key, validator=None)Iter.unzip(self)Iter.grouper(self, n: int, fillvalue=None)->"Iter"Iter.partition(self, pred)->"Iter"Iter.spy(self, n=1)->"Tuple[Iter, Iter]"Iter.peekable(self)->"more_itertools.peekable"Iter.seekable(self)->"more_itertools.seekable"Iter.windowed(self, n, fillvalue=None, step=1)->"Iter"Iter.substrings(self)Iter.substrings_indexes(self, reverse=False)Iter.stagger(self,offsets=(-1,0, 1), longest=False, fillvalue=None)Iter.pairwise(self)Iter.count_cycle(self, n=None)->"Iter"Iter.intersperse(self, e, n=1)->"Iter"Iter.padded(self, fillvalue: Optional[C] = None, n: Optional[int] = None, next_multiple: bool = False, )->"Iter[Union[T,C]]"Iter.repeat_last(self, default=None)->"Iter[T]"Iter.adjacent(self, pred, distance=1)->"Iter[Tuple[bool,T]]"Iter.groupby_transform(self, keyfunc:Optional[Callable[...,K]] = None, valuefunc:Optional[Callable[...,V]] = None, )->"Iter[Tuple[K,Iterable[V]]]"Iter.padnone(self)->"Iter[Union[T,None]]"Iter.ncycles(self, n)->"Iter[T]"Iter.collapse(self, base_type=None, levels=None)->"Iter"@class_or_instancemethod Iter.sort_together(self_or_cls, iterables,key_list=(0,),reverse=False)@class_or_instancemethod Iter.interleave(self_or_cls, *iterables)->"Iter"@class_or_instancemethod Iter.interleave_longest(self_or_cls, *iterables)->"Iter"@classmethod Iter.zip_offset(cls, *iterables, offsets, longest=False, fillvalue=None)->"Iter"Iter.dotproduct(self, vec2: Iterable)Iter.flatten(self)->"Iter[T]"@class_or_instancemethod Iter.roundrobin(self_or_cls: Union[Type[T], T], *iterables: C)->"Iter[Union[T,C]]"Iter.prepend(self, value: C)->"Iter[Union[T,C]]"🎧Iter.ilen(self)->"int"Iter.unique_to_each(self)->"Iter[T]"Iter.sample(self, k=1, weights=None)->"Iter"Iter.consecutive_groups(self, ordering=lambda x: x)Iter.run_length_encode(self)->"Iter[Tuple[T,int]]"Iter.run_length_decode(self)->"Iter"Iter.map_reduce(self, keyfunc, valuefunc=None, reducefunc=None)->"Dict"Iter.map_reduce_it(self, keyfunc:Callable[...,K], valuefunc:Optional[Callable[...,V]] = None, reducefunc:Optional[Callable[...,R]] = None, )->"Iter[Tuple[K,R]]"🎧Iter.exactly_n(self, n, predicate=bool)->"bool"Iter.all_equal(self)Iter.first_true(self)Iter.quantify(self)Iter.islice_extended(self, *args)Iter.first(self)Iter.last(self)Iter.one(self)Iter.only(self, default=None, too_long=ValueError)->"T"Iter.strip(self, pred)->"Iter[T]"Iter.lstrip(self, pred)->"Iter[T]"Iter.rstrip(self, pred)->"Iter[T]"Iter.filter_except(self, validator, *exceptions)->"Iter[T]"Iter.map_except(self, function, *exceptions)->"Iter"Iter.nth_or_last(self, n, default=_marker)->"T"Iter.nth(self, n, default=None)Iter.take(self, n: int)->"Iter"Iter.tail(self, n)->"Iter[T]"Iter.unique_everseen(self, key=None)->"Iter[T]"Iter.unique_justseen(self, key=None)->"Iter[T]"Iter.distinct_permutations(self)Iter.distinct_combinations(self, r)->"Iter[T]"Iter.circular_shifts(self)->"Iter[T]"Iter.partitions(self)->"Iter[T]"Iter.set_partitions(self, k=None)->"Iter[T]"Iter.powerset(self)@class_or_instancemethod Iter.random_product(self_or_cls, *args, repeat=1)Iter.random_permutation(self, r=None)Iter.random_combination(self, r)Iter.random_combination_with_replacement(self, r)Iter.nth_combination(self, r, index)@classmethod Iter.always_iterable(cls, obj,base_type=(str,bytes))->'Iter'Iter.always_reversible(self)@classmethod Iter.with_iter(cls, context_manager)@classmethod Iter.iter_except(cls, func, exception, first=None)->"Iter"Iter.locate(self, pred=bool, window_size=None)->"Iter"Iter.rlocate(self, pred=bool, window_size=None)->"Iter"Iter.replace(self, pred, substitutes, count=None, window_size=1)->"Iter"@classmethod Iter.numeric_range(cls, *args)->"Iter"Iter.side_effect(self, func, *args, chunk_size=None, before=None, after=None)Iter.iterate(self)Iter.difference(self, func=operator.sub, *, initial=None)Iter.make_decorator(self)Iter.SequenceView(self)Iter.time_limited(self, limit_seconds)->"Iter"🎧Iter.consume(self, n: Optional[int] = None)->"Optional[Iter[T]]"Iter.tabulate(self)🎤@classmethod Iter.repeatfunc(cls, func, *args, times=None)Iter.wrap(self, ends: "Sequence[T, T]" ="()")Iter.print(self,template="{i}:{v}")->"Iter[T]"🎤@classmethod Iter.from_queue(cls, q: queue.Queue, timeout=None, sentinel=None)Iter.into_queue(self, q: queue.Queue)->"Iter[T]"🎧Iter.send(self, collector: Generator, close_collector_when_done=False)->"None"Iter.send_also(self, collector: Generator)->"Iter"🎧 ⚠Iter.sorted(self, key=None, reverse=False)->"Iter[T]"🎧 ⚠Iter.reversed(self)->"Iter[T]"✨class Iter(Generic[T])This class is what allows chaining. Many of the methods in this class
return an instance ofIter, which allows further chaining. There
are two exceptions to this:sourcesandsinks.A “source” is usually aclassmethodwhich can be used as an
initializer to produce data via an iterable. For example, theIter.rangeclassmethod can be used to get a sequence of numbers:>>>Iter.range(1_000_000).take(3).collect()[0,1,2]Even though our range was a million elements, the iterator chaining
took only 3 of those elements before collecting.A “sink” is a method that is usually the last component of a processing
chain and often (but not always!) consumes the entire iterator. In the
example above, the call toIter.collectwas a sink. Note that we still
call it a sink even though it did not consume the entire iterator.We’re using the term “source” to refer to a classmethod ofIterthat
produces data; but, the most typical source is going to be data that
you provide.Itercan be called with anything that is iterable, including
sequences, iterators, mappings, sets, generators and so on.Examples:List>>>Iter([1,2,3]).map(lambdax:x*2).sum()12Generator>>>Iter((1,2,3)).map(lambdax:x*2).sum()12>>>defg():...foriin[1,2,3]:...yieldi>>>Iter(g()).map(lambdax:x*2).sum()12Iterator>>>Iter(iter([1,2,3])).map(lambdax:x*2).sum()12Dict>>>Iter(dict(a=1,b=2)).map(lambdax:x.upper()).collect()['A','B']>>>d=dict(a=1,b=2,c=3)>>>Iter(d.items()).starmap(lambdak,v:v).map(lambdax:x*2).sum()12A common error with generators is forgetting to actually evaluate, i.e.,
call a generator function. If you do this there’s a friendly error
pointing out the mistake:>>>defmygen():yield123>>>Iter(mygen).collect()Traceback(mostrecentcalllast):...TypeError:Itseemsyoupassedageneratorfunction,butyouprobablyintendedtopassagenerator.Remembertoevaluatethefunctiontoobtainageneratorinstance:defmygen():yield123Iter(mygen)# ERROR - a generator function object is not iterableIter(mygen())# CORRECT - a generator instance is iterable.>>>Iter(mygen()).collect()[123]Instance ofIterare resumable. Once an instance it created, it can
be partially iterated in successive calls, like the following example
shows:>>>it=Iter.range(1_000_000)>>>it.take(3).collect()[0,1,2]>>>it.take(4).collect()[3,4,5,6]>>># Consume most of the stream, collect the last few>>>it.consume(999_990).collect()[999997,999998,999999]This class implements the chaining. However, the module-level functions
inexcitertools, such asrange,zipand so on, also return
instances ofIter, so they allow the chaining to continue. These are
equivalent:>>>Iter.range(10).filter(lambdax:x>7).collect()[8,9]>>>range(10).filter(lambdax:x>7).collect()[8,9]It is intended that the module-level functions can act as drop-in
replacements for the builtins they wrap:>>> import builtins
>>> list(builtins.range(3))
[0, 1, 2]
>>> list(range(3))  # This is excitertools.range!
[0, 1, 2]
>>> list(Iter.range(3))
[0, 1, 2]In your own code where you might like to use theexcitertoolsversion ofrangeand the other functions, you can just import it and use it to access all the other
cool stuff:# mymodule.pyfromexcitertoolsimport(range,map,filter,reduce,repeat,count,enumerate,zip,...)deffunc(inputs):data=(map(lambdax:x+2,inputs).enumerate().filter(lambdax:x[1]>10)....collect())Alternatively, if you don’t want to hide the builtins you can do just
fine with importing this class only, or even importing the module only:# mymodule.py - same example as beforeimportexcitertoolsdeffunc(inputs):data=(excitertools.Iter(inputs).map(lambdax:x+2,inputs).enumerate().filter(lambdax:x[1]>10)....collect())# Do something with dataThere are several valuable additions to the standarditertoolsandmore-itertoolsfunctions. These usually involve sources and sinks,
which are ways of getting data into an iterator pipeline, and then
getting results out again. In the majority of documentation examples
shown here, theIter.collectmethod is used to collect all the
remaining data on a stream into a list; but in practice this is not
useful because large lists consume memory.In practice it is more useful to send iterator data to one of these
common sinks:filessocketsqueuesHTTP APIsCloud storage buckets(Ideas for more to add here?)Iterhas support for these use-cases, both for reading and for writing.@classmethod Iter.register(cls, *func)Add a new method toIter. Sure, you could subclassIterto get
new chaining features, but it would be neat to let all existingIterinstance just immediately have the new registered function
available.The new function must takeiterableas the first parameter.>>>defup(iterable):...forviniterable:...yieldv.upper()>>>Iter.register(up)>>>Iter('abc').up().collect()['A','B','C']>>>defpoly(iterable,a,b,c):...# Polynomials a.x^2 + b.x + c...forxiniterable:...yielda*x**2+b*x+c>>>Iter.register(poly)>>>Iter(range(-5,5,1)).poly(1,-5,6).collect()[56,42,30,20,12,6,2,0,0,2]Here’s a math round-trip rollercoaster.>>>importmath>>>deflog(iterable):...forxiniterable:...yieldmath.log(x)>>>defexp(iterable):...forxiniterable:...yieldmath.exp(x)>>>defrnd(iterable):...forxiniterable:...yieldround(x)>>>Iter.register(log,exp,rnd)>>>Iter(range(5)).exp().log().rnd().collect()[0,1,2,3,4]These are silly examples, but hopefully you get the idea.🎧Iter.collect(self, container=list)->"List[T]"This is the most common way of “realizing” an interable chain
into a concrete data structure. It should be the case that this
is where most of the memory allocation occurs.The default container is a list and you’ll see throughout this
documentation that most examples produce lists. However,
any container, and indeed any function, can be used as the sink.The basic example:>>>Iter(range(3)).collect()[0,1,2]>>>Iter(range(3)).collect(tuple)(0,1,2)You must pay attention to some things. For example, if your
iterable is a string, the characters of the string are what
get iterated over, and when you collect you’ll get a collection
of those atoms. You can however usestras your “container
function” and that will give you back a string. It’s like a join
with blank joiner.>>>Iter('abc').collect()['a','b','c']>>>Iter('abc').collect(str)'abc'With some types, things get a little more tricky. Takebytesfor example:>>>Iter(b'abc').collect()[97,98,99]You probably didn’t expect to get the integers back right? Anyhow,
you can usebytesas the “collection container”, just like
we did with strings and that will work:>>>Iter(b'abc').collect(bytes)b'abc'>>>Iter(b'abc').collect(bytearray)bytearray(b'abc')The other standard collections also work, here’s a set for
completeness.>>>Iter('abcaaaabbbbccc').collect(set)=={'a','b','c'}True✨ 🎤@classmethod Iter.open(cls, file,mode="r",buffering=-1,encoding=None, errors=None, newline=None, closefd=True, opener=None, )->"Iter"Wrap theopen()builtin precisely, but return anIterinstance to allow function chaining on the result.I know you’re thinking that we should always use a context
manager for files. Don’t worry, there is one being used
internally. When the iterator chain is terminated the underlying
file will be closed.>>> import tempfile
>>> with tempfile.TemporaryDirectory() as td:
...     # Put some random text into a temporary file
...     with open(td + 'text.txt', 'w') as f:
...         f.writelines(['abc\n', 'def\n', 'ghi\n'])
...
...     # Open the file, filter some lines, collect the result
...     Iter.open(td + 'text.txt').filter(lambda line: 'def' in line).collect()
['def\n']Note that this is a convenience method forreadingfrom a file,
not for writing. The function signature includes themodeparameter for parity with the builtinopen()function, but
only reading is supported.🎤@classmethod Iter.read_lines(cls, stream: IO[str], rewind=True)Read lines from a file-like object.First, let’s put some data in a file. We’ll be using that
file in the examples that follow.>>>importtempfile>>>td=tempfile.TemporaryDirectory()...# Put some random text into a temporary file>>>withopen(td.name+'text.txt','w')asf:...f.writelines(['abc\n','def\n','ghi\n'])...Use read_lines to process the file data>>>withopen(td.name+'text.txt')asf:...Iter.read_lines(f).filter(lambdaline:'def'inline).collect()['def\n']Therewindparameter can be used to read sections of a file.>>>withopen(td.name+'text.txt')asf:...part1=Iter.read_lines(f).take(1).collect()...part2=Iter.read_lines(f,rewind=False).collect()>>>part1['abc\n']>>>part2['def\n','ghi\n']>>>td.cleanup()🎤@classmethod Iter.read_bytes(cls, stream: IO[bytes], size:Union[Callable[[],int], int] =-1,rewind=True)Thesizeparameter can be used to control how many bytes are
read for each advancement of the iterator chain. Here we setsize=1which means we’ll get back one byte at a time.>>>importtempfile>>>td=tempfile.TemporaryDirectory()>>>filename=td.name+'bytes.bin'Put some random text into a temporary file:>>>withopen(filename,'wb')asf:...x=f.write(b'\x00'*100)...>>>withopen(filename,'rb')asf:...data=Iter.read_bytes(f,size=1).collect()...len(data)100>>>withopen(filename,'rb')asf:...data=Iter.read_bytes(f).collect()...len(data)1A little more ambitious. Becausesizeis a callable, we can use
adequeand aside_effectto pass information back into
the reader to control how many bytes are read in each chunk.In this example we’re reading 1 byte at a time. In a real example
you might have a sequence of headers and bodies, where headers
give size information about how many bytes are in the body
corresponding to that header. Then you can precisely read
each body in sequence.>>>read_sizes=deque([1])>>>withopen(filename,'rb')asf:...data=(...Iter....read_bytes(f,size=lambda:read_sizes.popleft())....side_effect(lambdabytes:read_sizes.append(1))....collect()...)...len(data)100Therewindparameter can be used to read sections of a file.>>>withopen(filename,'rb')asf:...part1=Iter.read_bytes(f,size=10).take(1).collect()...part2=Iter.read_bytes(f,rewind=False).collect()>>>part1[b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']>>>len(part2[0])90>>>td.cleanup()🎧Iter.write_text_to_stream(self, stream: IO[str], insert_newlines=True, flush=True)>>>importtempfile>>>td=tempfile.TemporaryDirectory()>>>filename=td.name+'text.txt'>>>data=['a','b','c']>>>withopen(filename,'w')asf:...Iter(data).map(str.upper).write_text_to_stream(f)...withopen(filename)asf2:...Iter.read_lines(f2).concat()'A\nB\nC'If some prior step adds newlines, or more commonly, newlines
originate with a data source and are simply carried through the
processing chain unaltered, disable the insertion of newlines:>>>withopen(filename,'w')asf:...Iter(data).map(str.upper).write_text_to_stream(f,insert_newlines=False)...withopen(filename)asf2:...Iter.read_lines(f2).concat()'ABC'Multiple successive writes may be slowed down by the defaultflush=Trueparameter. In this case you can delay flushing until
everything has been written.>>>withopen(filename,'w')asf:...Iter(data).map(str.upper).write_text_to_stream(f,flush=False)...Iter(data).map(str.upper).write_text_to_stream(f,flush=False)...Iter(data).map(str.upper).write_text_to_stream(f,flush=True)...withopen(filename)asf2:...Iter.read_lines(f2).concat()'A\nB\nCA\nB\nCA\nB\nC'>>>td.cleanup()🎧Iter.write_bytes_to_stream(self, stream: IO[bytes], flush=True)>>>importtempfile>>>td=tempfile.TemporaryDirectory()>>>filename=td.name+'bytes.bin'>>>data=[b'a',b'b',b'c']>>>withopen(filename,'wb')asf:...Iter(data).map(lambdax:x*2).write_bytes_to_stream(f)...withopen(filename,'rb')asf2:...Iter.read_bytes(f2).collect()[b'aabbcc']>>>withopen(filename,'wb')asf:...Iter(data).map(lambdax:x*2).write_bytes_to_stream(f)...withopen(filename,'rb')asf2:...Iter.read_bytes(f2).concat(b'')b'aabbcc'>>>withopen(filename,'wb')asf:...Iter(data).map(lambdax:x*2).write_bytes_to_stream(f)...withopen(filename,'rb')asf2:...Iter.read_bytes(f2,size=1).collect()[b'a',b'a',b'b',b'b',b'c',b'c']>>>withopen(filename,'wb')asf:...Iter(data).map(lambdax:x*2).write_bytes_to_stream(f)...withopen(filename,'rb')asf2:...Iter.read_bytes(f2,size=2).map(bytes.decode).collect()['aa','bb','cc']Flushing can be delayed if multiple parts are to be written.>>>withopen(filename,'wb')asf:...it=Iter(data)...it.map(lambdax:x*2).take(2).write_bytes_to_stream(f,flush=False)...it.map(lambdax:x*2).write_bytes_to_stream(f,flush=True)...withopen(filename,'rb')asf2:...Iter.read_bytes(f2,size=2).map(bytes.decode).collect()['aa','bb','cc']>>>td.cleanup()✨ 🎧Iter.write_to_file(self, file,mode="w",buffering=-1,encoding=None, errors=None, newline=None, closefd=True, opener=None, )>>>importtempfile>>>withtempfile.TemporaryDirectory()astd:...# Put some random text into a temporary file...withopen(td+'text.txt','w')asf:...f.writelines(['abc\n','def\n','ghi\n'])......# Open the file, transform, write out to new file....Iter.open(td+'text.txt').map(str.upper).write_to_file(td+'test2.txt')...# Read the new file, for the test...Iter.open(td+'test2.txt').collect()['ABC\n','DEF\n','GHI\n']🎤@classmethod Iter.range(cls, *args)->"Iter[int]"Therangefunction you all know and love.>>>Iter.range(3).collect()[0,1,2]>>>Iter.range(0).collect()[]Iter.zip(self, *iterables: Any)->"Iter[Tuple[T,...]]"Thezipfunction you all know and love. The only thing to
note here is that the first iterable is really what theIterinstance is wrapping. TheIter.zipinvocation brings in the
other iterables.Make anIterinstance, then callzipon that.>>>Iter('caleb').zip(range(10)).collect()[('c',0),('a',1),('l',2),('e',3),('b',4)]Use a classmethod to get an infinite stream usingIter.countand zip against that with more finite iterators.>>>Iter.count().zip(range(5),range(3,100,2)).collect()[(0,0,3),(1,1,5),(2,2,7),(3,3,9),(4,4,11)]It takes a few minutes to get used to that but feels comfortable
pretty quickly.Iter.takecan be used to stop infinite zip sequences:>>>Iter('caleb').cycle().enumerate().take(8).collect()[(0,'c'),(1,'a'),(2,'l'),(3,'e'),(4,'b'),(5,'c'),(6,'a'),(7,'l')]While we’re here (assuming you worked through the previous
example), note the difference if you switch the order of theIter.cycleandIter.enumeratecalls:>>>Iter('caleb').enumerate().cycle().take(8).collect()[(0,'c'),(1,'a'),(2,'l'),(3,'e'),(4,'b'),(0,'c'),(1,'a'),(2,'l')]If you understand how this works, everything else in _excitertools_
will be intuitive to use.🎧Iter.any(self)->"bool">>>Iter([0,0,0]).any()False>>>Iter([0,0,1]).any()True>>>Iter([]).any()False🎧Iter.all(self)->"bool">>>Iter([0,0,0]).all()False>>>Iter([0,0,1]).all()False>>>Iter([1,1,1]).all()TrueNow pay attention:>>>Iter([]).all()TrueThis behaviour has some controversy around it, but that’s how theall()builtin works so that’s what we do too. The way to
think about whatall()does is this: it returns False if there
is at least one element that is falsy.  Thus, if there are no elements
it follows that there are no elements that are falsy and that’s whyall([])== True.Iter.enumerate(self)->"Iter[Tuple[int,T]]">>>Iter('abc').enumerate().collect()[(0,'a'),(1,'b'),(2,'c')]>>>Iter([]).enumerate().collect()[]Iter.dict(self)->"Dict"In regular Python a dict can be constructed through an iterable
of tuples:>>>dict([('a',0),('b',1)]){'a':0,'b':1}Inexcitertoolswe prefer chaining so this method is a shortcut
for that:>>>d=Iter('abc').zip(count()).dict()>>>assertd=={'a':0,'b':1,'c':2}Iter.map(self, func:Union[Callable[...,C], str])->"Iter[C]"Themapfunction you all know and love.>>>Iter('abc').map(str.upper).collect()['A','B','C']>>>Iter(['abc','def']).map(str.upper).collect()['ABC','DEF']Using lambdas might seem convenient but in practice it turns
out that they make code difficult to read:>>>result=Iter('caleb').map(lambdax:(x,ord(x))).dict()>>>assertresult=={'a':97,'b':98,'c':99,'e':101,'l':108}It’s recommended that you make a separate function instead:>>>deff(x):...returnx,ord(x)>>>result=Iter('caleb').map(f).dict()>>>assertresult=={'a':97,'b':98,'c':99,'e':101,'l':108}I know many people prefer anonymous functions (often on
philosphical grounds) but in practice it’s just easier to make
a separate, named function.I’ve experimented with passing a string into the map, and usingeval()to make a lambda internally. This simplifies the code
very slightly, at the cost of using strings-as-code. I’m pretty
sure this feature will be removed so don’t use it.>>>result=Iter('caleb').map('x, ord(x)').dict()>>>assertresult=={'a':97,'b':98,'c':99,'e':101,'l':108}Iter.filter(self, function:"Optional[Callable[[T],bool]]"= None)->"Iter[T]"Themapfunction you all know and love.>>>Iter('caleb').filter(lambdax:xin'aeiou').collect()['a','e']There is a slight difference between this method signature and
the builtinfilter:  how the identity function is handled.
This is a consquence of chaining. In the function signature above
it is possible for us to give thefunctionparameter a
default value ofNonebecause the parameter appears towards
the end of the parameter list. Last, in fact.  In thebuiltin filter signatureit doesn’t allow for this because the predicate parameter appears
first.This is a long way of saying: if you just want to filter out
falsy values, no parameter is needed:>>>Iter([0,1,0,0,0,1,1,1,0,0]).filter().collect()[1,1,1,1]Using the builtin, you’d have to dofilter(None, iterable).You’ll find thatIter.mapandIter.filter(andIter.reduce, up next) work together very nicely:>>>defnot_eve(x):...returnx!='eve'>>>Iter(['bob','eve','alice']).filter(not_eve).map(str.upper).collect()['BOB','ALICE']The long chains get unwieldy so let’s rewrite that:>>>(...Iter(['bob','eve','alice'])....filter(not_eve)....map(str.upper)....collect()...)['BOB','ALICE']✨Iter.starfilter(self, function:"Optional[Callable[[T,...],bool]]"= None)->"Iter[T]"LikeIter.filter, but arg unpacking in lambdas will work.With the normalfilter, this fails:>>>Iter('caleb').enumerate().filter(lambdai,x:i>2).collect()Traceback(mostrecentcalllast):...TypeError:<lambda>()missing1requiredpositionalargument:'x'This is a real buzzkill.starfilteris very similar tostarmapin that tuples are unpacked when calling the function:>>>Iter('caleb').enumerate().starfilter(lambdai,x:i>2).collect()[(3,'e'),(4,'b')]Iter.filter_gt(self, value)->"Iter[T]"Convenience method>>>Iter([1,2,3]).filter_gt(1).collect()[2,3]Iter.filter_ge(self, value)->"Iter[T]"Convenience method>>>Iter([1,2,3]).filter_ge(2).collect()[2,3]Iter.filter_lt(self, value)->"Iter[T]"Convenience method>>>Iter([1,2,3]).filter_lt(3).collect()[1,2]Iter.filter_le(self, value)->"Iter[T]"Convenience method>>>Iter([1,2,3]).filter_le(2).collect()[1,2]Iter.filter_eq(self, value)->"Iter[T]"Convenience method>>>Iter([1,2,3]).filter_eq(2).collect()[2]Iter.filter_ne(self, value)->"Iter[T]"Convenience method>>>Iter([1,2,3]).filter_ne(2).collect()[1,3]Iter.filter_in(self, value: Sized)->"Iter[T]"Convenience method for membership testing. Note that the value
parameter must be at leastSizedbecause it gets reused
over and over for each pass of the iterator chain. For example,
passing in things likerange()will not work properly because
it will become progressively exhausted.>>>Iter([1,2,3]).filter_in([2,3,4,5]).collect()[2,3]>>>Iter([1,2,3]).filter_in(range(2,8).collect()).collect()[2,3]>>>Iter([1,2,3]).filter_in({2,3,4,5}).collect()[2,3]>>>Iter([1,2,3]).filter_in(dict.fromkeys({2,3,4,5})).collect()[2,3]Iter.filter_ni(self, value)->"Iter[T]"Convenience method for membership testing. Note that the value
parameter must be at leastSizedbecause it gets reused
over and over for each pass of the iterator chain. For example,
passing in things likerange()will not work properly because
it will become progressively exhausted.>>>Iter([1,2,3]).filter_ni([2,3,4,5]).collect()[1]>>>Iter([1,2,3]).filter_ni(range(2,8).collect()).collect()[1]>>>Iter([1,2,3]).filter_ni({2,3,4,5}).collect()[1]>>>Iter([1,2,3]).filter_ni(dict.fromkeys({2,3,4,5})).collect()[1]🎧Iter.reduce(self, func:Callable[...,T], *args)->"T"Thereducefunction you all know and…hang on, actuallyreduceis rather unloved. In the past I’ve found it very complex
to reason about, when looking at a bunch of nested function calls
in typicalitertoolscode. Hopefully iterable chaining makes
it easier to read code that usesreduce?Let’s check, does this make sense?>>>payments=[...('bob',100),...('alice',50),...('eve',-100),...('bob',19.95),...('bob',-5.50),...('eve',11.95),...('eve',200),...('alice',-45),...('alice',-67),...('bob',1.99),...('alice',89),...]>>>(...Iter(payments)....filter(lambdaentry:entry[0]=='bob')....map(lambdaentry:entry[1])....reduce(lambdatotal,value:total+value,0)...)116.44I intentionally omitted comments above so that you can try the
“readability experiment”, but in practice you would definitely
want to add some comments on these chains:>>>(...# Iterate over all payments...Iter(payments)...# Only look at bob's payments....filter(lambdaentry:entry[0]=='bob')...# Extract the value of the payment....map(lambdaentry:entry[1])...# Add all those payments together....reduce(lambdatotal,value:total+value,0)...)116.44reduceis a quite crude low-level tool. In many cases you’ll
find that there are other functions and methods better suited
to the situations you’ll encounter most often. For example,
there is alreadyIter.sumif you just want to add up numbers,
and it’s much easier to useIter.groupbyfor grouping than
to try to make that work withIter.reduce. Youcanmake it
work but it’ll be easier to useIter.groupby.🎧Iter.starreduce(self, function:Callable[...,T], initializer=0)->"T"Iter.starreduceis the same asIter.reduceexcept that args are
star-unpacked when passed intofunction. This is frequently
more convenient than the default behaviour.We can see this using the same example shown forIter.reduce.
The star unpacking makes it easier to just do the filtering
directly inside the reducer function.>>>payments=[...('bob',100),...('alice',50),...('eve',-100),...('bob',19.95),...('bob',-5.50),...('eve',11.95),...('eve',200),...('alice',-45),...('alice',-67),...('bob',1.99),...('alice',89),...]>>>(...Iter(payments)....starreduce(...lambdatot,name,value:tot+valueifname=='bob'elsetot,...0...)...)116.44This is how that looks if you avoid a lambda:>>>deff(tot,name,value):...ifname=='bob':...returntot+value...else:...returntot>>>Iter(payments).starreduce(f)116.44🎧Iter.sum(self)Exactly what you expect:>>>Iter(range(10)).sum()45🎧Iter.concat(self, glue: AnyStr = '')->"AnyStr"Joining strings (and bytes).>>>Iter(['hello','there']).concat()'hellothere'>>>Iter(['hello','there']).concat(' ')'hello there'>>>Iter(['hello','there']).concat(',')'hello,there'>>>Iter([b'hello',b'there']).concat(b',')b'hello,there'Iter.insert(self, glue: C)->"Iter[Union[C,T]]"Docstring TBD🎤@classmethod Iter.count(cls, *args)->"Iter[int]">>>Iter.count().take(3).collect()[0,1,2]>>>Iter.count(100).take(3).collect()[100,101,102]>>>Iter.count(100,2).take(3).collect()[100,102,104]♾Iter.cycle(self)->"Iter[T]">>>Iter('abc').cycle().take(8).collect()['a','b','c','a','b','c','a','b']>>>Iter('abc').cycle().take(8).concat('')'abcabcab'🎤 ♾@classmethod Iter.repeat(cls, elem: C, times=None)->"Iter[C]">>>Iter.repeat('c',times=3).collect()['c','c','c']Iter.accumulate(self, func=None, *, initial=None)Docstring TBD>>>Iter([1,2,3,4,5]).accumulate().collect()[1,3,6,10,15]>>>ifsys.version_info>=(3,8):...out=Iter([1,2,3,4,5]).accumulate(initial=100).collect()...assertout==[100,101,103,106,110,115]>>>Iter([1,2,3,4,5]).accumulate(operator.mul).collect()[1,2,6,24,120]Iter.chain(self, *iterables: Iterable[T])->"Iter[T]"Docstring TBD>>>Iter('ABC').chain('DEF').collect()['A','B','C','D','E','F']>>>Iter('ABC').chain().collect()['A','B','C']Iter.chain_from_iterable(self)->"Iter[T]"Docstring TBD>>>Iter(['ABC','DEF']).chain_from_iterable().collect()['A','B','C','D','E','F']Iter.compress(self, selectors)Replacement for the itertoolscompressfunction.  This version returns
an instance ofIterto allow further iterable chaining.>>>Iter('ABCDEF').compress([1,0,1,0,1,1]).collect()['A','C','E','F']Iter.dropwhile(self, pred)Docstring TBDIter.filterfalse(self, pred)Docstring TBDIter.groupby(self, key=None)Docstring TBDIter.islice(self, *args)->"Iter"Docstring TBDIter.starmap(self, func)Docstring TBDIter.takewhile(self, pred)Docstring TBDIter.tee(self, n=2)Docstring TBDIter.zip_longest(self, *iterables, fillvalue=None)Docstring TBDIter.chunked(self, n: int)->"Iter"Docstring TBDIter.ichunked(self, n: int)->"Iter"Docstring TBD@classmethod Iter.sliced(cls, seq: Sequence, n: int)->"Iter"Docstring TBDIter.distribute(self, n: int)->"Iter"Docstring TBDIter.divide(self, n: int)->"Iter"Docstring TBDIter.split_at(self, pred)Docstring TBDIter.split_before(self, pred)Docstring TBDIter.split_after(self, pred)Docstring TBDIter.split_into(self, sizes)Docstring TBDIter.split_when(self, pred)Docstring TBDIter.bucket(self, key, validator=None)This is the basic example, copied from the more-itertools
docs:>>>iterable=['a1','b1','c1','a2','b2','c2','b3']>>>b=Iter(iterable).bucket(key=lambdax:x[0])>>>sorted(b)['a','b','c']>>>list(b['a'])['a1','a2']Note that once consumed, you can’t iterate over the contents
of a group again.Iter.unzip(self)Docstring TBDIter.grouper(self, n: int, fillvalue=None)->"Iter"Docstring TBDIter.partition(self, pred)->"Iter"Docstring TBDIter.spy(self, n=1)->"Tuple[Iter, Iter]"Docstring TBDIter.peekable(self)->"more_itertools.peekable"Docstring TBD>>>p=Iter(['a','b']).peekable()>>>p.peek()'a'>>>next(p)'a'The peekable can be used to inspect what will be coming up.
But if you then want to resume iterator chaining, pass the
peekable back into anIterinstance.>>>p=Iter(range(10)).peekable()>>>p.peek()0>>>Iter(p).take(3).collect()[0,1,2]A peekable is not anIterinstance so it doesn’t provide
the iterator chaining methods. But if you want to get into
chaining, use theiter()method.>>>p=Iter(range(5)).peekable()>>>p.peek()0>>>p[1]1>>>p.iter().take(3).collect()[0,1,2]Peekables can be prepended. But then you usually want to go
right back to iterator chaining. Thus, theprependmethod
(on the returnedpeekableinstance) returns anIterinstance.>>>p=Iter(range(3)).peekable()>>>p.peek()0>>>p.prepend('a','b').take(4).collect()['a','b',0,1]Iter.seekable(self)->"more_itertools.seekable"Docstring TBDIter.windowed(self, n, fillvalue=None, step=1)->"Iter"Docstring TBDIter.substrings(self)Docstring TBDIter.substrings_indexes(self, reverse=False)Docstring TBDIter.stagger(self,offsets=(-1,0, 1), longest=False, fillvalue=None)>>>Iter([0,1,2,3]).stagger().collect()[(None,0,1),(0,1,2),(1,2,3)]>>>Iter(range(8)).stagger(offsets=(0,2,4)).collect()[(0,2,4),(1,3,5),(2,4,6),(3,5,7)]>>>Iter([0,1,2,3]).stagger(longest=True).collect()[(None,0,1),(0,1,2),(1,2,3),(2,3,None),(3,None,None)]Iter.pairwise(self)Referencemore_itertools.pairwise>>>Iter.count().pairwise().take(4).collect()[(0,1),(1,2),(2,3),(3,4)]Iter.count_cycle(self, n=None)->"Iter"Reference:more_itertools.count_cycle>>>Iter('AB').count_cycle(3).collect()[(0,'A'),(0,'B'),(1,'A'),(1,'B'),(2,'A'),(2,'B')]Iter.intersperse(self, e, n=1)->"Iter"Reference:more_itertools.intersperse>>>Iter([1,2,3,4,5]).intersperse('!').collect()[1,'!',2,'!',3,'!',4,'!',5]>>>Iter([1,2,3,4,5]).intersperse(None,n=2).collect()[1,2,None,3,4,None,5]Iter.padded(self, fillvalue: Optional[C] = None, n: Optional[int] = None, next_multiple: bool = False, )->"Iter[Union[T,C]]"Reference:more_itertools.padded>>>Iter([1,2,3]).padded('?',5).collect()[1,2,3,'?','?']>>>Iter([1,2,3,4]).padded(n=3,next_multiple=True).collect()[1,2,3,4,None,None]Iter.repeat_last(self, default=None)->"Iter[T]"Reference:more_itertools.repeat_last>>>Iter(range(3)).repeat_last().islice(5).collect()[0,1,2,2,2]>>>Iter(range(0)).repeat_last(42).islice(5).collect()[42,42,42,42,42]Iter.adjacent(self, pred, distance=1)->"Iter[Tuple[bool,T]]"Reference:more_itertools.adjacent>>>Iter(range(6)).adjacent(lambdax:x==3).collect()[(False,0),(False,1),(True,2),(True,3),(True,4),(False,5)]>>>Iter(range(6)).adjacent(lambdax:x==3,distance=2).collect()[(False,0),(True,1),(True,2),(True,3),(True,4),(True,5)]Iter.groupby_transform(self, keyfunc:Optional[Callable[...,K]] = None, valuefunc:Optional[Callable[...,V]] = None, )->"Iter[Tuple[K,Iterable[V]]]"Reference:more_itertools.groupby_transformThis example has been modified somewhat from the original. We’re usingstarmaphere to “unzip” the tuples produced by the group
transform.>>>iterable='AaaABbBCcA'>>>keyfunc=lambdax:x.upper()>>>valuefunc=lambdax:x.lower()>>>(...Iter(iterable)....groupby_transform(keyfunc,valuefunc)....starmap(lambdak,g:(k,''.join(g)))....collect()...)[('A','aaaa'),('B','bbb'),('C','cc'),('A','a')]>>>fromoperatorimportitemgetter>>>keys=[0,0,1,1,1,2,2,2,3]>>>values='abcdefghi'>>>iterable=zip(keys,values)>>>(...Iter(iterable)....groupby_transform(itemgetter(0),itemgetter(1))....starmap(lambdak,g:(k,''.join(g)))....collect()...)[(0,'ab'),(1,'cde'),(2,'fgh'),(3,'i')]Iter.padnone(self)->"Iter[Union[T,None]]"Reference:more_itertools.padnone>>>Iter(range(3)).padnone().take(5).collect()[0,1,2,None,None]Iter.ncycles(self, n)->"Iter[T]"Reference:more_itertools.ncycles>>>Iter(['a','b']).ncycles(3).collect()['a','b','a','b','a','b']Iter.collapse(self, base_type=None, levels=None)->"Iter"Reference:more_itertools.collapse>>>iterable=[(1,2),([3,4],[[5],[6]])]>>>Iter(iterable).collapse().collect()[1,2,3,4,5,6]>>>iterable=['ab',('cd','ef'),['gh','ij']]>>>Iter(iterable).collapse(base_type=tuple).collect()['ab',('cd','ef'),'gh','ij']>>>iterable=[('a',['b']),('c',['d'])]>>>Iter(iterable).collapse().collect()# Fully flattened['a','b','c','d']>>>Iter(iterable).collapse(levels=1).collect()# Only one level flattened['a',['b'],'c',['d']]@class_or_instancemethod Iter.sort_together(self_or_cls, iterables,key_list=(0,),reverse=False)Reference:more_itertools.sort_togetherThis can be called either as an instance method or a class method.
The classmethod form is more convenient if all the iterables are
already available. The instancemethod form is more convenient if
one of the iterables already goes through some transformation.Here are examples from the classmethod form, which mirror the
examples in themore-itertoolsdocumentation:>>>iterables=[(4,3,2,1),('a','b','c','d')]>>>Iter.sort_together(iterables).collect()[(1,2,3,4),('d','c','b','a')]>>>iterables=[(3,1,2),(0,1,0),('c','b','a')]>>>Iter.sort_together(iterables,key_list=(1,2)).collect()[(2,3,1),(0,0,1),('a','c','b')]>>>Iter.sort_together([(1,2,3),('c','b','a')],reverse=True).collect()[(3,2,1),('a','b','c')]Here is an examples using the instancemethod form:>>>iterables=[('a','b','c','d')]>>>Iter([4,3,2,1]).sort_together(iterables).collect()[(1,2,3,4),('d','c','b','a')]@class_or_instancemethod Iter.interleave(self_or_cls, *iterables)->"Iter"Reference:more_itertools.interleaveClassmethod form:>>>Iter.interleave([1,2,3],[4,5],[6,7,8]).collect()[1,4,6,2,5,7]Instancemethod form:>>>Iter([1,2,3]).interleave([4,5],[6,7,8]).collect()[1,4,6,2,5,7]@class_or_instancemethod Iter.interleave_longest(self_or_cls, *iterables)->"Iter"Reference:more_itertools.interleave_longestClassmethod form:>>>Iter.interleave_longest([1,2,3],[4,5],[6,7,8]).collect()[1,4,6,2,5,7,3,8]Instancemethod form:>>>Iter([1,2,3]).interleave_longest([4,5],[6,7,8]).collect()[1,4,6,2,5,7,3,8]@classmethod Iter.zip_offset(cls, *iterables, offsets, longest=False, fillvalue=None)->"Iter"Reference:more_itertools.zip_offset>>>Iter.zip_offset('0123','abcdef',offsets=(0,1)).collect()[('0','b'),('1','c'),('2','d'),('3','e')]>>>Iter.zip_offset('0123','abcdef',offsets=(0,1),longest=True).collect()[('0','b'),('1','c'),('2','d'),('3','e'),(None,'f')]Iter.dotproduct(self, vec2: Iterable)Reference:more_itertools.dotproduct>>>Iter([10,10]).dotproduct([20,20])400Iter.flatten(self)->"Iter[T]"Reference:more_itertools.flatten>>>Iter([[0,1],[2,3]]).flatten().collect()[0,1,2,3]@class_or_instancemethod Iter.roundrobin(self_or_cls: Union[Type[T], T], *iterables: C)->"Iter[Union[T,C]]"Reference:more_itertools.roundrobinClassmethod form:>>>Iter.roundrobin('ABC','D','EF').collect()['A','D','E','B','F','C']Instancemethod form:>>>Iter('ABC').roundrobin('D','EF').collect()['A','D','E','B','F','C']Iter.prepend(self, value: C)->"Iter[Union[T,C]]"Reference:more_itertools.prepend>>>value='0'>>>iterator=['1','2','3']>>>Iter(iterator).prepend(value).collect()['0','1','2','3']🎧Iter.ilen(self)->"int"Reference:more_itertools.ilen>>>Iter(xforxinrange(1000000)ifx%3==0).ilen()333334Iter.unique_to_each(self)->"Iter[T]"Reference:more_itertools.unique_to_each>>>Iter([{'A','B'},{'B','C'},{'B','D'}]).unique_to_each().collect()[['A'],['C'],['D']]>>>Iter(["mississippi","missouri"]).unique_to_each().collect()[['p','p'],['o','u','r']]Iter.sample(self, k=1, weights=None)->"Iter"Reference:more_itertools.sample>>>iterable=range(100)>>>Iter(iterable).sample(5).collect()[81,60,96,16,4]>>>iterable=range(100)>>>weights=(i*i+1foriinrange(100))>>>Iter(iterable).sample(5,weights=weights)[79,67,74,66,78]>>>data="abcdefgh">>>weights=range(1,len(data)+1)>>>Iter(data).sample(k=len(data),weights=weights)['c','a','b','e','g','d','h','f']>>># This one just to let the doctest run>>>iterable=range(100)>>>Iter(iterable).sample(5).map(lambdax:0<=x<100).all()TrueIter.consecutive_groups(self, ordering=lambda x: x)Reference:more_itertools.consecutive_groups>>>iterable=[1,10,11,12,20,30,31,32,33,40]>>>Iter(iterable).consecutive_groups().map(lambdag:list(g)).print('{v}').consume()[1][10,11,12][20][30,31,32,33][40]Iter.run_length_encode(self)->"Iter[Tuple[T,int]]"Reference:more_itertools.run_length>>>uncompressed='abbcccdddd'>>>Iter(uncompressed).run_length_encode().collect()[('a',1),('b',2),('c',3),('d',4)]Iter.run_length_decode(self)->"Iter"Reference:more_itertools.run_length>>>compressed=[('a',1),('b',2),('c',3),('d',4)]>>>Iter(compressed).run_length_decode().collect()['a','b','b','c','c','c','d','d','d','d']Iter.map_reduce(self, keyfunc, valuefunc=None, reducefunc=None)->"Dict"Reference:more_itertools.map_reduceThis interface mirrors whatmore-itertoolsdoes in that it returns
a dict. Seemap_reduce_it()for a slightly-modified interface
that returns the dict items as another iterator.>>>keyfunc=lambdax:x.upper()>>>d=Iter('abbccc').map_reduce(keyfunc)>>>sorted(d.items())[('A',['a']),('B',['b','b']),('C',['c','c','c'])]>>>keyfunc=lambdax:x.upper()>>>valuefunc=lambdax:1>>>d=Iter('abbccc').map_reduce(keyfunc,valuefunc)>>>sorted(d.items())[('A',[1]),('B',[1,1]),('C',[1,1,1])]>>>keyfunc=lambdax:x.upper()>>>valuefunc=lambdax:1>>>reducefunc=sum>>>d=Iter('abbccc').map_reduce(keyfunc,valuefunc,reducefunc)>>>sorted(d.items())[('A',1),('B',2),('C',3)]Note the warning given in themore-itertoolsdocs about how
lists are created before the reduce step. This means you always want
to filterbeforeapplying map_reduce, not after.>>>all_items=_range(30)>>>keyfunc=lambdax:x%2# Evens map to 0; odds to 1>>>categories=Iter(all_items).filter(lambdax:10<=x<=20).map_reduce(keyfunc=keyfunc)>>>sorted(categories.items())[(0,[10,12,14,16,18,20]),(1,[11,13,15,17,19])]>>>summaries=Iter(all_items).filter(lambdax:10<=x<=20).map_reduce(keyfunc=keyfunc,reducefunc=sum)>>>sorted(summaries.items())[(0,90),(1,75)]Iter.map_reduce_it(self, keyfunc:Callable[...,K], valuefunc:Optional[Callable[...,V]] = None, reducefunc:Optional[Callable[...,R]] = None, )->"Iter[Tuple[K,R]]"Reference:more_itertools.map_reduce>>>keyfunc=lambdax:x.upper()>>>Iter('abbccc').map_reduce_it(keyfunc).collect()[('A',['a']),('B',['b','b']),('C',['c','c','c'])]>>>keyfunc=lambdax:x.upper()>>>valuefunc=lambdax:1>>>Iter('abbccc').map_reduce_it(keyfunc,valuefunc).collect()[('A',[1]),('B',[1,1]),('C',[1,1,1])]>>>keyfunc=lambdax:x.upper()>>>valuefunc=lambdax:1>>>reducefunc=sum>>>Iter('abbccc').map_reduce_it(keyfunc,valuefunc,reducefunc).collect()[('A',1),('B',2),('C',3)]🎧Iter.exactly_n(self, n, predicate=bool)->"bool"Docstring TBD>>>Iter([True,True,False]).exactly_n(2)TrueIter.all_equal(self)Iter.first_true(self)Iter.quantify(self)Iter.islice_extended(self, *args)Reference:more_itertools.islice_extended>>>Iter('abcdefgh').islice_extended(-4,-1).collect()['e','f','g']>>>Iter.count().islice_extended(110,99,-2).collect()[110,108,106,104,102,100]Iter.first(self)Reference:more_itertools.firstIter.last(self)Reference:more_itertools.lastIter.one(self)Reference:more_itertools.oneIter.only(self, default=None, too_long=ValueError)->"T"Reference:more_itertools.one>>>Iter([]).only(default='missing')'missing'>>>Iter([42]).only(default='missing')42>>>Iter([1,2]).only()Traceback(mostrecentcalllast):...ValueError:...Iter.strip(self, pred)->"Iter[T]"Reference:more_itertools.strip>>>iterable=(None,False,None,1,2,None,3,False,None)>>>pred=lambdax:xin{None,False,''}>>>Iter(iterable).strip(pred).collect()[1,2,None,3]Iter.lstrip(self, pred)->"Iter[T]"Reference:more_itertools.lstrip>>>iterable=(None,False,None,1,2,None,3,False,None)>>>pred=lambdax:xin{None,False,''}>>>Iter(iterable).lstrip(pred).collect()[1,2,None,3,False,None]Iter.rstrip(self, pred)->"Iter[T]"Reference:more_itertools.rstrip>>>iterable=(None,False,None,1,2,None,3,False,None)>>>pred=lambdax:xin{None,False,''}>>>Iter(iterable).rstrip(pred).collect()[None,False,None,1,2,None,3]Iter.filter_except(self, validator, *exceptions)->"Iter[T]"Reference:more_itertools.filter_except>>>iterable=['1','2','three','4',None]>>>Iter(iterable).filter_except(int,ValueError,TypeError).collect()['1','2','4']Iter.map_except(self, function, *exceptions)->"Iter"Reference:more_itertools.map_except>>>iterable=['1','2','three','4',None]>>>Iter(iterable).map_except(int,ValueError,TypeError).collect()[1,2,4]Iter.nth_or_last(self, n, default=_marker)->"T"Reference:more_itertools.nth_or_last>>>Iter([0,1,2,3]).nth_or_last(2)2>>>Iter([0,1]).nth_or_last(2)1>>>Iter([]).nth_or_last(0,'some default')'some default'Iter.nth(self, n, default=None)Reference:more_itertools.nthIter.take(self, n: int)->"Iter"Reference:more_itertools.takeIter.tail(self, n)->"Iter[T]"Reference:more_itertools.tail>>>Iter('ABCDEFG').tail(3).collect()['E','F','G']Iter.unique_everseen(self, key=None)->"Iter[T]"Reference:more_itertools.unique_everseen>>>Iter('AAAABBBCCDAABBB').unique_everseen().collect()['A','B','C','D']>>>Iter('ABBCcAD').unique_everseen(key=str.lower).collect()['A','B','C','D']Be sure to read themore-itertoolsdocs whne using unhashable
items.>>>iterable=([1,2],[2,3],[1,2])>>>Iter(iterable).unique_everseen().collect()# Slow[[1,2],[2,3]]>>>Iter(iterable).unique_everseen(key=tuple).collect()# Faster[[1,2],[2,3]]Iter.unique_justseen(self, key=None)->"Iter[T]"Reference:more_itertools.unique_justseen>>>Iter('AAAABBBCCDAABBB').unique_justseen().collect()['A','B','C','D','A','B']>>>Iter('ABBCcAD').unique_justseen(key=str.lower).collect()['A','B','C','A','D']Iter.distinct_permutations(self)Reference:more_itertools.distinct_permutations>>>Iter([1,0,1]).distinct_permutations().sorted().collect()[(0,1,1),(1,0,1),(1,1,0)]Iter.distinct_combinations(self, r)->"Iter[T]"Reference:more_itertools.distinct_combinations>>>Iter([0,0,1]).distinct_combinations(2).collect()[(0,0),(0,1)]Iter.circular_shifts(self)->"Iter[T]"Reference:more_itertools.circular_shifts>>>Iter(range(4)).circular_shifts().collect()[(0,1,2,3),(1,2,3,0),(2,3,0,1),(3,0,1,2)]Iter.partitions(self)->"Iter[T]"Reference:more_itertools.partitions>>>Iter('abc').partitions().collect()[[['a','b','c']],[['a'],['b','c']],[['a','b'],['c']],[['a'],['b'],['c']]]>>>Iter('abc').partitions().print('{v}').consume()[['a','b','c']][['a'],['b','c']][['a','b'],['c']][['a'],['b'],['c']]>>>Iter('abc').partitions().map(lambdav:[''.join(p)forpinv]).print('{v}').consume()['abc']['a','bc']['ab','c']['a','b','c']Iter.set_partitions(self, k=None)->"Iter[T]"Reference:more_itertools.set_partitions>>>Iter('abc').set_partitions(2).collect()[[['a'],['b','c']],[['a','b'],['c']],[['b'],['a','c']]]Iter.powerset(self)Reference:more_itertools.powerset>>>Iter([1,2,3]).powerset().collect()[(),(1,),(2,),(3,),(1,2),(1,3),(2,3),(1,2,3)]@class_or_instancemethod Iter.random_product(self_or_cls, *args, repeat=1)Reference:more_itertools.random_product>>>Iter('abc').random_product(range(4),'XYZ').collect()['c',3,'X']>>>Iter.random_product('abc',range(4),'XYZ').collect()['c',0,'Z']>>>Iter('abc').random_product(range(0)).collect()Traceback(mostrecentcalllast):...IndexError:Cannotchoosefromanemptysequence>>>Iter.random_product(range(0)).collect()Traceback(mostrecentcalllast):...IndexError:CannotchoosefromanemptysequenceIter.random_permutation(self, r=None)Reference:more_itertools.random_permutation>>>Iter(range(5)).random_permutation().collect()[2,0,4,3,1]>>>Iter(range(0)).random_permutation().collect()[]Iter.random_combination(self, r)Reference:more_itertools.random_combination>>>Iter(range(5)).random_combination(3).collect()[0,1,4]>>>Iter(range(5)).random_combination(0).collect()[]Iter.random_combination_with_replacement(self, r)Reference:more_itertools.random_combination_with_replacement>>>Iter(range(3)).random_combination_with_replacement(5).collect()[0,0,1,2,2]>>>Iter(range(3)).random_combination_with_replacement(0).collect()[]Iter.nth_combination(self, r, index)Reference:more_itertools.nth_combination>>>Iter(range(9)).nth_combination(3,1).collect()[0,1,3]>>>Iter(range(9)).nth_combination(3,2).collect()[0,1,4]>>>Iter(range(9)).nth_combination(3,3).collect()[0,1,5]>>>Iter(range(9)).nth_combination(4,3).collect()[0,1,2,6]>>>Iter(range(9)).nth_combination(3,7).collect()[0,2,3]@classmethod Iter.always_iterable(cls, obj,base_type=(str,bytes))->'Iter'Reference:more_itertools.always_iterable>>>Iter.always_iterable([1,2,3]).collect()[1,2,3]>>>Iter.always_iterable(1).collect()[1]>>>Iter.always_iterable(None).collect()[]>>>Iter.always_iterable('foo').collect()['foo']>>>Iter.always_iterable(dict(a=1),base_type=dict).collect()[{'a':1}]Iter.always_reversible(self)Reference:more_itertools.always_reversibleThis is likereversed()but it also operates on things that
wouldn’t normally be reversible, like generators. It does this with
internal caching, so be careful with memory use.@classmethod Iter.with_iter(cls, context_manager)Reference:more_itertools.with_iterNote: Any context manager which returns an iterable is a candidate forIter.with_iter.>>>importtempfile>>>withtempfile.TemporaryDirectory()astd:...withopen(td+'text.txt','w')asf:...f.writelines(['abc\n','def\n','ghi\n'])...Iter.with_iter(open(td+'text.txt')).map(lambdax:x.upper()).collect()['ABC\n','DEF\n','GHI\n']See also:Iter.open🛠 TODO: perhaps we should get rid ofIter.openand just use this?@classmethod Iter.iter_except(cls, func, exception, first=None)->"Iter"Reference:more_itertools.iter_except>>>l=[0,1,2]>>>Iter.iter_except(l.pop,IndexError).collect()[2,1,0]Iter.locate(self, pred=bool, window_size=None)->"Iter"Reference:more_itertools.locate>>>Iter([0,1,1,0,1,0,0]).locate().collect()[1,2,4]>>>Iter(['a','b','c','b']).locate(lambdax:x=='b').collect()[1,3]>>>iterable=[0,1,2,3,0,1,2,3,0,1,2,3]>>>pred=lambda*args:args==(1,2,3)>>>Iter(iterable).locate(pred=pred,window_size=3).collect()[1,5,9]>>>fromitertoolsimportcount>>>frommore_itertoolsimportseekable>>>source=(3*n+1if(n%2)elsen//2fornincount())>>>it=Iter(source).seekable()>>>pred=lambdax:x>100>>># TODO: can we avoid making two instances?>>>indexes=Iter(it).locate(pred=pred)>>>i=next(indexes)>>>it.seek(i)>>>next(it)106Iter.rlocate(self, pred=bool, window_size=None)->"Iter"Reference:more_itertools.rlocate>>>Iter([0,1,1,0,1,0,0]).rlocate().collect()# Truthy at 1, 2, and 4[4,2,1]>>>pred=lambdax:x=='b'>>>Iter('abcb').rlocate(pred).collect()[3,1]>>>iterable=[0,1,2,3,0,1,2,3,0,1,2,3]>>>pred=lambda*args:args==(1,2,3)>>>Iter(iterable).rlocate(pred=pred,window_size=3).collect()[9,5,1]Iter.replace(self, pred, substitutes, count=None, window_size=1)->"Iter"Reference:more_itertools.replace>>>iterable=[1,1,0,1,1,0,1,1]>>>pred=lambdax:x==0>>>substitutes=(2,3)>>>Iter(iterable).replace(pred,substitutes).collect()[1,1,2,3,1,1,2,3,1,1]>>>iterable=[1,1,0,1,1,0,1,1,0]>>>pred=lambdax:x==0>>>substitutes=[None]>>>Iter(iterable).replace(pred,substitutes,count=2).collect()[1,1,None,1,1,None,1,1,0]>>>iterable=[0,1,2,5,0,1,2,5]>>>window_size=3>>>pred=lambda*args:args==(0,1,2)# 3 items passed to pred>>>substitutes=[3,4]# Splice in these items>>>Iter(iterable).replace(...pred,substitutes,window_size=window_size...).collect()[3,4,5,3,4,5]@classmethod Iter.numeric_range(cls, *args)->"Iter"Reference:more_itertools.numeric_range>>>Iter.numeric_range(3.5).collect()[0.0,1.0,2.0,3.0]>>>fromdecimalimportDecimal>>>start=Decimal('2.1')>>>stop=Decimal('5.1')>>>Iter.numeric_range(start,stop).collect()[Decimal('2.1'),Decimal('3.1'),Decimal('4.1')]>>>fromfractionsimportFraction>>>start=Fraction(1,2)# Start at 1/2>>>stop=Fraction(5,2)# End at 5/2>>>step=Fraction(1,2)# Count by 1/2>>>Iter.numeric_range(start,stop,step).collect()[Fraction(1,2),Fraction(1,1),Fraction(3,2),Fraction(2,1)]>>>Iter.numeric_range(3,-1,-1.0).collect()[3.0,2.0,1.0,0.0]Iter.side_effect(self, func, *args, chunk_size=None, before=None, after=None)Reference:more_itertools.side_effect>>>deff(item):...ifitem==3:...raiseException('got 3')>>>Iter.range(5).side_effect(f).consume()Traceback(mostrecentcalllast):...Exception:got3>>>func=lambdaitem:print('Received{}'.format(item))>>>Iter.range(2).side_effect(func).consume()Received0Received1This version ofside_effectalso allows extra args:>>>func=lambdaitem,format_str='Received{}':print(format_str.format(item))>>>Iter.range(2).side_effect(func).consume()Received0Received1>>>func=lambdaitem,format_str='Received{}':print(format_str.format(item))>>>Iter.range(2).side_effect(func,'Got{}').consume()Got0Got1Iter.iterate(self)Iter.difference(self, func=operator.sub, *, initial=None)Reference:more_itertools.difference>>>iterable=[0,1,3,6,10]>>>Iter(iterable).difference().collect()[0,1,2,3,4]>>>iterable=[1,2,6,24,120]# Factorial sequence>>>func=lambdax,y:x//y>>>Iter(iterable).difference(func).collect()[1,2,3,4,5]Iter.make_decorator(self)Iter.SequenceView(self)Iter.time_limited(self, limit_seconds)->"Iter"Reference:more_itertools.time_limited>>>fromtimeimportsleep>>>defgenerator():...yield1...yield2...sleep(0.2)...yield3>>>Iter(generator()).time_limited(0.1).collect()[1,2]🎧Iter.consume(self, n: Optional[int] = None)->"Optional[Iter[T]]"If n is not provided, the entire iterator is consumed andNoneis returned. Otherwise, an iterator willalwaysbe
returned, even if n is greater than the number of items left in
the iterator.In this example, the source has more elements than what we consume,
so there will still be data available on the chain:>>>range(10).consume(5).collect()[5,6,7,8,9]We can bump up the count of how many items can be consumed. Note that
even thoughnis greater than the number of items in the source,
it is still required to callIter.collectto consume the remaining
items.>>>range(10).consume(50).collect()[]Finally, ifnis not provided, the entire stream is consumed.
In this scenario,Iter.collectwould fail since nothing is being
returned from the consume call.>>>assertrange(10).consume()isNoneIter.tabulate(self)🎤@classmethod Iter.repeatfunc(cls, func, *args, times=None)Docstring TBD>>>Iter.repeatfunc(operator.add,3,5,times=4).collect()[8,8,8,8]Iter.wrap(self, ends: "Sequence[T, T]" ="()")Other examples for ends: ‘”’ * 2, or ‘`’ * 2, or ‘[]’ etc.Iter.print(self,template="{i}:{v}")->"Iter[T]"Printing during the execution of an iterator. Mostly useful
for debugging. Returns another iterator instance through which
the original data is passed unchanged. This means you can include
aprint()step as necessary to observe data during iteration.>>>Iter('abc').print().collect()0:a1:b2:c['a','b','c']>>>(...Iter(range(5))....print('before filter{i}:{v}')....filter(lambdax:x>2)....print('after filter{i}:{v}')....collect()...)beforefilter0:0beforefilter1:1beforefilter2:2beforefilter3:3afterfilter0:3beforefilter4:4afterfilter1:4[3,4]🎤@classmethod Iter.from_queue(cls, q: queue.Queue, timeout=None, sentinel=None)Wrap a queue with an iterator interface. This allows it to participate
in chaining operations. The iterator will block while waiting for
new values to appear on the queue. This is useful: it allows you
to easily and safely pass data between threads or processes, and
feed the incoming data into a pipeline.The sentinel value, defaultNone, will terminate the iterator.>>>q=queue.Queue()>>># This line puts stuff onto a queue>>>range(10).chain([None]).map(q.put).consume()>>># This is where we consume data from the queue:>>>Iter.from_queue(q).filter(lambdax:2<x<9).collect()[3,4,5,6,7,8]IfNonehad not been chained onto the data, the iterator would
have waited inIter.collectforever.Iter.into_queue(self, q: queue.Queue)->"Iter[T]"This is a sink, likeIter.collect, that consumes data from
an iterator chain and puts the data into the given queue.>>>q=queue.Queue()>>># This demonstrates the queue sink>>>range(5).into_queue(q).consume()>>># Code below is only for verification>>>out=[]>>>finished=False>>>whilenotfinished:...try:...out.append(q.get_nowait())...exceptqueue.Empty:...finished=True>>>out[0,1,2,3,4]🎧Iter.send(self, collector: Generator, close_collector_when_done=False)->"None"See also:more_itertools.consumerSend data into a generator. You do not have to first callnext()on the generator.Iter.sendwill do this for you.⚠ Look carefully at the examples below; you’ll see that theyieldkeyword is wrapped in a second set of parens, e.g.output.append((yield)). This is required!Simple case:>>>output=[]>>>defcollector():...whileTrue:...output.append((yield))>>>Iter.range(3).send(collector())>>>output[0,1,2]Note that the generator isnotclosed by default after the iterable is
exhausted. But this can be changed. If you choose to close the
generator, use the parameter:>>>output=[]>>>defcollector():...whileTrue:...output.append((yield))>>>g=collector()>>>Iter.range(3).send(g,close_collector_when_done=True)>>>Iter.range(3).send(g)Traceback(mostrecentcalllast):...StopIterationThe default behaviour is that the generator is left open which means you
can keep using it for other iterators:>>>output=[]>>>defcollector():...whileTrue:...output.append((yield))>>>g=collector()>>>Iter.range(3).send(g)>>>Iter.range(10,13).send(g)>>>Iter.range(100,103).send(g)>>>output[0,1,2,10,11,12,100,101,102]If the generator is closed before the iteration is complete,
you’ll get aStopIterationexception:>>>output=[]>>>defcollector():...foriinrange(3):...output.append((yield))>>>Iter.range(5).send(collector())Traceback(mostrecentcalllast):...StopIterationNote thatIter.sendis a sink, so no further chaining is allowed.Iter.send_also(self, collector: Generator)->"Iter"Reference:more_itertools.consumerSome ideas around a reverse iterator as a sink. Usually you have
first to “send” aNoneinto a generator if you want to send
more values into it (or callnext()on it), but we handle
that automatically.Simple case:>>>output=[]>>>defcollector():...whileTrue:...output.append((yield))>>>Iter.range(3).send_also(collector()).collect()[0,1,2]>>>output[0,1,2]However, if the caller already started the generator, that
works too:>>>output=[]>>>defcollector():...whileTrue:...output.append((yield))>>>g=collector()>>>next(g)# This "starts" the generator>>>Iter.range(3).send_also(g).collect()[0,1,2]>>>output[0,1,2]If the generator is closed before the iteration is complete,
you’ll get an exception (Python 3.7+):>>>output=[]>>>defcollector():...foriinbuiltins.range(3):...output.append((yield))>>>Iter.range(50).send_also(collector()).collect()Traceback(mostrecentcalllast):...RuntimeErrorNote that the above doesn’t happen in Python < 3.7 (which includes
pypy 7.3.1 that matches Python 3.6.9 compatibility). Instead, you
collect out the items up to until the point that the collector
returns; in this case, you’d get [0, 1, 2]. This change was made
as part ofPEP 479.Regardless, for any Python it’s recommended that your generator
live at least as long as the iterator feeding it.🎧 ⚠Iter.sorted(self, key=None, reverse=False)->"Iter[T]"Simple wrapper for thesortedbuiltin.Calling this will read the entire stream before producing
results.>>>Iter('bac').sorted().collect()['a','b','c']>>>Iter('bac').sorted(reverse=True).collect()['c','b','a']>>>Iter('bac').zip([2,1,0]).sorted(key=lambdatup:tup[1]).collect()[('c',0),('a',1),('b',2)]🎧 ⚠Iter.reversed(self)->"Iter[T]"Simple wrapper for thereversedbuiltin.Calling this will read the entire stream before producing
results.>>>Iter(range(4)).reversed().collect()[3,2,1,0]Experiments and Provisional Ideas🛠class IterDict(UserDict)The idea here was to make a custom dict where several of
the standard dict methods returnIterinstances, which can then
be chained. I’m not sure if this will be kept yet.IterDict.keys(self)->"Iter"IterDict.values(self)->"Iter"IterDict.items(self)->"Iter"IterDict.update(self, *args, **kwargs)->"IterDict"insert_separator(iterable: Iterable[Any], glue: Any)->"Iterable[Any]"Similar functionality can be obtained with, e.g.,
interleave, as in>>>result=Iter('caleb').interleave(Iter.repeat('x')).collect()>>>result==list('cxaxlxexbx')TrueBut you’ll see a trailing “x” there, which join avoids. join
makes sure to only add the glue separator if another element
has arrived.It can handle strings without any special considerations, but it doesn’t
do any special handling for bytes and bytearrays. For that, rather
look atconcat().Related projectsIt turns out the idea of chaining iterators is not new. There are many
libraries that offer similar features:My fork of a now-missing library:chained-iterable.https://github.com/olirice/flupyhttps://github.com/ddstte/chiterhttps://github.com/neverendingqs/pyiterablehttps://github.com/alliefitter/iterable_collectionshttps://github.com/halprin/iterator-chainhttps://github.com/jagill/python-chainzhttps://github.com/ZianVW/IterPipehttps://github.com/Evelyn-H/iterchainhttps://github.com/EntilZha/PyFunctionalhttps://github.com/dwt/fluentSomewhat related:https://github.com/jreese/aioitertoolsDev InstructionsSetup$python-mvenvvenv$sourcevenv/bin/activate(venv)$pipinstall-e.[dev,test]Testing(venv)$pytest--covDocumentationTo regenerate the documentation, fileREADME.rst:(venv)$pythonregenerate_readme.py-mexcitertools.py>README.rstReleasingTo do a release, we’re usingbumpymcbumpface.
Make sure that is set up correctly according to its own documentation. I
like to usepipxto install and
manage these kinds of tools.$bumpymcbumpface--push-git--push-pypiWork is a necessary evil to be avoided.Mark Twain