Package: xtuples
xtuplesTable of ContentsInstallationOverviewPerformanceLicenseInstallationpip install xtuplesOverviewimporttypingimportxtuplesasxtxtuples (xt) provides:#1: xt.iTuple: a bunch of (primarily) itertools / functools boilerplate, wrapped into a tuple subclass.#2: xt.nTuple.decorate: a decorator to inject .pipe(), .partial(), and a dict of any user defined methods, into a NamedTuple.iTupleThe idea with #1 is to facilitate a functional style of programming, utilising method chaining to mimic the function pipelines seen in languages like f#.For instance, a naive way to get all the squared primes under x, might be:defprimes_under(x):return(xt.iTuple.range(x).filter(lambdav:v>1).fold(lambdaprimes,v:(primes.append(v)ifnotprimes.any(lambdaprime:v%prime==0)elseprimes),initial=xt.iTuple()))sq_primes_under_10=primes_under(10).map(lambdav:v**2)iTuple has type annotations for a reasonable range of overloads of common methods, such that mypy should, without requiring explicit annotations, be able to track types through methods like zip(), map(), filter(), and so on.See ./tests/test_mypy.py for particular examples.nTupleThe idea with #2 is to ease method re-use between, and interface definitions on, NamedTuples, where rather than inheriting methods, we inject them in (on top of a type signature stub).For instance, we can share the function f between NamedTuples A and B, as so:classHas_X(typing.Protocol):x:intdeff(self:Has_X)->int:returnself.x+1@xt.nTuple.decorate(f=f)classA(typing.NamedTuple):x:inty:floatdeff(self:A)->int:...# type: ignore[empty-body]@xt.nTuple.decorate(f=f)classB(typing.NamedTuple):x:inty:floatdeff(self:B)->int:...# type: ignore[empty-body]Annoyingly, we have to set mypy to ignore the empty body, but we still get type checking on the implementation (at f).JAXWorth highlighting is the compatibility this promotes withJAX, an auto-grad / machine learning framework from the Google Brain / Deepmind folks.Because both iTuple and nTuple are kinds of tuple, JAX can take derivatives of / through without any further work (TODO: have a jax.register function on iTuple, so we don't need to call .pipe(tuple)).Furthermore, because both iTuple and nTuple are immutable, generally very little refactoring is required for an xtuples code-base to be compliant with the (somewhat opinionated) functional purity requirements of the JAX jit compiler.xt.fxtuples also exposes:#3: xt.f: a module of sister methods to those in xt.iTuple, designed to provide / be used as arguments for their bound iTuple cousins.For instance, instead of:cum_sq_primes_under_10=(iTuple.range(10).map(primes_under).map(lambdait:it.map(lambdav:v**2)))We can write:cum_sq_primes_under_10=(iTuple.range(10).map(primes_under).map(xt.f.map(lambdav:v**2)))Ie. it simply saves us from writing out quite so many lambdas.PerformancePerformance using xtuples is generally not worse than a canonical equivalent implementation, and can sometimes be significantly better.iTupleFor instance, as iTuple is simply a subclass of the built-in tuple, it has very similar performance characteristics.CreationCreation is slightly slower than for an equivalent length list:%timeitxtuples.iTuple.range(10**2)%timeitlist(range(10**2))1.56 µs ± 9.4 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
588 ns ± 11.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)%timeitxtuples.iTuple.range(10**3)%timeitlist(range(10**3))11.9 µs ± 128 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
7.97 µs ± 60.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)%timeitxtuples.iTuple.range(10**4)%timeitlist(range(10**4))132 µs ± 1.51 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
89.6 µs ± 817 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)%timeitxtuples.iTuple.range(10**6)%timeitlist(range(10**6))40.9 ms ± 1.12 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
24.5 ms ± 424 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)MemoryWhereas memory usage (comparable for small sizes), gets increasingly more efficient with size:memory={}foriinrange(5):memory[i]=dict(iTuple=asizeof(xtuples.iTuple.range(10**i)),list=asizeof(list(range(10**i))),)memory{0: {'iTuple': 160, 'list': 88},
 1: {'iTuple': 232, 'list': 448},
 2: {'iTuple': 952, 'list': 4048},
 3: {'iTuple': 8152, 'list': 40048},
 4: {'iTuple': 80152, 'list': 400048}}ex_iTuple=xtuples.iTuple.range(100)ex_list=list(range(100))ex_range=range(100)Iteration & IndexingIteration is very similar:%timeitforxinex_iTuple:pass%timeitforxinex_list:pass764 ns ± 4.57 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
721 ns ± 24.1 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)And whilst elementwise indexing is clearly slower:%timeitforiinrange(100):ex_iTuple[i]%timeitforiinrange(100):ex_list[i]18.6 µs ± 240 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
2.84 µs ± 75.3 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)And so is slice indexing:%timeitex_iTuple[10:20]%timeitex_list[10:20]667 ns ± 10.3 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
93.9 ns ± 0.998 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)It is worth noting that per element indexing is not all that common using xtuples (as the canonical implementation is much more likely to use .map() and co).Function applicationElementwise function application with .map() ismuchfaster than the equivalent loop or list comprehension:add_2=functools.partial(operator.add,2)deff_loop_map(f,l):res=[]forvinl:res.append(f(v))returnres%timeitex_iTuple.map(add_2)%timeitf_loop_map(add_2,ex_list)%timeit[add_2(x)forxinex_list]%timeitlist(map(add_2,ex_list))5.57 µs ± 104 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
8.33 s ± 248 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
5.96 s ± 181 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
5.12 s ± 609 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)As is elementwise filtering:deff_loop_filter(f):res=[]foriinex_list:iff(i):res.append(i)returnresf=lambdax:x%2==0%timeitex_iTuple.filter(f)%timeitf_loop_filter(f)%timeit[vforvinex_listiff(v)]15.2 µs ± 41.6 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
6.68 s ± 139 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
7.13 s ± 113 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)And, so are both fold and cumulative fold:deff_loop_fold():acc=0foriinex_list:acc=operator.add(acc,i)returnacc%timeitex_iTuple.fold(operator.add)%timeitf_loop_fold()3.11 µs ± 48.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
4.87 s ± 89.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)So, as mentioned below, the answer to the poor .append() performance is just to use .foldcum() instead:deff_loop_foldcum():res=[]acc=0foriinex_list:acc=operator.add(acc,i)res.append(acc)returnres%timeitex_iTuple.foldcum(operator.add)%timeitf_loop_foldcum()5.91 µs ± 19 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
15.2 s ± 1.67 s per loop (mean ± std. dev. of 7 runs, 1 loop each)AppendAppending ismuchslower, which is clearly to some extent a 'gotcha'.%timeitex_iTuple.append(1)%timeitex_list.append(1)1.56 µs ± 22.9 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
69.3 ns ± 6.18 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)Having said that, the canonical xtuples implementation is much more likely to use .map() .foldcum() or similar than .append().And, as we've already seen, .map() and .foldcum() aremuchfaster than the for-loop & append() implementations (so just do that instead - I personally also find it much more readable).Prepend / ExtendPrepending to the tuple ismuchfaster than with the list, though the relevant comparison is probably a deque (given that list is not at all optimised for left-append):%timeitex_iTuple.prepend(1)%timeitex_list.insert(0,1)1.44 µs ± 11 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
109 ms ± 10.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)Extend is somewhat slower (but is nowhere near as bad as append):%timeitxtuples.iTuple.range(100).extend([1])%timeitxtuples.iTuple.range(100).extend(list(range(100)))%timeitlist(range(100)).extend([1])%timeitlist(range(100)).extend(list(range(100)))3.93 µs ± 32.3 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
5.58 µs ± 43.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
769 ns ± 8.37 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
1.48 µs ± 19 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)And flatten ismuchfaster:ex_iTuple_nested=ex_iTuple.map(lambdav:[v])ex_list_nested=[[v]forvinex_list]deff_loop_flatten(l):forvinl:yield fromv%timeitex_iTuple_nested.flatten()%timeitlist(f_loop_flatten(ex_list_nested))%timeitlist(itertools.chain(*ex_list_nested))5.38 µs ± 81.1 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
29.5 s ± 1.14 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
1min 19s ± 14.9 s per loop (mean ± std. dev. of 7 runs, 1 loop each)SummaryOverall, iTuple performance and memory usage is comparable - if not better - than a raw list.The one clear weakness is .append() - however, the canonical xtuples implementation would use .map() .foldcum() etc. instead (which are actuallyfasterthan the equivalent .append() implementation).Named TuplenTuple doesnot(in comparison to iTuple) define a base class for us to sub-class.Rather, it provides a decorator - nTuple.decorate - that adds .pipe() .partial() and a dict of user defined methods to a given NamedTuple.As such, performance is essentially just that of built-in NamedTuples (ie. generally very strong).@xtuples.nTuple.decorate(update_x=lambdaself,x:self._replace(x=x),update_s=lambdaself,s:self._replace(s=s),)classExample(typing.NamedTuple):x:ints:strclassExample_Cls:x:ints:strdef__init__(self,x,s):self.x=xself.s=s@dataclasses.dataclass(frozen=True,eq=True)classExample_DC:x:ints:strex_nTuple=Example(1,"a")ex_dict=dict(x=1,s="a")ex_cls=Example_Cls(1,"a")ex_dc=Example_DC(1,"a")NOTE, re-registering: ExampleFor instance, NamedTuples are significantly more memory efficient than any of the possible alternatives:dict(nTuple=asizeof(ex_nTuple),dict=asizeof(ex_dict),cls=asizeof(ex_cls),dataclass=asizeof(ex_dc),){'nTuple': 144, 'dict': 432, 'cls': 352, 'dataclass': 352}%timeitExample(1,"a")%timeitdict(x=1,s="a")%timeitExample_Cls(1,"a")%timeitExample_DC(1,"a")257 ns ± 8.92 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
111 ns ± 1.53 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
209 ns ± 2.58 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
443 ns ± 3.05 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)Whilst providing comparable (if not slightly faster) field access times:%timeitex_nTuple.x%timeitex_nTuple[0]%timeitex_dict["x"]%timeitex_cls.x%timeitex_dc.x27.7 ns ± 0.303 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
25.1 ns ± 0.0815 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
29.7 ns ± 0.181 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
30.3 ns ± 0.492 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
34.9 ns ± 0.125 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)Writes are, however, slower - the price we pay for immutability (but are still notably faster than the frozen dataclass equivalent):%timeitex_dict["x"]=1%timeitex_cls.x=1%timeitex_nTuple._replace(x=1)&timeitex_nTuple.update_x(x)%timeitex_nTuple.update(x=1)%timeitdataclasses.replace(ex_dc,x=1)32.4 ns ± 0.467 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
43.8 ns ± 0.524 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)
735 ns ± 5.93 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
1.03 µs ± 3.35 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
1.28 µs ± 4.37 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)Like frozen dataclasses, NamedTuples are conveniently hashable (in comparison to dicts, for instance, which aren't), and do so based on value (versus standard classes which use object ids by default):dict(nTuple=hash(ex_nTuple)==hash(Example(1,"a")),cls=hash(ex_cls)==hash(Example_Cls(1,"a")),dataclass=hash(ex_dc)==hash(Example_DC(1,"a")),){'nTuple': True, 'cls': False, 'dataclass': True}This is particularly useful in combination with iTuple, which is also hashable (making combinations of the two recursively hashable):@xtuples.nTuple.decorateclassExample_Nested(typing.NamedTuple):x:ints:strit:xtuples.iTuplehash(Example_Nested(1,"s",xtuples.iTuple()))==hash(Example_Nested(1,"s",xtuples.iTuple()))TrueFinally, sorting is both provided by default (again, in comparison to dicts and classes), and works as one would expect (ie. by the first field, then the second field, etc.):xtuples.iTuple([Example(2,"a"),Example(1,"b"),Example(1,"a"),]).sort()iTuple(Example(x=1, s='a'), Example(x=1, s='b'), Example(x=2, s='a'))Licensextuplesis distributed under the terms of theMITlicense.