Package: roys-pyxtension
pyxtensionpyxtensionis a pure Python MIT-licensed library that includes Scala-like streams (usingFluent Interface pattern), Json with attribute access syntax, and other common-use stuff.Note:Drop support & maintenance for Python 2.x version, due toPy2 death.Although Py2 version will remain in the repository, I won't update PyPi package, so the last Py2 version of thepyxtensionavailable atPyPiwill remain1.12.7Starting with1.13.0I've migrated the packaging & distributing method toWheel.Installationpip install roys-pyxtensionor from Github:git clone https://github.com/roymanigley/roys-pyxtension.git
cd roy-pyxtension
python setup.py installorgit submodule add https://github.com/roymanigley/roys-pyxtension.gitModules overviewJson.pyJsonAdictsubclass to represent a Json object. You should be able to use this
absolutely anywhere you can use adict. While this is probably the class you
want to use, there are a few caveats that follow from this being adictunder
the hood.Never again will you have to write code like this:body={'query':{'filtered':{'query':{'match':{'description':'addictive'}},'filter':{'term':{'created_by':'ASU'}}}}}From now on, you may simply write the following three lines:body=Json()body.query.filtered.query.match.description='addictive'body.query.filtered.filter.term.created_by='ASU'streams.pystreamstreamsubclassescollections.Iterable. It's the same Python iterable, but with more added methods, suitable for multithreading and multiprocess processings.
Used to create stream processing pipelines, similar to those used inScalaandMapReduceprogramming model.
Those who usedApache SparkRDDfunctions will find this model of processing very easy to use.streamsNever again will you have to write code like this:>lst=xrange(1,6)>reduce(lambdax,y:x*y,map(lambda_:_*_,filter(lambda_:_%2==0,lst)))64From now on, you may simply write the following lines:>the_stream=stream(xrange(1,6))>the_stream.\filter(lambda_:_%2==0).\map(lambda_:_*_).\reduce(lambdax,y:x*y)64A Word CountMap-Reducenaive example using multiprocessing mapcorpus=["MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster.","At Google, MapReduce was used to completely regenerate Google's index of the World Wide Web","Conceptually similar approaches have been very well known since 1995 with the Message Passing Interface standard having reduce and scatter operations."]defreduceMaps(m1,m2):fork,vinm2.iteritems():m1[k]=m1.get(k,0)+vreturnm1word_counts=stream(corpus).\mpmap(lambdaline:stream(line.lower().split(' ')).countByValue()).\reduce(reduceMaps)Basic methodson_next(f)The passed function does return None but returns a streamfor_each(f)The passed function will proceed all elements passed in the by the stream but returns Nonemap(f)Identic with builtinmapbut returns a streammpmap(self, f: Callable[[_K], _V], poolSize: int = cpu_count(), bufferSize: Optional[int] = None)Parallel ordered map usingmultiprocessing.Pool.imap().It can replace themapwhen need to split computations to multiple cores, and order of results matters.It spawns at mostpoolSizeprocesses and applies theffunction.It won't take more thanbufferSizeelements from the input unless it was already required by output, so you can use it withtakeWhileon infinite streams and not be afraid that it will continue work in background.The elements in the result stream appears in the same order they appear in the initial iterable.:type f: (T) -> V
:rtype: `stream`mpfastmap(self, f: Callable[[_K], _V], poolSize: int = cpu_count(), bufferSize: Optional[int] = None)Parallel ordered map usingmultiprocessing.Pool.imap_unordered().It can replace themapwhen the ordered of results doesn't matter.It spawns at mostpoolSizeprocesses and applies theffunction.It won't take more thanbufferSizeelements from the input unless it was already required by output, so you can use it withtakeWhileon infinite streams and not be afraid that it will continue work in background.The elements in the result stream appears in the unpredicted order.:type f: (T) -> V
:rtype: `stream`fastmap(self, f: Callable[[_K], _V], poolSize: int = cpu_count(), bufferSize: Optional[int] = None)Parallel unordered map using multithreaded pool.
It can replace themapwhen the ordered of results doesn't matter.It spawns at mostpoolSizethreads and applies theffunction.The elements in the result stream appears in theunpredictedorder.It won't take more thanbufferSizeelements from the input unless it was already required by output, so you can use it withtakeWhileon infinite streams and not be afraid that it will continue work in background.Because of CPythonGILit's most usefull for I/O or CPU intensive consuming native functions, or on Jython or IronPython interpreters.:type f: (T) -> V:rtype:streammtmap(self, f: Callable[[_K], _V], poolSize: int = cpu_count(), bufferSize: Optional[int] = None)Parallel ordered map using multithreaded pool.
It can replace themapand the order of output stream will be the same as of the input.It spawns at mostpoolSizethreads and applies theffunction.The elements in the result stream appears in thepredictedorder.It won't take more thanbufferSizeelements from the input unless it was already required by output, so you can use it withtakeWhileon infinite streams and not be afraid that it will continue work in background.Because of CPythonGILit's most usefull for I/O or CPU intensive consuming native functions, or on Jython or IronPython interpreters.:type f: (T) -> V:rtype:streamflatMap(predicate=_IDENTITY_FUNC):param predicate: is a function that will receive elements of self collection and return an iterableBy default predicate is an identity function:type predicate: (V)-> collections.Iterable[T]:return: will return stream of objects of the same type of elements from the stream returned by predicate()Example:stream([[1,2],[3,4],[4,5]]).flatMap().toList()==[1,2,3,4,4,5]filter(predicate)identic with builtin filter, but returns streamreversed()returns reversed streamexists(predicate)Tests whether a predicate holds for some of the elements of this sequence.:rtype: boolExample:stream([1,2,3]).exists(0)->Falsestream([1,2,3]).exists(1)->TruekeyBy(keyfunc = _IDENTITY_FUNC)Transforms stream of values to a stream of tuples (key, value):param keyfunc: function to map values to keys:type keyfunc: (V) -> T:return: stream of Key, Value pairs:rtype: stream[( T, V )]Example:stream([1,2,3,4]).keyBy(lambda_:_%2)->[(1,1),(0,2),(1,3),(0,4)]groupBy()groupBy([keyfunc]) -> Make an iterator that returns consecutive keys and groups from the iterable.The iterable needs not to be sorted on the same key function, but the keyfunction need to return hasable objects.:param keyfunc: [Optional] The key is a function computing a key value for each element.:type keyfunc: (T) -> (V):return: (key, sub-iterator) grouped by each value of key(value).:rtype: stream[ ( V, slist[T] ) ]Example:stream([1,2,3,4]).groupBy(lambda_:_%2)->[(0,[2,4]),(1,[1,3])]countByValue()Returns a collections.Counter of valuesExamplestream(['a','b','a','b','c','d']).countByValue()=={'a':2,'b':2,'c':1,'d':1}distinct()Returns stream of distinct values. Values must be hashable.stream(['a','b','a','b','c','d']).distinct()=={'a','b','c','d'}reduce(f, init=None)same arguments with builtin reduce() functiontoSet()returns sset() instancetoList()returns slist() instancetoMap()returns sdict() instancesorted(key=None, cmp=None, reverse=False)same arguments with builtin sorted()size()returns length of stream. Use carefully on infinite streams.join(f)Returns a string joined by f. Proivides same functionality as str.join() builtin method.if f is basestring, uses it to join the stream, else f should be a callable that returns a string to be used for joinmkString(f)identic with join(f)take(n)returns first n elements from streamhead()returns first element from streamzip()the same behavior with itertools.izip()unique(predicate=_IDENTITY_FUNC)Returns a stream of unique (according to predicate) elements appearing in the same order as in original stream

The items returned by predicate should be hashable and comparable.Statistics related methodsentropy()calculates the Shannon entropy of the values from streampstddev()Calculates the population standard deviation.mean()returns the arithmetical mean of the valuessum()returns the sum of elements from streammin(key=_IDENTITY_FUNC)same functionality with builtin min() funcionmin_default(default, key=_IDENTITY_FUNC)same functionality with min() but returns :default: when called on empty streamsmax()same functionality with builtin max()maxes(key=_IDENTITY_FUNC)returns a stream of max values from streammins(key=_IDENTITY_FUNC)returns a stream of min values from streamOther classesslistInheritsstreams.streamand built-inlistclasses, and keeps in memory a list allowing faster index accessssetInheritsstreams.streamand built-insetclasses, and keeps in memory the whole set of valuessdictInheritsstreams.streamand built-indict, and keeps in memory the dict object.defaultstreamdictInheritsstreams.sdictand adds functionality  ofcollections.defaultdictfrom stdlibJsonJsonis a module that provides mapping objects that allow their elements to be accessed both as keys and as attributes:>frompyxtension.JsonimportJson>a=Json({'foo':'bar'})>a.foo'bar'>a['foo']'bar'Attribute access makes it easy to create convenient, hierarchical settings objects:withopen('settings.yaml')asfileobj:settings=Json(yaml.safe_load(fileobj))cursor=connect(**settings.db.credentials).cursor()cursor.execute("SELECT column FROM table;")Basic UsageJson comes with two different classes,Json, andJsonList.
Json is fairly similar to nativedictas it extends it an is a mutable mapping that allow creating, accessing, and deleting key-value pairs as attributes.JsonListis similar to nativelistas it extends it and offers a way to transform thedictobjects from inside also inJsoninstances.ConstructionDirectly from a JSON string>Json('{"key1": "val1", "lst1": [1,2] }'){u'key1':u'val1',u'lst1':[1,2]}Fromtuples:>Json(('key1','val1'),('lst1',[1,2])){'key1':'val1','lst1':[1,2]}# keep in mind that you should provide at least two tuples with key-value pairsAs a built-indict>Json([('key1','val1'),('lst1',[1,2])]){'key1':'val1','lst1':[1,2]}Json({'key1':'val1','lst1':[1,2]}){'key1':'val1','lst1':[1,2]}Convert to adict>json=Json({'key1':'val1','lst1':[1,2]})>json.toOrig(){'key1':'val1','lst1':[1,2]}Valid NamesAny key can be used as an attribute as long as:The key represents a valid attribute (i.e., it is a string comprised only of
alphanumeric characters and underscores that doesn't start with a number)The key does not shadow a class attribute (e.g., get).Attributes vs. KeysThere is a minor difference between accessing a value as an attribute vs.
accessing it as a key, is that when a dict is accessed as an attribute, it will
automatically be converted to aJsonobject. This allows you to recursively
access keys::>attr=Json({'foo':{'bar':'baz'}})>attr.foo.bar'baz'Relatedly, by default, sequence types that aren'tbytes,str, orunicode(e.g.,lists,tuples) will automatically be converted totuples, with any
mappings converted toJson:>attr=Json({'foo':[{'bar':'baz'},{'bar':'qux'}]})>forsub_attrinattr.foo:>print(sub_attr.bar)'baz''qux'To get this recursive functionality for keys that cannot be used as attributes,
you can replicate the behavior by using dict syntax onJsonobject::>json=Json({1:{'two':3}})>json[1].two3JsonListusage examples:> json = Json('{"lst":[1,2,3]}')
> type(json.lst)
<class 'pyxtension.Json.JsonList'>

> json = Json('{"1":[1,2]}')
> json["1"][1]
2Assignment as keys will still work::>json=Json({'foo':{'bar':'baz'}})>json['foo']['bar']='baz'>json.foo{'bar':'baz'}frozendictfrozendictis a simple immutable dictionary, where you can't change the internal variables of the class, and they are all immutable objects. Reinvoking__init__also doesn't alter the object.The API is the same asdict, without methods that can change the immutability.frozendictis also hashable and can be used as keys for other dictionaries, of course with the condition that all values of the frozendict are also hashable.>>>frompyxtensionimportfrozendict>>>fd=frozendict({"A":"B","C":"D"})>>>print(fd){'A':'B','C':'D'}>>>fd["A"]="C"TypeError:objectisimmutable>>>hash(fd)-5063792767678978828Licensepyxtension is released under a GNU Public license.
The idea forJsonmodule was inspired fromaddictandAttrDict,
but it has a better performance with lower memory consumption.AlternativesThere are other libraries that support Fluent Interface streams as alternatives to Pyxtension, but being much more poor in features for streaming:https://pypi.org/project/lazy-streams/https://pypi.org/project/pystreams/https://pypi.org/project/fluentpy/https://github.com/matthagy/scalapshttps://pypi.org/project/infixpy/mentionedherehttps://github.com/sspipe/sspipeand something quite different from Fluent patterm, that makes kind of Piping:https://github.com/sspipe/sspipeandhttps://github.com/JulienPalard/Pipe