Package: pyfunctional
PyFunctionalFeaturesPyFunctionalmakes creating data pipelines easy by using chained functional operators. Here are a
few examples of what it can do:Chained operators:seq(1, 2, 3).map(lambda x: x * 2).reduce(lambda x, y: x + y)Expressive and feature complete APIRead and writetext,csv,json,jsonl,sqlite,gzip,bz2, andlzma/xzfilesParallelize "embarrassingly parallel" operations likemapeasilyComplete documentation, rigorous unit test suite, 100% test coverage, and CI which provide
robustnessPyFunctional's API takes inspiration from Scala collections, Apache Spark RDDs, and Microsoft
LINQ.Table of ContentsInstallationExamplesSimple ExampleAggregates and JoinsReading and Writing SQLite3Data Interchange with PandasWriting to FilesParallel ExecutionGithub Shortform DocumentationStreams, Transformations, and ActionsStreams APITransformations and Actions APIsLazy ExecutionContributing and Bug FixesChangelogInstallationPyFunctionalis available onpypiand can be
installed by running:# Install from command line$pipinstallpyfunctionalThen in python run:from functional import seqExamplesPyFunctionalis useful for many tasks, and can natively open several common file types. Here
are a few examples of what you can do.Simple Examplefromfunctionalimportseqseq(1,2,3,4)\.map(lambdax:x*2)\.filter(lambdax:x>4)\.reduce(lambdax,y:x+y)# 14# or if you don't like backslash continuation(seq(1,2,3,4).map(lambdax:x*2).filter(lambdax:x>4).reduce(lambdax,y:x+y))# 14Streams, Transformations and ActionsPyFunctionalhas three types of functions:Streams: read data for use by the collections API.Transformations: transform data from streams with functions such asmap,flat_map, andfilterActions: These cause a series of transformations to evaluate to a concrete value.to_list,reduce, andto_dictare examples of actions.In the expressionseq(1, 2, 3).map(lambda x: x * 2).reduce(lambda x, y: x + y),seqis the
stream,mapis the transformation, andreduceis the action.Filtering a list of account transactionsfromfunctionalimportseqfromcollectionsimportnamedtupleTransaction=namedtuple('Transaction','reason amount')transactions=[Transaction('github',7),Transaction('food',10),Transaction('coffee',5),Transaction('digitalocean',5),Transaction('food',5),Transaction('riotgames',25),Transaction('food',10),Transaction('amazon',200),Transaction('paycheck',-1000)]# Using the Scala/Spark inspired APIsfood_cost=seq(transactions)\.filter(lambdax:x.reason=='food')\.map(lambdax:x.amount).sum()# Using the LINQ inspired APIsfood_cost=seq(transactions)\.where(lambdax:x.reason=='food')\.select(lambdax:x.amount).sum()# Using PyFunctional with fnfromfnimport_food_cost=seq(transactions).filter(_.reason=='food').map(_.amount).sum()Aggregates and JoinsThe account transactions example could be done easily in pure python using list comprehensions. To
show some of the thingsPyFunctionalexcels at, take a look at a couple of word count examples.words='I dont want to believe I want to know'.split(' ')seq(words).map(lambdaword:(word,1)).reduce_by_key(lambdax,y:x+y)# [('dont', 1), ('I', 2), ('to', 2), ('know', 1), ('want', 2), ('believe', 1)]In the next example we have chat logs formatted injson lines (jsonl)which
contain messages and metadata. A typical jsonl file will have one valid json on each line of a file.
Below are a few lines out ofexamples/chat_logs.jsonl.{"message":"hello anyone there?","date":"10/09","user":"bob"}{"message":"need some help with a program","date":"10/09","user":"bob"}{"message":"sure thing. What do you need help with?","date":"10/09","user":"dave"}fromoperatorimportaddimportremessages=seq.jsonl('examples/chat_logs.jsonl')# Split words on space and normalize before doing word countdefextract_words(message):returnre.sub('[^0-9a-z ]+','',message.lower()).split(' ')word_counts=messages\.map(lambdalog:extract_words(log['message']))\.flatten().map(lambdaword:(word,1))\.reduce_by_key(add).order_by(lambdax:x[1])Next, lets continue that example but introduce a json database of users fromexamples/users.json.
In the previous example we showed howPyFunctionalcan do word counts, in the next example lets
show howPyFunctionalcan join different data sources.# First read the json fileusers=seq.json('examples/users.json')#[('sarah',{'date_created':'08/08','news_email':True,'email':'sarah@gmail.com'}),...]email_domains=users.map(lambdau:u[1]['email'].split('@')[1]).distinct()# ['yahoo.com', 'python.org', 'gmail.com']# Join users with their messagesmessage_tuples=messages.group_by(lambdam:m['user'])data=users.inner_join(message_tuples)# [('sarah',#    (#      {'date_created':'08/08','news_email':True,'email':'sarah@gmail.com'},#      [{'date':'10/10','message':'what is a...','user':'sarah'}...]#    )#  ),...]# From here you can imagine doing more complex analysisCSV, Aggregate Functions, and Set functionsInexamples/camping_purchases.csvthere are a list of camping purchases. Lets do some cost
analysis and compare it the required camping gear list stored inexamples/gear_list.txt.purchases=seq.csv('examples/camping_purchases.csv')total_cost=purchases.select(lambdarow:int(row[2])).sum()# 1275most_expensive_item=purchases.max_by(lambdarow:int(row[2]))# ['4', 'sleeping bag', ' 350']purchased_list=purchases.select(lambdarow:row[1])gear_list=seq.open('examples/gear_list.txt').map(lambdarow:row.strip())missing_gear=gear_list.difference(purchased_list)# ['water bottle','gas','toilet paper','lighter','spoons','sleeping pad',...]In addition to the aggregate functions shown above (sumandmax_by) there are many more.
Similarly, there are several more set like functions in addition todifference.Reading/Writing SQLite3PyFunctionalcan read and write to SQLite3 database files. In the example below, users are read
fromexamples/users.dbwhich stores them as rows with columnsid:Intandname:String.db_path='examples/users.db'users=seq.sqlite3(db_path,'select * from user').to_list()# [(1, 'Tom'), (2, 'Jack'), (3, 'Jane'), (4, 'Stephan')]]sorted_users=seq.sqlite3(db_path,'select * from user order by name').to_list()# [(2, 'Jack'), (3, 'Jane'), (4, 'Stephan'), (1, 'Tom')]Writing to a SQLite3 database is similarly easyimportsqlite3fromcollectionsimportnamedtuplewithsqlite3.connect(':memory:')asconn:conn.execute('CREATE TABLE user (id INT, name TEXT)')conn.commit()User=namedtuple('User','id name')# Write using a specific queryseq([(1,'pedro'),(2,'fritz')]).to_sqlite3(conn,'INSERT INTO user (id, name) VALUES (?, ?)')# Write by inserting values positionally from a tuple/list into named tableseq([(3,'sam'),(4,'stan')]).to_sqlite3(conn,'user')# Write by inferring schema from namedtupleseq([User(name='tom',id=5),User(name='keiga',id=6)]).to_sqlite3(conn,'user')# Write by inferring schema from dictseq([dict(name='david',id=7),dict(name='jordan',id=8)]).to_sqlite3(conn,'user')# Read everything back to make sure it wrote correctlyprint(list(conn.execute('SELECT * FROM user')))# [(1, 'pedro'), (2, 'fritz'), (3, 'sam'), (4, 'stan'), (5, 'tom'), (6, 'keiga'), (7, 'david'), (8, 'jordan')]Writing to filesJust asPyFunctionalcan read fromcsv,json,jsonl,sqlite3, and text files, it can
also write them. For complete API documentation see the collections API table or the official docs.Compressed FilesPyFunctionalwill auto-detect files compressed withgzip,lzma/xz, andbz2. This is done
by examining the first several bytes of the file to determine if it is compressed so therefore
requires no code changes to work.To write compressed files, everyto_function has a parametercompressionwhich can be set to
the defaultNonefor no compression,gziporgzfor gzip compression,lzmaorxzfor lzma
compression, andbz2for bz2 compression.Parallel ExecutionThe only change required to enable parallelism is to importfrom functional import pseqinstead offrom functional import seqand usepseqwhere you would useseq. The following
operations are run in parallel with more to be implemented in a future release:map/selectfilter/filter_not/whereflat_mapParallelization uses pythonmultiprocessingand squashes chains of embarrassingly parallel
operations to reduce overhead costs. For example, a sequence of maps and filters would be executed
all at once rather than in multiple loops usingmultiprocessingDocumentationShortform documentation is below and full documentation is atdocs.pyfunctional.org.Streams APIAll ofPyFunctionalstreams can be accessed through theseqobject. The primary way to create
a stream is by callingseqwith an iterable. Theseqcallable is smart and is able to accept
multiple types of parameters as shown in the examples below.# Passing a listseq([1,1,2,3]).to_set()# [1, 2, 3]# Passing direct argumentsseq(1,1,2,3).map(lambdax:x).to_list()# [1, 1, 2, 3]# Passing a single valueseq(1).map(lambdax:-x).to_list()# [-1]seqalso provides entry to other streams as attribute functions as shown below.# number rangeseq.range(10)# text fileseq.open('filepath')# json fileseq.json('filepath')# jsonl fileseq.jsonl('filepath')# csv fileseq.csv('filepath')seq.csv_dict_reader('filepath')# sqlite3 db and sql queryseq.sqlite3('filepath','select * from data')For more information on the parameters that these functions can take, reference thestreams documentationTransformations and Actions APIsBelow is the complete list of functions which can be called on a stream object fromseq. For
complete documentation referencetransformation and actions API.FunctionDescriptionTypemap(func)/select(func)Mapsfunconto elements of sequencetransformationstarmap(func)/smap(func)Applyfuncto sequence withitertools.starmaptransformationfilter(func)/where(func)Filters elements of sequence to only those wherefunc(element)isTruetransformationfilter_not(func)Filters elements of sequence to only those wherefunc(element)isFalsetransformationflatten()Flattens sequence of lists to a single sequencetransformationflat_map(func)funcmust return an iterable. Mapsfuncto each element, then merges the result to one flat sequencetransformationgroup_by(func)Groups sequence into(key, value)pairs wherekey=func(element)andvalueis from the original sequencetransformationgroup_by_key()Groups sequence of(key, value)pairs bykeytransformationreduce_by_key(func)Reduces list of(key, value)pairs usingfunctransformationcount_by_key()Counts occurrences of eachkeyin list of(key, value)pairstransformationcount_by_value()Counts occurrence of each value in a listtransformationunion(other)Union of unique elements in sequence andothertransformationintersection(other)Intersection of unique elements in sequence andothertransformationdifference(other)New sequence with unique elements present in sequence but not inothertransformationsymmetric_difference(other)New sequence with unique elements present in sequence orother, but not bothtransformationdistinct()Returns distinct elements of sequence. Elements must be hashabletransformationdistinct_by(func)Returns distinct elements of sequence usingfuncas a keytransformationdrop(n)Drop the firstnelements of the sequencetransformationdrop_right(n)Drop the lastnelements of the sequencetransformationdrop_while(func)Drop elements whilefuncevaluates toTrue, then returns the resttransformationtake(n)Returns sequence of firstnelementstransformationtake_while(func)Take elements whilefuncevaluates toTrue, then drops the resttransformationinit()Returns sequence without the last elementtransformationtail()Returns sequence without the first elementtransformationinits()Returns consecutive inits of sequencetransformationtails()Returns consecutive tails of sequencetransformationzip(other)Zips the sequence withothertransformationzip_with_index(start=0)Zips the sequence with the index starting atstarton the right sidetransformationenumerate(start=0)Zips the sequence with the index starting atstarton the left sidetransformationcartesian(*iterables, repeat=1)Returns cartesian product from itertools.producttransformationinner_join(other)Returns inner join of sequence with other. Must be a sequence of(key, value)pairstransformationouter_join(other)Returns outer join of sequence with other. Must be a sequence of(key, value)pairstransformationleft_join(other)Returns left join of sequence with other. Must be a sequence of(key, value)pairstransformationright_join(other)Returns right join of sequence with other. Must be a sequence of(key, value)pairstransformationjoin(other, join_type='inner')Returns join of sequence with other as specified byjoin_type. Must be a sequence of(key, value)pairstransformationpartition(func)Partitions the sequence into elements which satisfyfunc(element)and those that don'ttransformationgrouped(size)Partitions the elements into groups of sizesizetransformationsorted(key=None, reverse=False)/order_by(func)Returns elements sorted according to pythonsortedtransformationreverse()Returns the reversed sequencetransformationslice(start, until)Sequence starting atstartand including elements up tountiltransformationhead()/first()Returns first element in sequenceactionhead_option()Returns first element in sequence orNoneif its emptyactionlast()Returns last element in sequenceactionlast_option()Returns last element in sequence orNoneif its emptyactionlen()/size()Returns length of sequenceactioncount(func)Returns count of elements in sequence wherefunc(element)is Trueactionempty()ReturnsTrueif the sequence has zero lengthactionnon_empty()ReturnsTrueif sequence has non-zero lengthactionall()ReturnsTrueif all elements in sequence are truthyactionexists(func)ReturnsTrueiffunc(element)for any element in the sequence isTrueactionfor_all(func)ReturnsTrueiffunc(element)isTruefor all elements in the sequenceactionfind(func)Returns the element that first evaluatesfunc(element)toTrueactionany()ReturnsTrueif any element in sequence is truthyactionmax()Returns maximal element in sequenceactionmin()Returns minimal element in sequenceactionmax_by(func)Returns element with maximal valuefunc(element)actionmin_by(func)Returns element with minimal valuefunc(element)actionsum()/sum(projection)Returns the sum of elements possibly using a projectionactionproduct()/product(projection)Returns the product of elements possibly using a projectionactionaverage()/average(projection)Returns the average of elements possibly using a projectionactionaggregate(func)/aggregate(seed, func)/aggregate(seed, func, result_map)Aggregate usingfuncstarting withseedor first element of list then applyresult_mapto the resultactionfold_left(zero_value, func)Reduces element from left to right usingfuncand initial valuezero_valueactionfold_right(zero_value, func)Reduces element from right to left usingfuncand initial valuezero_valueactionmake_string(separator)Returns string withseparatorbetween eachstr(element)actiondict(default=None)/to_dict(default=None)Converts a sequence of(Key, Value)pairs to adictionary. Ifdefaultis not None, it must be a value or zero argument callable which will be used to create acollections.defaultdictactionlist()/to_list()Converts sequence to a listactionset() / to_set()Converts sequence to a setactionto_file(path)Saves the sequence to a file at path with each element on a newlineactionto_csv(path)Saves the sequence to a csv file at path with each element representing a rowactionto_jsonl(path)Saves the sequence to a jsonl file with each element being transformed to json and printed to a new lineactionto_json(path)Saves the sequence to a json file. The contents depend on if the json root is an array or dictionaryactionto_sqlite3(conn, tablename_or_query, *args, **kwargs)Save the sequence to a SQLite3 db. The target table must be created in advance.actionto_pandas(columns=None)Converts the sequence to a pandas DataFrameactioncache()Forces evaluation of sequence immediately and caches the resultactionfor_each(func)Executesfuncon each element of the sequenceactionLazy ExecutionWhenever possible,PyFunctionalwill compute lazily. This is accomplished by tracking the list
of transformations that have been applied to the sequence and only evaluating them when an action is
called. InPyFunctionalthis is called tracking lineage. This is also responsible for the
ability forPyFunctionalto cache results of computation to prevent expensive re-computation.
This is predominantly done to preserve sensible behavior and used sparingly. For example, callingsize()will cache the underlying sequence. If this was not done and the input was an iterator,
then further calls would operate on an expired iterator since it was used to compute the length.
Similarly,repralso caches since it is most often used during interactive sessions where its
undesirable to keep recomputing the same value. Below are some examples of inspecting lineage.deftimes_2(x):print(x)return2*xelements=seq(1,1,2,3,4).map(times_2).distinct()elements._lineage# Lineage: sequence -> map(times_2) -> distinctl_elements=elements.to_list()# Prints: 1# Prints: 1# Prints: 2# Prints: 3# Prints: 4elements._lineage# Lineage: sequence -> map(times_2) -> distinct -> cachel_elements=elements.to_list()# The cached result is returned so times_2 is not called and nothing is printedFiles are given special treatment if opened through theseq.openand related APIs.functional.util.ReusableFileimplements a wrapper around the standard python file to support
multiple iteration over a single file object while correctly handling iteration termination and
file closing.Road Map IdeaSQL based query planner and interpreter_lambda operatorContributing and Bug FixesAny contributions or bug reports are welcome. Thus far, there is a 100% acceptance rate for pull
requests and contributors have offered valuable feedback and critique on code. It is great to hear
from users of the package, especially what it is used for, what works well, and what could be
improved.To contribute, create a fork ofPyFunctional, make your changes, then make sure that they pass.
In order to be merged, all pull requests must:Pass all the unit testsPass all the pylint tests, or ignore warnings with explanation of why its correct to do soNot significantly reduce covrage without a good reasoncoveralls.io)Edit theCHANGELOG.mdfile in theNext Releaseheading with changesContactGitter for chatSupported Python VersionsPyFunctional1.4 and above supports and is tested against Python 3.6, Python 3.7, and PyPy3PyFunctional1.4 and above does not support python 2.7PyFunctional1.4 and above works in Python 3.5, but is not tested against itPyFunctional1.4 and above partially works in 3.8, parallel processing currently has issues, but other feature work finePyFunctional1.3 and below supports and was tested against Python 2.7, Python 3.5, Python 3.6, PyPy2, and PyPy3ChangelogChangelogAbout meTo learn more about me (the author) visit my webpage atpedro.ai.I createdPyFunctionalwhile using Python extensivel, and finding that I missed the
ease of use for manipulating data that Spark RDDs and Scala collections have. The project takes the
best ideas from these APIs as well as LINQ to provide an easy way to manipulate data when using
Scala is not an option or PySpark is overkill.ContributorsThese people have generously contributed their time to improvingPyFunctionalversaeadrian17lucidfrontier45DigenisChuyuHsujsemric