Package: unpythonic
Unpythonic: Python meets Lisp and HaskellIn the spirit oftoolz, we provide missing features for Python, mainly from the list processing tradition, but with some Haskellisms mixed in. We extend the language with a set ofsyntactic macros. We also provide an in-process, backgroundREPLserver for live inspection and hot-patching. The emphasis is onclear, pythonic syntax,making features work together, andobsessive correctness.Some hypertext features of this README, such as local links to detailed documentation, and expandable example highlights, are not supported when viewed on PyPI;view on GitHubto have those work properly.DependenciesNone required.mcpyrateoptional, to enable the syntactic macro layer, an interactive macro REPL, and some example dialects.The 0.15.x series should run on CPython 3.6, 3.7, 3.8 and 3.9, and PyPy3 (language versions 3.6 and 3.7); theCIprocess verifies the tests pass on those platforms.Long-term support roadmap.DocumentationREADME: you are here.Pure-Python feature setSyntactic macro feature setExamples of creating dialects usingmcpyrate: Python the way you want it.REPL server: interactively hot-patch your running Python program.Troubleshooting: possible solutions to possibly common issues.Design notes: for more insight into the design choices ofunpythonic.Essays: for writings on the philosophy ofunpythonic, things that inspired it, and related discoveries.Additional reading: links to material relevant in the context ofunpythonic.Contribution guidelines: for understanding the codebase, or if you're interested in making a code or documentation PR.The features ofunpythonicare built out of, in increasing order ofmagic:Pure Python (e.g. batteries foritertools),Macros driving a pure-Python core (do,let),Pure macros (e.g.continuations,lazify,dbg).Whole-module transformations, a.k.a. dialects (e.g.Lispy).This depends on the purpose of each feature, as well as ease-of-use considerations. See the design notes for more information.ExamplesSmall, limited-space overview of the overall flavor. There is a lot more that does not fit here, especially in the pure-Python feature set. We give here simple examples that arenotnecessarily of the most general form supported by the constructs. See thefull documentationandunit testsfor more examples.Unpythonic in 30 seconds: Pure PythonLoop functionally, with tail call optimization.[docs]fromunpythonicimportlooped,looped_over@loopeddefresult(loop,acc=0,i=0):ifi==10:returnaccelse:returnloop(acc+i,i+1)# tail call optimized, no call stack blowup.assertresult==45@looped_over(range(3),acc=[])defresult(loop,i,acc):acc.append(lambdax:i*x)# fresh "i" each time, no mutation of loop counter.returnloop()assert[f(10)forfinresult]==[0,10,20]Introduce dynamic variables.[docs]fromunpythonicimportdyn,make_dynvarmake_dynvar(x=42)# set a default valuedeff():assertdyn.x==17withdyn.let(x=23):assertdyn.x==23g()assertdyn.x==17defg():assertdyn.x==23assertdyn.x==42withdyn.let(x=17):assertdyn.x==17f()assertdyn.x==42Interactively hot-patch your running Python program.[docs]To opt in, add just two lines of code to your main program:fromunpythonic.netimportserverserver.start(locals={})# automatically daemonicimporttimedefmain():whileTrue:time.sleep(1)if__name__=='__main__':main()Or if you just want to take this for a test run, start the built-in demo app:python3-munpythonic.net.serverOnce a server is running, to connect:python3-munpythonic.net.client127.0.0.1This gives you a REPL, inside your live process, with all the power of Python. You canimportlib.reloadany module, and throughsys.modules, inspect or overwrite any name at the top level of any module. You canpickle.dumpyour data. Or do anything you want with/to the live state of your app.You can have multiple REPL sessions connected simultaneously. When your app exits (for any reason), the server automatically shuts down, closing all connections if any remain. But exiting the client leaves the server running, so you can connect again later - that's the whole point.Optionally, if you havemcpyrate, the REPL sessions support importing, invoking and defining macros.Industrial-strength scan and fold.[docs]Scan and fold accept multiple iterables, like in Racket.fromoperatorimportaddfromunpythonicimportscanl,foldl,unfold,take,Valuesasserttuple(scanl(add,0,range(1,5)))==(0,1,3,6,10)defop(e1,e2,acc):returnacc+e1*e2assertfoldl(op,0,(1,2),(3,4))==11defnextfibo(a,b):returnValues(a,a=b,b=a+b)asserttuple(take(10,unfold(nextfibo,1,1)))==(1,1,2,3,5,8,13,21,34,55)Industrial-strength curry.[docs]We bind arguments to parameters like Python itself does, so it does not matter whether arguments are passed by position or by name during currying. We support@genericmultiple-dispatch functions.We also feature a Haskell-inspired passthrough system: any args and kwargs that are not accepted by the call signature will be passed through. This is useful when a curried function returns a new function, which is then the target for the passthrough. See the docs for details.fromunpythonicimportcurry,generic,foldr,composerc,cons,nil,ll@currydeff(x,y):returnx,yassertf(1,2)==(1,2)assertf(1)(2)==(1,2)assertf(1)(y=2)==(1,2)assertf(y=2)(x=1)==(1,2)@currydefadd3(x,y,z):returnx+y+z# actually uses partial application so these work, tooassertadd3(1)(2)(3)==6assertadd3(1,2)(3)==6assertadd3(1)(2,3)==6assertadd3(1,2,3)==6@currydeflispyadd(*args):returnsum(args)assertlispyadd()==0# no args is a valid arity here@genericdefg(x:int,y:int):return"int"@genericdefg(x:float,y:float):return"float"@genericdefg(s:str):return"str"g=curry(g)assertcallable(g(1))assertg(1)(2)=="int"assertcallable(g(1.0))assertg(1.0)(2.0)=="float"assertg("cat")=="str"assertg(s="cat")=="str"# simple example of passthroughmymap=lambdaf:curry(foldr,composerc(cons,f),nil)myadd=lambdaa,b:a+bassertcurry(mymap,myadd,ll(1,2,3),ll(2,4,6))==ll(3,6,9)Multiple-dispatch generic functions, like in CLOS or Julia.[docs]fromunpythonicimportgeneric@genericdefmy_range(stop:int):# create the generic function and the first multimethodreturnmy_range(0,1,stop)@genericdefmy_range(start:int,stop:int):# further registrations add more multimethodsreturnmy_range(start,1,stop)@genericdefmy_range(start:int,step:int,stop:int):returnstart,step,stopThis is a purely run-time implementation, so it doesnotgive performance benefits, but it can make code more readable, and makes it modular to add support for new input types (or different call signatures) to an existing function later.Holy traitsare also a possibility:importtypingfromunpythonicimportgeneric,augmentclassFunninessTrait:passclassIsFunny(FunninessTrait):passclassIsNotFunny(FunninessTrait):pass@genericdeffunny(x:typing.Any):# defaultraiseNotImplementedError(f"`funny` trait not registered for anything matching{type(x)}")@augment(funny)deffunny(x:str):# noqa: F811returnIsFunny()@augment(funny)deffunny(x:int):# noqa: F811returnIsNotFunny()@genericdeflaugh(x:typing.Any):returnlaugh(funny(x),x)@augment(laugh)deflaugh(traitvalue:IsFunny,x:typing.Any):returnf"Ha ha ha,{x}is funny!"@augment(laugh)deflaugh(traitvalue:IsNotFunny,x:typing.Any):returnf"{x}is not funny."assertlaugh("that")=="Ha ha ha, that is funny!"assertlaugh(42)=="42 is not funny."Conditions: resumable, modular error handling, like in Common Lisp.[docs]Contrived example:fromunpythonicimporterror,restarts,handlers,invoke,use_value,unboxclassMyError(ValueError):def__init__(self,value):# We want to act on the value, so save it.self.value=valuedeflowlevel(lst):_drop=object()# gensym/nonceout=[]forkinlst:# Provide several different error recovery strategies.withrestarts(use_value=(lambdax:x),halve=(lambdax:x//2),drop=(lambda:_drop))asresult:ifk>9000:error(MyError(k))# This is reached when no error occurs.# `result` is a box, send k into it.result<<k# Now the result box contains either k,# or the return value of one of the restarts.r=unbox(result)# get the value from the boxifrisnot_drop:out.append(r)returnoutdefhighlevel():# Choose which error recovery strategy to use...withhandlers((MyError,lambdac:use_value(c.value))):assertlowlevel([17,10000,23,42])==[17,10000,23,42]# ...on a per-use-site basis...withhandlers((MyError,lambdac:invoke("halve",c.value))):assertlowlevel([17,10000,23,42])==[17,5000,23,42]# ...without changing the low-level code.withhandlers((MyError,lambda:invoke("drop"))):assertlowlevel([17,10000,23,42])==[17,23,42]highlevel()Conditions only shine in larger systems, with restarts set up at multiple levels of the call stack; this example is too small to demonstrate that. The single-level case here could be implemented as a error-handling mode parameter for the example's only low-level function.With multiple levels, it becomes apparent that this mode parameter must be threaded through the API at each level, unless it is stored as a dynamic variable (seeunpythonic.dyn). But then, there can be several types of errors, and the error-handling mode parameters - one for each error type - have to be shepherded in an intricate manner. A stack is needed, so that an inner level may temporarily override the handler for a particular error type...The condition system is the clean, general solution to this problem. It automatically scopes handlers to their dynamic extent, and manages the handler stack automatically. In other words, it dynamically binds error-handling modes (for several types of errors, if desired) in a controlled, easily understood manner. The local programmability (i.e. the fact that a handler is not just a restart name, but an arbitrary function) is a bonus for additional flexibility.If this sounds a lot like an exception system, that's because conditions are the supercharged sister of exceptions. The condition model cleanly separates mechanism from policy, while otherwise remaining similar to the exception model.Lispy symbol type.[docs]Roughly, asymbolis a guaranteed-internedstring.Agensymis a guaranteed-uniquestring, which is useful as a nonce value. It's similar to the pythonic idiomnonce = object(), but with a nice repr, and object-identity-preserving pickle support.fromunpythonicimportsym# lispy symbolsandwich=sym("sandwich")hamburger=sym("sandwich")# symbol's identity is determined by its name, onlyasserthamburgerissandwichassertstr(sandwich)=="sandwich"# symbols have a nice str()assertrepr(sandwich)=='sym("sandwich")'# and eval-able repr()asserteval(repr(sandwich))issandwichfrompickleimportdumps,loadspickled_sandwich=dumps(sandwich)unpickled_sandwich=loads(pickled_sandwich)assertunpickled_sandwichissandwich# symbols survive a pickle roundtripfromunpythonicimportgensym# gensym: make new uninterned symboltabby=gensym("cat")scottishfold=gensym("cat")asserttabbyisnotscottishfoldpickled_tabby=dumps(tabby)unpickled_tabby=loads(pickled_tabby)assertunpickled_tabbyistabby# also gensyms survive a pickle roundtripLispy data structures.[docs forbox] [docs forcons] [docs forfrozendict]fromunpythonicimportbox,unbox# mutable single-item containercat=object()cardboardbox=box(cat)assertcardboardboxisnotcat# the box is not the catassertunbox(cardboardbox)iscat# but the cat is inside the boxassertcatincardboardbox# ...also syntacticallydog=object()cardboardbox<<dog# hey, it's my box! (replace contents)assertunbox(cardboardbox)isdogfromunpythonicimportcons,nil,ll,llist# lispy linked listslst=cons(1,cons(2,cons(3,nil)))assertll(1,2,3)==lst# make linked list out of elementsassertllist([1,2,3])==lst# convert iterable to linked listfromunpythonicimportfrozendict# immutable dictionaryd1=frozendict({'a':1,'b':2})d2=frozendict(d1,c=3,a=4)assertd1==frozendict({'a':1,'b':2})assertd2==frozendict({'a':4,'b':2,'c':3})Allow a lambda to call itself. Name a lambda.[docs forwithself] [docs fornamelambda]fromunpythonicimportwithself,namelambdafact=withself(lambdaself,n:n*self(n-1)ifn>1else1)# see @trampolined to do this with TCOassertfact(5)==120square=namelambda("square")(lambdax:x**2)assertsquare.__name__=="square"assertsquare.__qualname__=="square"# or e.g. "somefunc.<locals>.square" if inside a functionassertsquare.__code__.co_name=="square"# used by stack tracesBreak infinite recursion cycles.[docs]fromtypingimportNoReturnfromunpythonicimportfix@fix()defa(k):returnb((k+1)%3)@fix()defb(k):returna((k+1)%3)asserta(0)isNoReturnBuild number sequences by example. Slice general iterables.[docs fors] [docs forislice]fromunpythonicimports,isliceseq=s(1,2,4,...)asserttuple(islice(seq)[:10])==(1,2,4,8,16,32,64,128,256,512)Memoize functions and generators.[docs formemoize] [docs forgmemoize]fromitertoolsimportcount,takewhilefromunpythonicimportmemoize,gmemoize,islicencalls=0@memoize# <-- important partdefsquare(x):globalncallsncalls+=1returnx**2assertsquare(2)==4assertncalls==1assertsquare(3)==9assertncalls==2assertsquare(3)==9assertncalls==2# called only once for each unique set of arguments# "memoize lambda": classic evaluate-at-most-once thunkthunk=memoize(lambda:print("hi from thunk"))thunk()# the message is printed only the first timethunk()@gmemoize# <-- important partdefprimes():# FP sieve of Eratosthenesyield2fornincount(start=3,step=2):ifnotany(n%p==0forpintakewhile(lambdax:x*x<=n,primes())):yieldnasserttuple(islice(primes())[:10])==(2,3,5,7,11,13,17,19,23,29)Functional updates.[docs]fromitertoolsimportrepeatfromunpythonicimportfupt=(1,2,3,4,5)s=fup(t)[0::2]<<repeat(10)asserts==(10,2,10,4,10)assertt==(1,2,3,4,5)fromitertoolsimportcountfromunpythonicimportimemoizet=(1,2,3,4,5)s=fup(t)[::-2]<<imemoize(count(start=10))()asserts==(12,2,11,4,10)assertt==(1,2,3,4,5)Live list slices.[docs]fromunpythonicimportviewlst=list(range(10))v=view(lst)[::2]# [0, 2, 4, 6, 8]v[2:4]=(10,20)# re-slicable, still live.assertlst==[0,1,2,3,10,5,20,7,8,9]lst[2]=42assertv==[0,42,10,20,8]Pipes: method chaining syntax for regular functions.[docs]fromunpythonicimportpiped,exitpipedouble=lambdax:2*xinc=lambdax:x+1x=piped(42)|double|inc|exitpipeassertx==85The point is usability: in a function composition using pipe syntax, data flows from left to right.Unpythonic in 30 seconds: Language extensions with macrosunpythonic.test.fixtures: a minimalistic test framework for macro-enabled Python.[docs]fromunpythonic.syntaximportmacros,test,test_raises,fail,error,warn,thefromunpythonic.test.fixturesimportsession,testset,terminate,returns_normallydeff():raiseRuntimeError("argh!")defg(a,b):returna*bfail["this line should be unreachable"]count=0defcounter():globalcountcount+=1returncountwithsession("simple framework demo"):withtestset():test[2+2==4]test_raises[RuntimeError,f()]test[returns_normally(g(2,3))]test[g(2,3)==6]# Use `the[]` (or several) in a `test[]` to declare what you want to inspect if the test fails.# Implicit `the[]`: in comparison, the LHS; otherwise the whole expression. Used if no explicit `the[]`.test[the[counter()]<the[counter()]]withtestset("outer"):withtestset("inner 1"):test[g(6,7)==42]withtestset("inner 2"):test[NoneisNone]withtestset("inner 3"):# an empty testset is considered 100% passed.passwithtestset("inner 4"):warn["This testset not implemented yet"]withtestset("integration"):try:importblarglyexceptImportError:error["blargly not installed, cannot test integration with it."]else:...# blargly integration tests go herewithtestset(postproc=terminate):test[2*2==5]# fails, terminating the nearest dynamically enclosing `with session`test[2*2==4]# not reachedWe provide the low-level syntactic constructstest[],test_raises[]andtest_signals[], with the usual meanings. The last one is for testing code that uses conditions and restarts; seeunpythonic.conditions.The test macros also come in block variants,with test,with test_raises,with test_signals.As usual in test frameworks, the testing constructs behave somewhat likeassert, with the difference that a failure or error will not abort the whole unit (unless explicitly asked to do so).let: expression-local variables.[docs]fromunpythonic.syntaximportmacros,let,letseq,letrecx=let[[a<<1,b<<2]ina+b]y=letseq[[c<<1,# LET SEQuential, like Scheme's let*c<<2*c,c<<2*c]inc]z=letrec[[evenp<<(lambdax:(x==0)oroddp(x-1)),# LET mutually RECursive, like in Schemeoddp<<(lambdax:(x!=0)andevenp(x-1))]inevenp(42)]let-over-lambda: stateful functions.[docs]fromunpythonic.syntaximportmacros,dlet# Up to Python 3.8, use `@dlet(x << 0)` instead@dlet[x<<0]# let-over-lambda for Pythondefcount():returnx<<x+1# `name << value` rebinds in the let envassertcount()==1assertcount()==2do: code imperatively in any expression position.[docs]fromunpythonic.syntaximportmacros,do,local,deletex=do[local[a<<21],local[b<<2*a],print(b),delete[b],# do[] local variables can be deleted, too4*a]assertx==84Automatically apply tail call optimization (TCO), à la Scheme/Racket.[docs]fromunpythonic.syntaximportmacros,tcowithtco:# expressions are automatically analyzed to detect tail position.evenp=lambdax:(x==0)oroddp(x-1)oddp=lambdax:(x!=0)andevenp(x-1)assertevenp(10000)isTrueCurry automatically, à la Haskell.[docs]fromunpythonic.syntaximportmacros,autocurryfromunpythonicimportfoldr,composercascompose,cons,nil,llwithautocurry:defadd3(a,b,c):returna+b+cassertadd3(1)(2)(3)==6mymap=lambdaf:foldr(compose(cons,f),nil)double=lambdax:2*xassertmymap(double,(1,2,3))==ll(2,4,6)Lazy functions, a.k.a. call-by-need.[docs]fromunpythonic.syntaximportmacros,lazifywithlazify:defmy_if(p,a,b):ifp:returna# b never evaluated in this code pathelse:returnb# a never evaluated in this code pathassertmy_if(True,23,1/0)==23assertmy_if(False,1/0,42)==42Genuine multi-shot continuations (call/cc).[docs]fromunpythonic.syntaximportmacros,continuations,call_ccwithcontinuations:# enables also TCO automatically# McCarthy's amb() operatorstack=[]defamb(lst,cc):ifnotlst:returnfail()first,*rest=tuple(lst)ifrest:remaining_part_of_computation=ccstack.append(lambda:amb(rest,cc=remaining_part_of_computation))returnfirstdeffail():ifstack:f=stack.pop()returnf()# Pythagorean triples using amb()defpt():z=call_cc[amb(range(1,21))]# capture continuation, auto-populate cc argy=call_cc[amb(range(1,z+1))]x=call_cc[amb(range(1,y+1))]ifx*x+y*y!=z*z:returnfail()returnx,y,zt=pt()whilet:print(t)t=fail()# note pt() has already returned when we call this.Unpythonic in 30 seconds: Language extensions with dialectsThedialects subsystem ofmcpyratemakes Python into a language platform, à laRacket. We provide some example dialects based onunpythonic's macro layer. Seedocumentation.Lispython: automatic TCO and an implicit return statement.[docs]Also comes with automatically named, multi-expression lambdas.fromunpythonic.dialectsimportdialects,Lispython# noqa: F401deffactorial(n):deff(k,acc):ifk==1:returnaccf(k-1,k*acc)f(n,acc=1)assertfactorial(4)==24factorial(5000)# no crashsquare=lambdax:x**2assertsquare(3)==9assertsquare.__name__=="square"# - brackets denote a multiple-expression lambda body#   (if you want to have one expression that is a literal list,#    double the brackets: `lambda x: [[5 * x]]`)# - local[name << value] makes an expression-local variableg=lambdax:[local[y<<2*x],y+1]assertg(10)==21Pytkell: Automatic currying and implicitly lazy functions.[docs]fromunpythonic.dialectsimportdialects,Pytkell# noqa: F401fromoperatorimportadd,muldefaddfirst2(a,b,c):returna+bassertaddfirst2(1)(2)(1/0)==3asserttuple(scanl(add,0,(1,2,3)))==(0,1,3,6)asserttuple(scanr(add,0,(1,2,3)))==(0,3,5,6)my_sum=foldl(add,0)my_prod=foldl(mul,1)my_map=lambdaf:foldr(compose(cons,f),nil)assertmy_sum(range(1,5))==10assertmy_prod(range(1,5))==24double=lambdax:2*xassertmy_map(double,(1,2,3))==ll(2,4,6)Listhell: Prefix syntax for function calls, and automatic currying.[docs]fromunpythonic.dialectsimportdialects,Listhell# noqa: F401fromoperatorimportadd,mulfromunpythonicimportfoldl,foldr,cons,nil,ll(print,"hello from Listhell")my_sum=(foldl,add,0)my_prod=(foldl,mul,1)my_map=lambdaf:(foldr,(compose,cons,f),nil)assert(my_sum,(range,1,5))==10assert(my_prod,(range,1,5))==24double=lambdax:2*xassert(my_map,double,(q,1,2,3))==(ll,2,4,6)InstallationPyPIpip3 install unpythonic --userorsudo pip3 install unpythonicGitHubClone (or pull) from GitHub. Then,python3 setup.py install --userorsudo python3 setup.py installUninstallUninstallation must be invoked in a folder which has no subfolder calledunpythonic, so thatpiprecognizes it as a package name (instead of a filename). Then,pip3 uninstall unpythonicorsudo pip3 uninstall unpythonicSupportNot working as advertised? Missing a feature? Documentation needs improvement?In case of a problem, seeTroubleshootingfirst. Then:Issue reportsandpull requestsare welcome.Contribution guidelines.Whileunpythonicis intended as a serious tool for improving productivity as well as for teaching, right now my work priorities mean that it's developed and maintained on whatever time I can spare for it. Thus getting a response may take a while, depending on which project I happen to be working on.LicenseAll original code is released under the 2-clauseBSD license.For sources and licenses of fragments originally seen on the internet, seeAUTHORS.AcknowledgementsThanks toTUTfor letting me teachRAK-19006 in spring term 2018; early versions of parts of this library were originally developed as teaching examples for that course. Thanks to @AgenttiX for early feedback.Relevant readingLinks to blog posts, online articles and papers on topics relevant in the context ofunpythonichave been collected toa separate document.If you like both FP and numerics, we havesome examplesbased on various internet sources.