Package: cpy2py
Multi-intepreter execution environmentcpy2pyallows multiple interpreters to act as one application. In parallel
to the main interpreter, other interpreters are run to execute parts of
the application.Table of ContentsQuick GuideTwinterpreters and TwinMastersTwinObjectsTwinFunctionsDebuggingCurrent StatusFeaturesGotchas/LimitationsPerformanceQuick GuideTwinterpreters and TwinMastersA twinterpreter is simply another interpreter running as a subprocess -
with some glue and magic sprinkled on it. You can control and create them
using acpy2py.TwinMaster.You should only ever worry about two methods:TwinMaster.startlaunches the twinterpreter.TwinMaster.executeexecutes
an arbitrary callable in the twinterpreter.fromcpy2pyimportTwinMasterfrommy_moduleimportmy_functiontwinterpreter=TwinMaster('pypy')twinterpreter.start()if__name__=="__main__":twinterpreter.execute(my_function,1,2,3,'ka-pow!',doctor="who?")TwinObjectsThe real power ofcpy2pyare Twins - objects living in one
twinterpreter and being represented by proxies in any other interpeter.
Using twins, you can seamlessly split your application across multiple
twinterpreters.You create twins by inheriting fromcpy2py.TwinObjectinstead ofobjectand
setting a__twin_id__. That’s it.fromcpy2pyimportTwinObjectclassSuperComputer(TwinObject):__twin_id__='pypy'# makes class native to pypy twinterpeterdefmegaloop(self,x,y):returnsum(a+bforainrange(x)forbinrange(y))classCWrapper(TwinObject):__twin_id__='python'# makes class native to python twinterpeterdefcallme(self,who,what="buy milk"):returnsome_clib.c_fcn_cll_cplx_xmpl(who,what)If you don’t set__twin_id__on a child ofcpy2py.TwinObject,
the class will always be native to the main interpreter. Handy for all
the stuff that’s needed everywhere but really doesn’t belong anywhere.note:At the moment, you have to explicitly start a class’s native
twinterpreter before instantiating the class. Only the main
interpreter is always available, of course.TwinFunctionsInstead of full-fletched objects, you can also define functions as twins.
These are automatically called in their native twinterpreter.fromcpy2pyimporttwinfunction@twinfunction('pypy')defsuperlooper(count=1000,add=3,start=0):for_inrange(count):start+=addreturnaddprint(superlooper(int(1E6),1))note:Acpy2py.twinfunctionis a regular function wrapping a
callable. Unlike acpy2py.TwinObject, it will not pass
attribute assignments.DebuggingThe core ofcpy2pysupports someloggingfacilities.
All such loggers are children of the__cpy2py__logger. By default,
no active handlers are attached and propagation is disabled. If needed,
you reconfigure them like any otherlogginglogger to suit your
needs.
Note that if python is run with the-Oflag, several logging calls are
skipped entirely to improve performance.For small scale debugging, one can set the environment variableCPY2PY_DEBUG. If it is defined and not empty, logging output
is written tostderr. In addition, if it names a validlogginglevel, that logging level is used.Note that loggers are meant for development and only address the internal
state. Your application should not depend on this information. Unlesscpy2pymisbehaves (or you suspect it to), ignore its logging.Current StatusCPy2Py is stable at its core, but still has some features missing.
What’s there is more than sufficient to significantly enhance your applications.FeaturesSeamlessly integrates into python code.All internals are wrapped away behind the plain python interfaces.
No eval, exec or code strings required.Lightweight hooks optimize objects and functions for use withcpy2py.If needed,anypickle’able callable can be dispatched to another interpreter.Objects natively integrate with twinterpreters.Objects can live in a specific interpreter, with proxies replacing them in others.
Classes and instances transparently interact withcpy2pyin the background.Both class and instance attributes work as expected.
Methods, classmethods, staticmethods and descriptors are fully supported.Inheritance is fully supported, including multiple inheritance.
Affiliation to interpreters can be changed freely.A wide range of interpeters is supported.Pure python, no dependencies means perfect portability.Any interpreter compatible with python 2.6 to 3.7 is supported.Virtual Environments work out of the box.Tested with cpython and pypy, on Linux and Mac OSX.Gotchas/LimitationsImporting functions and classes from__main__may fail if the module can only be imported via its path.By default, calls across interpreters are blocking and not threadsafe.
If recursion switches between twinterpreters,cpy2py.TwinMastermust use the'async'kernel.Module level settings are not synchronized.
For example, configuration ofloggingis not applied to twinterpreters.
Usecpy2py.twinterpreter.group_state.TwinGroupStatefor initialisation,
write modules aware of twinterpreters, or use immutable module-level initializers.Aweakrefto objects only takes local references into account, not cross-interpreter references.PerformanceDispatching to another twinterpreter adds about 200 - 300 us of overhead.
This is mainly due to serialization for the IPC between the interpreters.
Using the asynchronous kernel, there is an additional overhead for creating threads.In general, twinterpreters get faster the shorter they have to wait between requests.pypytwinterpreters benefit from a high number of requests, allowing their JIT to warm up.
Python3 connections are the fastest, provided that both twinterpreters support pickle protocol 4.A notable fraction of time is spent on debugging output vialogging.
Even if no output is produced,cpy2pyis optimized to a point where thelogging callis noticeable.
If needed, any per-call logging can be disabled by running python in optimized mode.
See the python documentation on the-Ooption andPYTHONOPTIMIZEenvironment variable.You can benchmark the overhead yourself using thecpy2py_benchmarktools.pypy215x15k30x5k300x1pypy2187 ±  1.5 us228 ±  2.5 us505 ± 51.8 uspypy3165 ±  1.3 us209 ±  2.4 us402 ±  8.0 uspython2.7178 ±  0.6 us139 ±  0.3 us239 ±  7.6 uspython3.4149 ±  0.4 us118 ±  0.2 us258 ±  8.0 us