Package: python-bond
ContentsOverviewPractical examplesIncremental code-base migrationMixing Python 2/3 code basesRemote/parallel computationPrivilege separationAPIInitializationbond.BondMethodsExceptionsQuoted expressionsLanguage supportPythonPHPPerlJavaScriptCommon limitationsGeneral/support mailing listAuthors and CopyrightLatest release notesThe Python modulebondsupports transparent remote/recursive evaluation
between Python and another interpreter through automatic call serialization.In poorer words, abondlets you call functions in other languages as they
were normal Python functions. Italsoallows other languages tocall Python
functionsas if they were native.Remote output is also transparently redirected locally, and since the
evaluation is performed through a persistent co-process, you can actually spawn
interpreters on different hosts through “ssh” efficiently.bondcurrently supports PHP, Perl, JavaScript (Node.js) and Python itself.Overview>>># Let's bond with a PHP interpreter>>>frombondimportmake_bond>>>php=make_bond('PHP')>>>php.eval_block('echo "Hello world!\n";')Helloworld!>>># Make an expensive split function using PHP's explode>>>split=php.callable('explode')>>>split(' ',"Hello world splitted by PHP!")[u'Hello',u'world',u'splitted',u'by',u'PHP!']>>># Call Python from PHP>>>defcall_me():...print("Hi, this is Python talking!")>>>php.export(call_me)>>>php.eval('call_me()')Hi,thisisPythontalking!>>># Use some remote resources>>>remote_php=make_bond('PHP','ssh remote php')>>>remote_php.eval_block('function call_me() { echo "Hi from " . system("hostname") . "!"; }')>>>remote_php.eval('call_me()')Hifromremote!>>># Bridge two worlds!>>>perl=make_bond('Perl')>>>php.proxy('explode',perl)>>># note: explode is now available to Perl, but still executes in PHP>>>perl.eval('explode("=", "Mind=blown!")')[u'Mind',u'blown!']Practical examplesIncremental code-base migrationI originally neededbondfor migrating a large PHP project to Python. Withbondyou can rewrite a program incrementally, while still executing all the
existing code unchanged. You can start by rewriting just a single function in
an empty shell, wrapping your existing code:frombondimportmake_bondimportsysphp=make_bond('PHP')php.eval_block('include("my_original_program.php");')defnew_function(arg)# do something herepassphp.export(new_function,'function_to_be_replaced')php.call('main',sys.argv)Mixing Python 2/3 code basesYou can usebondto mix Python 2/3 code. Python <=> Python bonds
automatically use pickling as a protocol, which makes serialization almost
invisible.In this scenario, you can start writing new code directly on Python 3, while
using Python 2 only for the libraries which are still missing.For example, you can useMechanizeon Python 3 with minimal changes:frombondimportmake_bondpy2=make_bond('Python','python2',trans_except=False)py2.eval_block('import mechanize; br = mechanize.Browser()')py2.call('br.open','http://www.example.com')title=py2.call('br.title')eval_blockis only being used as an example here to make it self-contained.
A more reasonable solution for larger chunks of code is to split the source
into a distinct file that can be loaded at once in the remote interpreter:frombondimportmake_bondpy2=make_bond('Python','python2',trans_except=False)py2.eval_block('import .mypython2lib')This reduces the amount of clutter and keeps the distinction between new and
legacy code clear. You should also keep in mind that since the remote language
is itself Python, expressions themselves (for whenevercallis
insufficient) can be quoted just by usingrepr.Similarly, you can usebondto combine regular CPython and PyPy runtimes
(all the required modules work as expected on PyPy):frombondimportmake_bondpypy=make_bond('Python','pypy')Remote/parallel computationYou can easily usebondto perform remote/parallel computation. Nobody
stops you from having multiple interpreters at the same time: you can create
multiple bonds to setup a poor-man’s distributed system with minimal effort:# setup the workersfrombondimportmake_bondhosts=['host1','host2','host3']nodes=[make_bond('Python','ssh{}python'.format(host))forhostinhosts]# load our libraries firstfornodeinnodes:node.eval_block('from library import *')# execute "do_something" remotely on each workerfromthreadingimportThreadthreads=[Thread(target=lambda:node.call('do_something'))fornodeinnodes]forthreadinthreads:thread.start()# collect the resultsresults=[thread.join()forthreadinthreads]Distributed producer/consumer schemes also come for free by proxying calls:host1.eval_block(r'''def consumer(data):
   # do something with data
   pass
''')host2.eval_block(r'''def producer():
    while True:
       data = function()
       consumer(data)
''')host1.proxy('consumer',host2)host2.call('producer')It’s even more interesting if you realize that the producers/consumers don’t
even have to be written in the same language, and don’t know that the call is
actually being forwarded.bonddoesn’t even need to be installed remotely: the required setup is
injected directly into a live interpreter. The wire protocol is simple enough
that any language supporting an interactive REPL can be called. In fact,the
drivers themselvesare designed to
be used from any other language.Privilege separationThere might be times when it makes sense to create an unprivileged context to
perform potentially dangerous operations, such as decoding an uploaded file on
which you have zero trust. A common approach would be to communicate with an
unprivileged daemon built for the purpose, but it usually requires dedicated
effort. Running such processes usingbondinstead is almost trivial:# early in the setup phase of our daemon we create a bond using# passwordless sudo, changing to an unprivileged userpy=make_bond('Python','sudo -u nobody python',trans_except=False,protocol='JSON')py.eval_block('from mylibrary import decode_file')# make decode_file() available as a normal functiondecode_file=py.callable('decode_file')# assuming decode_file() takes a file name which is at least readable by# the unprivileged user, we can just take it's return valuedata=decode_file(path)Contrarily to other examples involving Python, here we actually restrict the
serialization protocol to plainJSON. Nothing changes from the caller (our)
perspective, except that the bond now can’t share with us anything beyond
trivial types. Python <=> Python bonds usepickleby default, which is not
sensible here aspickleallows arbitrary Python structures and handlers to
be run (including bytecode itself).If just running the context as another user is not enough, then setting up an
LXC container doesn’t add much complexity, since we can just uselxc-executeto attach directly to the new instance’s STDIO:py=make_bond('Python','lxc-execute -n <name> -f <config> /path/to/python',trans_except=False,protocol='JSON')This way an ephemeral container is started/destroyed automatically along with
our daemon. The container itself can expose just a few shared/read-only
directories, or nothing at all if the entire I/O is built on top ofbond.APIInitializationAbond.Bondobject is not normally constructed directly, but by using thebond.make_bond()function:importbondinterpreter=bond.make_bond('language')The first argument should be the desired language name (“JavaScript”, “PHP”,
“Perl”, “Python”). The list of supported languages can be fetched dynamically
usingbond.list_drivers().You can override the default interpreter command using the second argument,
which allows to specify any shell command to be executed:importbondpy=bond.make_bond('Python','ssh remote python3')An additionallistof arguments to the interpreter can be provided using the
third argument,args:importbondpy=bond.make_bond('Python','ssh remote python3',['-E','-OO'])Thearguments, contrarily to the command, are automatically quoted.Some command line arguments may be supplied automatically by the driver to
force an interactive shell; for example “-i” is supplied if Python is
requested. You can disable default arguments by usingdef_args=False.The following keyword arguments are supported:cwd:Working directory for the interpreter (defaults to current working
directory).env:Environment for the interpreter (defaults toos.environ).def_args:Enable (default) or suppress default, extra command-line arguments to the
interpreter.timeout:Defines the timeout for the underlying communication protocol. Note thatbondcannot distinguish between a slow call or noise generated while the
interpreter is set up. Defaults to 60 seconds.logfile:Accepts a file handle which is used to log the entire communication with the
underlying interpreter for debugging purposes.trans_except:Enables/disables “transparent exceptions”. Exceptions are always first class,
but whentrans_exceptis enabled, the exception objects themselves will
be forwarded across the bond. Iftrans_exceptis disabled (the default
for all languages except Python), then local exceptions will always contain a
string representation of the remote exception instead, which avoids
serialization errors.protocol:Forces a specific serialization protocol to be chosen. It’s automatically
selected when not specified, and usually matches “JSON”.bond.BondMethodsThe resultingbond.Bondclass has the following methods:eval(code):Evaluate and return the value of asingle statementof code in the
interpreter.eval_block(code):Execute a “code” block inside the top-level of the interpreter. Any construct
which is legal by the current interpreter is allowed. Nothing is returned.ref(code):Return a reference to ansingle, unevaluated statementof code, which can
be later used in eval(), eval_block() or as animmediateargument to call().
SeeQuoted expressions.close():Terminate the communication with the interpreter.call(name, *args):Call a function “name” in the interpreter using the supplied list of
arguments *args (apply *args to a callable statement defined by “name”).
The arguments are automatically converted to their other language’s
counterpart. The return value is captured and converted back to Python as
well.callable(name):Return a function that calls “name”:explode=php.callable('explode')# Now you can call explode as a normal, local functionexplode(' ','Hello world')export(func, name):Export a local function “func” so that can be called on the remote language
as “name”. If “name” is not specified, use the local function name directly.
Note that “func” must be a functionreference, not a function name.proxy(name, other, remote):Export a function “name” from the currentbondto “other”, named as
“remote”. If “remote” is not provided, the same value as “name” is used.interact():Start an interactive session with the underlying interpreter. By default, all
input lines are executed with bond.eval_block(). If “!” is pre-pended,
execute a single statement with bond.eval() and print it’s return value. You
can continue the statement on multiple lines by leaving a trailing “". Type
Ctrl+C to abort a multi-line block without executing it.ExceptionsAll exceptions thrown by thebondmodule are of base typeRuntimeError<=BondException.BondException:Thrown during initialization or unrecoverable errors.TerminatedException:Thrown when the bond exits unexpectedly.SerializationException:Thrown when an object/exception which is sentorreceived cannot be
serialized by the current protocol. Thesideattribute can be either
“local” (when attempting tosend) or “remote” (whenreceiving). ASerializationExceptionis not fatal.RemoteException:Thrown for uncaught remote exceptions. The “data” attribute contains either
the error message (withtrans_except=False) or the remote exception
itself (trans_except=True).Beware that bothSerializationException(withside="remote") andRemoteExceptionmay actually be originating from uncaughtlocalexceptions when an exported function is called. Pay attention to the error
text/data in these cases, as it will contain several nested exceptions.Quoted expressionsbondhas minimal support for quoted expressions, through the use ofBond.ref().ref()returns a reference to a unevaluated statement that
can be fed back toeval(),eval_block(), or as animmediate(i.e.:
not nested) argument tocall(). References are bound to the interpreter
that created them.ref()allows to “call” methods that take remote un-serializable arguments,
such as file descriptors, without the use of a support function and/or eval:pl=make_bond('Perl')pl.eval_block('open($fd, ">file.txt");')fd=pl.ref('$fd')pl.call('syswrite',fd,"Hello world!")pl.call('close',fd)Sinceref()objects cannot be nested, there are still cases where it might
be necessary to use a support function. To demonstrate, we rewrite the above
example without quoted expressions, while still allowing an argument (“Hello
world!”) to be local:pl=make_bond('Perl')pl.eval_block('open($fd, ">file.txt");')pl.eval_block('sub syswrite_fd { syswrite($fd, shift()); };')pl.call('syswrite_fd',"Hello world!")pl.eval('close($fd)')Or more succinctly:pl.call('sub { syswrite($fd, shift()); }',"Hello world!")Language supportPythonPython, as the identity language, has no restriction on data types. Everything
is pickled on both sides, including exceptions.Serialization:Performed locally and remotely usingcPicklein Python 2 orpicklein Python 3.Serialization exceptions on the remote side are of base typeTypeError<=_BOND_SerializationException.Python 2 / Python 3 / PyPy:You can freely mix Python versions between hosts/interpreters (that is: you can
run Python 3 code from a Python 2 host and vice-versa). You’ll need to disable
transparent exceptions between major versions though, as the exception
hierarchy is different:# assuming a python2.7 environmentfrombondimportmake_bondpy=make_bond('Python','python3',trans_except=False)PHPRequirements:The PHP’s >= 5.3 command line interpreter needs to be installed. On
Debian/Ubuntu, the required package isphp5-cli.Serialization:Performed remotely usingJSON. Implement theJsonSerializableinterface to
tweak which/how objects are encoded.Serialization exceptions on the remote side are of base type_BOND_SerializationException. The detailed results of the error can
also be retrieved usingjson_last_error.Limitations:PHP <= 5.3 doesn’t support theJsonSerializableinterface, and thus lacks
the ability of serializing arbitrary objects.You cannot usecallon a built-in function such as “echo”. You have to
use a real function instead, like “print”. You can still call “echo” by usingevaloreval_block.Unfortunately, you cannot catch “fatal errors” in PHP. If the evaluated code
triggers a fatal error it will terminate the bond without appeal. A common
example of such error can be attempting to use an undefined variable or
function (which could happen while prototyping).Due to the inability to override built-in functions,error_reporting()is
not completely transparent and always returns 0. It shouldn’t be used to
control the display error level. Use_BOND_error_reporting()instead,
which has the same usage/signature as the built-in function.PerlPerl is a quirky language, due to its syntax. We assume here you’re an
experienced Perl developer.Requirements:Perl >= 5.14 is required, with the following modules:JSONData::DumpIO::StringOn Debian/Ubuntu, the required packages arelibjson-perllibdata-dump-perlandlibio-string-perl.Serialization:Performed remotely usingJSON. Implement theTO_JSONmethod on
blessed references to tweak which/how objects are encoded.Serialization exceptions on the remote side are generated by dying with a_BOND_SerializationException@ISA.Gotchas:By default, evaluation is forced in array context, as otherwise most of the
built-ins working with arrays would return an useless scalar. Use the
“scalar” keyword for the rare cases when you really need it to.You can “call” any function-like statement, as long as the last argument is
expected to be an argument list. This allows you to call builtins directly:perl.call('map { $_ + 1 }',[1,2,3])You can of course “call” a statement that returns anyCODE. Meaning that
you can call references to functions as long as you dereference them first:perl.call('&$fun_ref',...)perl.call('&{ $any->{expression}}',...)Likewise you can “call” objects methods directly:perl.call('$object->method',...)eval_blockintroduces a new block. Variables declared as “my” will not be
visible into a subsequenteval_block. Use a fully qualified name or “our”
to define variables that should persist across blocks:perl.eval_block('our $variable = 1;')perl.eval_block('do_something_with($variable);')JavaScriptJavaScript is supported throughNode.js.Requirements:Node.js v0.6.12 and v0.10.29 have been tested. On Debian/Ubuntu, the required
package isnodejs.Serialization:Performed remotely usingJSON. Implement thetoJSONproperty to tweak which/how objects are encoded.Serialization exceptions on the remote side are of base typeTypeError<=_BOND_SerializationException.Limitations:Currently the code expects an unix-like environment with/dev/stdinto
perform synchronous I/O.Since there’s no distinction between “plain” objects (dictionaries) and any
other object, almost everything will be silently serialized. Define a custom
“toJSON” property on your “real” objects to control this behavior.When executing a remote JavaScript bond with Node.js <= 0.6, you need to
manually invoke the REPL, as follows:js=make_bond('JavaScript',"ssh remote node -e 'require\(\\\"repl\\\"\).start\(\)'",def_args=False)When executing “node” locally, or when using Node.js >= 0.10, this is not
required (the “-i” flag is automatically provided).Common limitationsExcept for Python, only basic types (booleans, numbers, strings, lists/arrays
and maps/dictionaries) can be transferred between the interpreters.Serialization is performed locally usingJSON. Implement a customJSONEncoderto tweak which/how objects are encoded.If an object that cannot be serialized reaches a “call”, “eval”, or even a
non-local return such as anerror or exception, it will generate aSerializationExceptionon the local (Python) side.Strings arealwaysUTF-8 encoded.References are implicitly broken asobjects are transferred by value. This
is obvious, as you’re talking with a separate process, but it can easily be
forgotten due to the blurring of the boundary.Calling functions across the bridge is slow, also in Python, due to the
serialization. But the execution speed of the functions themselves isnot
affected. This might be perfectly reasonable if there are only occasional
calls between languages, and/or the calls themselves take a significant
fraction of time.General/support mailing listIf you are interested in announcements and development discussions aboutbond, you can subscribe to thebond-develmailing list by sending an
empty email to <bond-devel+subscribe@thregr.org>.You can contact the main author directly at <wavexx@thregr.org>, though using
the general list is encouraged.Authors and Copyrightpython-bondcan be found athttp://www.thregr.org/~wavexx/software/python-bond/“python-bond” is distributed under the GNU GPLv2+ license (see COPYING).Copyright(c) 2014-2015 by wave++ “Yuri D’Elia” <wavexx@thregr.org>.python-bond’s GIT repository is publicly accessible at:git://src.thregr.org/python-bondor athttps://github.com/wavexx/python-bondLatest release notesPerformance/documentation tweaks.