Package: remoteobj
remoteobjInteracting with objects across processes.This uses multiprocessing pipes to send messages and operations from the main process to the remote process and send back the return value.Basically this lets do things like call.close()or.pause(), or.readyon an object that was sent to another process. But it also supports more than that!What's included:Proxy: remote proxy for communicating with forked objects. This was the initial purpose of this packageclassBlah:def__init__(self):self.remote=remoteobj.Proxy(self)defsay_hi(self):print('hi from',mp.current_process().name)defrun(self):# remote process functionwithself.remote.listen_():whileTrue:...# do other stuffself.remote.process_requests()obj=Blah()mp.Process(target=obj.run).start()obj.remote.wait_until_listening()obj.remote.say_hi()# hi from Process-1...util.process:multiprocessing.Processbut with fewer hurdles. Makes it more like you're writing normal python. Handles remote exceptions and return/yield values.deffunc(x):yield fromrange(x)raiseValueError('error from{}'.format(mp.current_process().name))try:withremoteobj.util.process(func,4)asp:foriinp.result:print(i)exceptValueErrorase:print('woah!!',e)# prints 0, 1, 2, 3, woah!! error from Process-1Except: context manager for capturing remote exceptions, assigning them to different groups, and sending them back to the main processexc=remoteobj.Except()defremote_func(exc):withexc(raises=False):try:withexc('init'):do_init()withexc('work'):do_work()finally:withexc('cleanup'):do_cleanup()p=mp.Process(target=remote_func,args=(exc,))p.start()...p.join()exc.raise_any('work')# if none, its a noopexc.raise_any('cleanup')exc.raise_any()# will raise the last caught exceptionLocalExcept: the same context manager interface but without the inter-process communicationexc=remoteobj.LocalExcept()withexc(raises=False):try:withexc('init'):do_init()do_other_stuff()withexc('work'):do_work()finally:withexc('cleanup'):do_cleanup()exc.raise_any('work')exc.raise_any('cleanup')exc.raise_any()NOTE:This is still in alpha and the interface is still evolving (though things are starting to stabilize) so be sure to pin a version or stay up to date with changes. Also, let me know if you have thoughts/suggestions/requests! For info on how this works, seehereInstallpipinstallremoteobjFull Example (Proxy)Here we have an object that we want to run in a separate process. We want to be able to get/set the object's state as it's running so we wrap the object in a remoteProxyobject which will run in the background of the remote process and execute the calls that we make in the main process.importtimeimportremoteobjimportmultiprocessingasmpclassObject:'''Remote object to do some counting.'''def__init__(self):self.remote=remoteobj.Proxy(self)count=0on,stop=False,Falsedeftoggle(self,state=True):self.on=statedefrun(self):'''Remote process'''# starts thread to execute requestswithself.remote.listen_(bg=True):whilenotself.stopandself.remote.listening_:ifself.on:self.count+=1time.sleep(0.1)# start processobj=Object()p=mp.Process(target=obj.run)p.start()# make sure we've started up# pass p so we aren't left hanging if the process diesobj.remote.wait_until_listening(p)# turn on, get starting count, count should be increasingobj.remote.toggle(True)x=obj.remote.count.get_()time.sleep(1)assertobj.remote.count.get_()>x# 10 > 1# turn off, count should stay the sameobj.remote.toggle(False)x=obj.remote.count.get_()time.sleep(1)assertobj.remote.count.get_()==x# 10 == 10# turn back on, count should increase againobj.remote.toggle(True)x=obj.remote.count.__# alias for get_()time.sleep(1)assertobj.remote.count.__>x# 20 > 10# set the remote stop attribute to exit and joinobj.remote.stop=True# you can set attrs too?? (ﾟﾛﾟ*)p.join()Basic UsageProxyThis explains the mechanics that are going on in more detail.NOTE:to avoid name conflicts many of the proxy methods use underscore suffixes (i.e.get_()). This also applies toutil.processarguments. See below.importremoteobj# building some arbitrary objectclassIdk(list):something=5another=[]obj=Idk()# creating a remote proxy to interact with# we want to make sure that the proxy gets# sent along with it so we can handle remote# requests.r_obj=obj.remote=remoteobj.Proxy(obj)# ... now send obj to mp.Process and start listener thread ...# then meanwhile back in the *main* process# call a methodr_obj.append(5)# now the remote object has 5 appended to itassertlen(r_obj)==1# getting an attribute returns a proxy so you can chainassertisinstance(r_obj.another.append,remoteobj.Proxy)# calling will automatically resolve a proxyr_obj.another.append(6)# now another == [6]# you can access keys, but we still allow chaining# so they're proxies tooassertisinstance(r_obj[0],remoteobj.Proxy)assertisinstance(r_obj.another[0],remoteobj.Proxy)# to make the request and get the value, doassertremoteobj.get(r_obj[0])==5# or more conciselyassertr_obj[0].__==5# or if you prefer a less witch-y wayassertr_obj[0].get_()==5# you can even set keys and attributes on remote objectsr_obj[0]=6r_obj.something=10# len() is a special case, but in most cases, you can't# just blindly pass a remote object to a function and# expect it to work all the time.assertstr(r_obj)==<Remote<Idkobjectat...>:(?)>assertr_obj.passto(str)==<Idkobjectat...># the first is the local instance, while the second is the# remote instance. To pass an object to a function to execute# on the remote side, use `passto` which will pickle the function# reference.# So you probably wouldn't want to pass an object method like that# because it would pickle and unpickle the object instance which# is most likely **not** what you want.# but I'm not your mom.NOTE:you cannot get/set attributes that begin with an underscore. All underscore attributes reference the proxy object itself.Now on the remote side:defrun(obj):# indicate that we're listeningwithobj.remote:# do whatever nonsense you needvalue=0whileTrue:# do nonsenseforxinobj:value=x*obj.somethingforx,yinzip(obj,obj.another):value-=y/x*obj.somethingtime.sleep(0.4)# this will make the requestsobj.remote.process_requests()# after exiting, listening is set to false,# clients will fail or return their default# immediately because we have notified that# we will not be processing any more requests.# Here's how to run the listener in a background threaddefrun(obj):# start background thread to listen.withobj.remote.listen_():whileobj.remote.listening_:...# don't need to call obj.remote.process_requests()Proxy OperationsThese are the operations that a remote view can handle, which covers the main ways of accessing information from an object. Let me know if there are others that I'm missing.NOTE:Any operation that says it returns aProxycan be chained.
If a remote method returnsself, that will be caught and a remote proxy will be returned to allow for chaining as well.__call__(obj(*a, **kw)): retrieves return value.to return a proxy instead, do either:Proxy(obj, eager_proxy=True)to get all as proxies orobj.method(_proxy=True)for a one-time thing__getitem__(obj[key]): returns proxy__getattr__(obj.attr): returns proxy__setitem__(obj[key] = value): evaluates__setattr__(obj.attr = value): evaluates__delitem__(del obj[key]): evaluates__delattr__(del obj.attr): evaluates__contains__(x in obj): evaluateslen(len(obj)): evaluatespassto(func(obj)): pass object to a functione.g.obj.passto(str)is equivalent tostr(obj)you can also pass args:obj.passto(func, *a, **kw)To resolve a proxy, you can do one of three equivalent styles:remoteobj.get(obj.attr, default=False)- makes it clearer thatobj.attris being sent to the main processobj.attr.get_(default='asdf')- access via chaining - convenient, somewhat clearobj.attr.__- an attempt at a minimalist interface, doesn't handle default value, not super clear. it's the easiest on the eyes once you know what it means, but I agree that the obscurity is a bit of an issue.Enhanced ProcessesSometimes themultiprocessing.Processis a bit lacking in terms of interface so I wrote a lightweight wrapper that:has a cleaner signature -process(func, *a, **kw)=>func(*a, **kw)other arguments, such asname,grouphave a underscore suffix (e.g.name_,group_,daemon_)can be used as a context managerwith process(...):which will join upon exiting.pulls the process name from the function name by defaultdefaults todaemon_=Truewill raise the remote exception (usingremoteobj.Except())sends backreturnandyieldvalues viap.resultdefremote_func(x,y):time.sleep(1)return10withremoteobj.util.process(remote_func,5,2)asp:assertp.name=='remote_func-1'assertp.result==None# called before return...# do some other stuff# now the process has joined# and we can access the return value of the process!assertp.result==10# now...# wait for it....defremote_func(x,y):foriinrange(x,y):yieldiwithremoteobj.util.process(remote_func,5,10)asp:# p.result returns a generator which will yield the values from the# remote sidea_generator=p.resultassertlist(a_generator)==list(range(5,10))Sending Process ExceptionsSending exceptions back from another process is always such a pain because you have to deal with all of the inter-process communication scaffolding, setting up queues, etc. and it can make your code messy.TheExceptclass lets you catch exceptions and add them to different named groups. This is useful if you need to separate out exceptions for setup errors, processing errors, or clean up errors.How it works: Define anExceptobject. In your remote process usecatchas a context manager and any matching exceptions raised in that context will be pickled with its traceback and appended to its queue.# define an exception handlercatch=remoteobj.Except()# or be more specificcatch=remoteobj.Except(ValueError,TypeError)defremote_process(catch):withcatch:raiseValueError('!!!')withcatch('hi'):# named exception contextsraiseTypeError('hi')p=mp.Process(target=remote_process,args=(catch,))p.start()p.join()catch.raise_any('hi')# will raise hi# orcatch.raise_any()# will raise any exception# orcatch.raise_any(None)# will raise any exception in the default contextLocal ExceptionsWe can use the same syntax and context mechanics without the inter-process communication to catch errors locally.# define an exception handlercatch=remoteobj.LocalExcept(raises=True)try:withcatch:raiseValueError('!!!')except:withcatch('hi',raises=False):raiseTypeError('hello')catch.raise_any('hi')catch.raise_any()HowProxyworksWe override basic python operators so that they return an object that represents a chain of operations (Proxy,Viewobjects).Viewobjects represents a chain of operations - this is mainly an internal interface.Proxyobjects represents a chain of operations linked to an object.When we go to resolve a chain of operations, wefirst acquire a lock so that the listening state can't change and no other requests can be made at the same time.check if the remote instance is listeningwe send the set of operations over a pipe and then wait for the value to come out the other sideonce it returns we check the return values, raise any exception, and return.On the remote side, we:poll the connection checking for op requests and once we find one:acquire a write lockevaluate the view on the proxy objecthandle exceptions then place the result and exception in the pipe to be sent backIf there is no listening process, either a default value will be returned (if you provided one viaget_(default=False)) or aRuntimeErrorwill be raised.It is useful to callproxy.wait_until_listening()while the remote process is starting up so that you don't get aRuntimeErrordue to the listener not having started up yet.If a remote object gets called from the same process as the listening process then it will bypass the pipes and evaluate it directly. This means that if you use threads instead of processes, no data will be sent over pipes.AdvancedimportremoteobjclassA:def__init__(self):self.remote=remoteobj.Proxy(self)defasdf(self):return5classB(A):x=0defasdf(self):return6defchain(self):x+=1returnselfobj=B()Accessing super()# call super methodassertobj.remote.asdf()==6assertobj.remote.super.asdf()==6# is equivalent to: super(type(obj), obj).asdf()Remote methods that chainA common pattern is to have a function return self so that you can chain methods together. But that doesn't work when you're sending an object back from another process because it'll get pickled and it'll no longer be the same object.So there is a special-case - if the return value is self, it will mark it as such and on the other end, it will return the base proxy object.# remote chainingassertobj.remote.x.__==0# check start valueassertobj.remote.chain().chain().chain().x.__==3# equivalent to doing this locallyassertobj.x==0assertobj.chain().chain().chain().x==3DeadlocksWhen dealing with concurrent programming, you always have to be concerned about dead-locking your program.One area where deadlocking could be a problem is if a client process starts to request an operation as the listening process starts to clean up.To prevent that, when the listening process is closing, it will either fulfill outstanding requests (default behavior) or refuse them (Proxy(fulfill_final=False)).