Package: afar
AfarOne man's magic is another man's engineeringRobert A. HeinleinInstallationafarmay be installed with pip:pipinstallafaror withconda:condainstall-cconda-forgeafarWhat is it?afarallows you to run code on a remoteDaskclusterusing context managers andIPython magics.  For example:importafarfromdask.distributedimportClientclient=Client()withafar.run,remotely:importdask_cudfdf=dask_cudf.read_parquet("s3://...")result=df.sum().compute()Outside the context,resultis aDask Futurewhose data resides on a worker.result.result()is necessary to copy the data locally.By default, only the last assignment is saved.  One can specify which variables to save:withafar.run("one","two"),remotely:one=1two=one+1oneandtwoare now both Futures.  They can be used directly in otherafar.runcontexts:withafar.runasdata,remotely:three=one+twoassertthree.result()==3assertdata["three"].result()==3dataabove is a dictionary of variable names to Futures.  It may be necessary at times to get the data from here.  Alternatively, you may pass a mapping toafar.runto use as the data.run=afar.run(data={"four":4})withrun,remotely:seven=three+fourassertrun.data["seven"].result()==7If you want to automatically gather the data locally (to avoid calling.result()), useafar.getinstead ofafar.run:withafar.get,remotely:five=two+threeassertfive==5Interactivity in JupyterThere are several enhancements when usingafarin Jupyter Notebook or Qt console, JupyterLab, or any IPython-based frontend that supports rich display.The rich repr of the final expression will be displayed if it's not an assignment:withafar.run,remotely:three+seven# displays 10!Printing is captured and displayed locally:withafar.run,remotely:print(three)print(seven,file=sys.stderr)# 3# 7These are done asynchronously usingipywidgets.Magic!First loadafarmagic extension:%load_extafarNow you can useafaras line or cell magic.%%afaris likewith afar.run, remotely:.  It can optionally accept a list of variable names to save:%%afarx,yx=1y=x+1andz=%afarx+yIs this a good idea?I don't know, but it sure is a joy to use 😃 !For motivation, seehttps://github.com/dask/distributed/issues/4003It's natural to be skeptical of unconventional syntax.  And magic.afaris both unconventional and magical, yet it also works well and is surprisinglyfun!  Why not give it a try to see what you think?We're still exploring the usability ofafarand want to hear what you think.  As you're learningafar, please ask yourself questions such as:can we spell anything better?does this offer opportunities?what is surprising?what is lacking?Here's an example of an opportunity:on_gpus=afar.remotely(resources={"GPU":1})withafar.run,on_gpus:...This now works!  Keyword arguments toremotelywill be passed toclient.submit.I don't know about you, but I think this is starting to look and feel kinda nice, and it could probably be even better :)Caveats and GotchasRepeatedly copying dataafarautomatically gets the data it needs--and only the data it needs--from the outer scope
and sends it to the Dask cluster to compute on.  Since we don't know whether local data has been modified
between calls toafar, we serialize and send local variables every time we userunorget.
This is generally fine: it works, it's safe, and is usually fast enough.  However, if you do this
frequently with large-ish data, the performance could suffer, and you may be using
more memory on your local machine than necessary.With Dask, a common pattern is to send data to the cluster withscatterand get aFutureback.  This works:A=np.arange(10**7)A=client.scatter(A)withafar.run,remotely:B=A+1# A and B are now both Futures; their data is on the clusterAnother option is to passdatatorun:run=afar.run(data={"A":np.arange(10**7)})withafar.run,remotely:B=A+1# run.data["A"] and B are now both Futures; their data is on the clusterHere's a nifty trick to use if you're in an IPython notebook: usedata=globals()!run=afar.run(data=globals())A=np.arange(10**7)withrun,remotely:B=A+1# A and B are now both Futures; their data is on the clusterMutating remote dataAs with any Dask workload, one should be careful to not modify remote data that may be reused.Mutating local dataSimilarly, code run remotely isn't able to mutate local variables.  For example:d={}withafar.run,remotely:d['key']='value'# d == {}✨ This code is highly experimental and magical! ✨