Package: achilles
achillesDistributed/parallel computing in modern Python based on themultiprocessing.PoolAPI (map,imap,imap_unordered).What/why is it?The purpose ofachillesis to make distributed/parallel computing as easy as possible by limiting the required configuration, hiding the details (server/node/controller architecture) and exposing a simple interface based on the popularmultiprocessing.PoolAPI.achillesprovides developers with entry-level capabilities for concurrency across a network of machines (see PEP 372 on the intent behind addingmultiprocessingto the standard library ->https://www.python.org/dev/peps/pep-0371/) using a server/node/controller architecture.Theachilles_server,achilles_nodeandachilles_controllerare designed to run cross-platform/cross-architecture. The server/node/controller may be hosted on a single machine (for development) or deployed across heterogeneous resources.achillesis comparable to excellent Python packages likepathos/pyina,Parallel PythonandSCOOP, but different in certain ways:Designed for developers familiar with themultiprocessingmodule in the standard library with simplicity and ease of use in mind.In addition to the blockingmapAPI which requires that developers wait for all computation to be finished before accessing results (common in such packages),imap/imap_unorderedallow developers to process results as they are returned to theachilles_controllerby theachilles_server.achillesallows for composable scalability and novel design patterns as:Iterables including lists, lists of lists and generator functions (as first-class object - generator expressions will not work as generators cannot be serialized bypickle/dill) are accepted as arguments.TIP: Use generator functions together withimaporimap_unorderedto perform distributed computation on arbitrarily large data.Thedillserializer is used to transfer data between the server/node/controller andmultiprocess(fork ofmultiprocessingthat uses thedillserializer instead ofpickle) is used to performPool.mapon theachilles_nodes, so developers are freed from some of the constraints of thepickleserializer.Installpip install achillesQuick StartStart anachilles_serverlistening for connections fromachilles_nodesat a certain endpoint specified as arguments or in an.envfile in theachillespackage's directory.Then simply importmap,imap, and/orimap_unorderedfromachilles_mainand use them dynamically in your own code (under the hood they create and closeachilles_controllers).map,imapandimap_unorderedwill distribute your function to eachachilles_nodeconnected to theachilles_server. Then, theachilles_serverwill distribute arguments to eachachilles_node(load balanced and made into a list of arguments if the arguments' type is not already a list) which will then perform your function on the arguments usingmultiprocess.Pool.map.Eachachilles_nodefinishes its work, returns the results to theachilles_serverand waits to receive another argument. This process is repeated until all of the arguments have been exhausted.runAchillesServer(host=None, port=None, username=None, secret_key=None)-> run on your local machine or on another machine connected to your networkin:fromachilles.lineReceiver.achilles_serverimportrunAchillesServer# host = IP address of the achilles_server# port = port to listen on for connections from achilles_nodes (must be an int)# username, secret_key used for authentication with achilles_controllerrunAchillesServer(host='127.0.0.1',port=9999,username='foo',secret_key='bar')# OR generate an .env file with a default configuration so that# arguments are no longer required to runAchillesServer()# use genConfig() to overwritefromachilles.lineReceiver.achilles_serverimportrunAchillesServer,genConfiggenConfig(host='127.0.0.1',port=9999,username='foo',secret_key='bar')runAchillesServer()out:ALERT: achilles_server initiated at 127.0.0.1:9999
Listening for connections...runAchillesNode(host=None, port=None)-> run on your local machine or on another machine connected to your networkin:fromachilles.lineReceiver.achilles_nodeimportrunAchillesNode# genConfig() is also available in achilles_node, but only expects host and port argumentsrunAchillesNode(host='127.0.0.1',port=9999)out:GREETING: Welcome! There are currently 1 open connections.

Connected to achilles_server running at 127.0.0.1:9999
CLIENT_ID: 0Examples of how to use the 3 most commonly usedmultiprocessing.Poolmethods inachilles:Note:map,imapandimap_unorderedcurrently accept iterables including - but not limited - to lists, lists of lists, and generator functions asachilles_args.Also note:  if there isn't already a.envconfiguration file in theachillespackage directory, must usegenConfig(host, port, username, secret_key)before using or includehost,port,usernameandsecret_keyas arguments when usingmap,imap,imap_unordered.map(func, args, callback=None, chunksize=1, host=None, port=None, username=None, secret_key=None)in:fromachilles.lineReceiver.achilles_mainimportmapdefachilles_function(arg):returnarg**2defachilles_callback(result):returnresult**2if__name__=="__main__":results=map(achilles_function,[1,2,3,4,5,6,7,8,9,10],achilles_callback,chunksize=1)print(results)out:ALERT: Connection to achilles_server at 127.0.0.1:9999 and authentication successful.

[[1, 16, 81, 256, 625, 1296, 2401, 4096], [6561, 10000]]imap(func, args, callback=None, chunksize=1, host=None, port=None, username=None, secret_key=None)in:fromachilles.lineReceiver.achilles_mainimportimapdefachilles_function(arg):returnarg**2defachilles_callback(result):returnresult**2if__name__=="__main__":forresultinimap(achilles_function,[1,2,3,4,5,6,7,8,9,10],achilles_callback,chunksize=1):print(result)out:ALERT: Connection to achilles_server at 127.0.0.1:9999 and authentication successful.

{'ARGS_COUNTER': 0, 'RESULT': [1, 16, 81, 256, 625, 1296, 2401, 4096]}
{'ARGS_COUNTER': 8, 'RESULT': [6561, 10000]}imap_unordered(func, args, callback=None, chunksize=1, host=None, port=None, username=None, secret_key=None)in:fromachilles.lineReceiver.achilles_mainimportimap_unordereddefachilles_function(arg):returnarg**2defachilles_callback(result):returnresult**2if__name__=="__main__":forresultinimap_unordered(achilles_function,[1,2,3,4,5,6,7,8,9,10],achilles_callback,chunksize=1):print(result)out:ALERT: Connection to achilles_server at 127.0.0.1:9999 and authentication successful.

{'ARGS_COUNTER': 8, 'RESULT': [6561, 10000]}
{'ARGS_COUNTER': 0, 'RESULT': [1, 16, 81, 256, 625, 1296, 2401, 4096]}HowachillesworksUnder the hoodTwistedAn event-driven networking engine written in Python and MIT licensed.dilldillextends Pythonâ€™spicklemodule for serializing and de-serializing Python objects to the majority of the built-in Python types.multiprocessmultiprocess is a fork of multiprocessing that usesdillinstead ofpicklefor serialization.multiprocessingis a package for the Python language which supports the spawning of processes using the API of the standard libraryâ€™s threading module.ExamplesSee theexamplesdirectory for tutorials on various use cases, including:Square numbers/run multiple jobs sequentiallyWord count (TO DO)How to kill clusterfromachilles.lineReceiver.achilles_mainimportkillCluster# simply use the killCluster() command and verify your intent at the prompt# killCluster() will search for an .env configuration file in the achilles package's directory# if it does not exist, specify host, port, username and secret_key as arguments# a command is sent to all connected achilles_nodes to stop the Twisted reactor and exit() the process# optionally, you can pass command_verified=True to proceed directly with killing the clusterkillCluster(command_verified=True)Caveats/Things to knowachilles_nodes use all of the CPU cores available on the host machine to performmultiprocess.Pool.map(pool = multiprocess.Pool(multiprocess.cpu_count())).achillesleaves it up to the developer to ensure that the correct packages are installed onachilles_nodes to perform the function distributed by theachilles_serveron behalf of theachilles_controller. Current recommended solution is to SSH into each machine andpip installarequirements.txtfile.All import statements required by the developer's function, arguments and callback must be included in the definition of the function.Theachilles_serveris currently designed to handle one job at a time. For more complicated projects, I highly recommend checking outDask(especiallydask.distributed) and learning more about directed acyclic graphs (DAGs).Fault tolerance: if someachilles_nodedisconnects before returning expected results, the argument will be distributed to anotherachilles_nodefor computation instead of being lost.callback_errorargument has yet to be implemented, so detailed information regarding errors can only be gleaned from the interpreter used to launch theachilles_server,achilles_nodeorachilles_controller. Deploying the server/node/controller on a single machine is recommended for development.achillesperforms load balancing at runtime and assignsachilles_nodes arguments bycpu_count*chunksize.Defaultchunksizeis 1.Increasing thechunksizeis an easy way to speed up computation and reduce the amount of time spent transferring data between the server/node/controller.If your arguments are already lists, thechunksizeargument is not used.Instead, one argument/list will be distributed to the connectedachilles_nodes at a time.If your arguments are load balanced, the results returned are contained in lists of lengthachilles_node's cpu_count*chunksize.map:Final result ofmapis an ordered list of load balanced lists (the final result is not flattened).imap:Results are returned as computation is finished in dictionaries that include the following keys:RESULT: load balanced list of results.ARGS_COUNTER: index of first argument (0-indexed).Results are ordered.The first result will correspond to the next result after the last result in the preceding results packet's list of results.Likely to be slower thanimmap_unordereddue toachilles_controlleryielding ordered results.imap_unordered(see below) yields results as they are received, whileimapyields results as they are received only if the argument'sARGS_COUNTERis expected based on the length of theRESULTlist in the preceding results packet. Otherwise, aresult_bufferis checked for the results packet with the expectedARGS_COUNTERand the current results packet is added to theresult_buffer. If it is not found,achilles_controllerwill not yield results until a results packet with the expectedARGS_COUNTERis received.imap_unordered:Results are returned as computation is finished in dictionaries that include the following keys:RESULT: load balanced list of results.ARGS_COUNTER: index of first argument (0-indexed).Results are not ordered.Results packets are yielded as they are received (after anyachilles_callbackhas been performed on it).Fastest way of consuming results received from theachilles_server.achillesis in the early stages of active development and your suggestions/contributions are kindly welcomed.achillesis written and maintained by Alejandro PeÃ±a. Email me at adpena at gmail dot com.