Package: patio
PATIOPATIO is an acronym forPythonAsynchronousTasks for AsyncIO.MotivationI wanted to create an easily extensible library, for distributed task execution,
likecelery, only targeting asyncio as the main
design approach.By design, the library should be suitable for small projects and the really
large distributed projects. The general idea is that the user simply splits
the projects code base into functions of two roles – background tasks and triggers of these background tasks.
Also, this should help your project to scale horizontally. It allows to make
workers or callers available across the network using embedded TCP, or using
plugins to communicate through the existing messaging infrastructure.QuickstartThe minimal example, which executes tasks in a thread pool:importasynciofromfunctoolsimportreducefrompatioimportRegistryfrompatio.brokerimportMemoryBrokerfrompatio.executorimportThreadPoolExecutorrpc=Registry()@rpc("mul")defmultiply(*args:int)->int:returnreduce(lambdax,y:x*y,args)asyncdefmain():asyncwithThreadPoolExecutor(rpc,max_workers=4)asexecutor:asyncwithMemoryBroker(executor)asbroker:print(awaitasyncio.gather(*[broker.call("mul",1,2,3)for_inrange(100)]))if__name__=='__main__':asyncio.run(main())TheThreadPoolExecutorin this example is the entity that will execute the
tasks. If the tasks in your project are asynchronous, you can selectAsyncExecutor, and then the code will look like this:importasynciofromfunctoolsimportreducefrompatioimportRegistryfrompatio.brokerimportMemoryBrokerfrompatio.executorimportAsyncExecutorrpc=Registry()@rpc("mul")asyncdefmultiply(*args:int)->int:# do something asynchronouslyawaitasyncio.sleep(0)returnreduce(lambdax,y:x*y,args)asyncdefmain():asyncwithAsyncExecutor(rpc,max_workers=4)asexecutor:asyncwithMemoryBroker(executor)asbroker:print(awaitasyncio.gather(*[broker.call("mul",1,2,3)for_inrange(100)]))if__name__=='__main__':asyncio.run(main())These examples may seem complicated, but don't worry, the next section details
the general concepts and hopefully a lot will become clear to you.The main conceptsThe main idea in developing this library was to create a maximally
modular and extensible system that can be expanded with third-party
integrations or directly in the user's code.The basic elements from which everything is built are:Registry- Key-Value like store for the functionsExecutor- The thing which execute functions from the registryBroker- The actor who distributes tasks in your distributed
(or local) system.RegistryThis is a container of functions for their subsequent execution.
You can register a function by specific name or without it,
in which case the function is assigned a unique name that depends on
the source code of the function.This registry does not necessarily have to match on the calling and called
sides, but for functions that you register without a name it is must be,
and then you should not need to pass the function name but the function
itself when you will call it.An instance of the registry must be transferred to the broker,
the first broker in the process of setting up will block the registry
to write, that is, registering new functions will be impossible.An optionalprojectparameter, this is essentially like a namespace
that will help avoid clash functions in different projects with the same
name. It is recommended to specify it and the broker should also use this
parameter, so it should be the same value within the same project.You can either manually register elements or use a
registry instance as a decorator:frompatioimportRegistryrpc=Registry(project="example")# Will be registered with auto generated name@rpcdefmul(a,b):returna*b@rpc('div')defdiv(a,b):returna/bdefpow(a,b):returna**bdefsub(a,b):returna-b# Register with auto generated namerpc.register(pow)rpc.register(sub,"sub")Alternatively usingregistermethod:frompatioimportRegistryrpc=Registry(project="example")defpow(a,b):returna**bdefsub(a,b):returna-b# Register with auto generated namerpc.register(pow)rpc.register(sub,"sub")Finally, you can register functions explicitly, as if it were
just a dictionary:frompatioimportRegistryrpc=Registry(project="example")defmul(a,b):returna*brpc['mul']=mulExecutorAn Executor is an entity that executes local functions from registry.
The following executors are implemented in the package:AsyncExecutor- Implements pool of asynchronous tasksThreadPoolExecutor- Implements pool of threadsProcessPoolExecutor- Implements pool of processesNullExecutor- Implements nothing and exists just for forbid execute
anything explicitly.Its role is to reliably execute jobs without taking too much so as not to
cause a denial of service, or excessive memory consumption.The executor instance is passing to the broker, it's usually applies
it to the whole registry. Therefore, you should understand what functions
the registry must contain to choose kind of an executor.BrokerThe basic approach for distributing tasks is to shift the responsibility
for the distribution to the user implementation. In this way, task
distribution can be implemented through third-party brokers, databases,
or something else.This package is implemented by the following brokers:MemoryBroker- To distribute tasks within a single process.
A very simple implementation, in case you don't know yet how
your application will develop and just want to leave the decision
of which broker to use for later, while laying the foundation for
switching to another broker.TCPBroker- Simple implementation of the broker just using TCP,
both Server and Client mode is supported for both the task executor
and the task provider.MemoryBrokerIt's a good start if you don't need to assign tasks in a distributed
fashion right now.In fact, it's a simple way to run tasks in the executor from the other
places in your project.TCPBrokerIt allows you to make your tasks distributed without resorting to external
message brokers.The basic idea of TCP broker implementation is that in terms of
performing tasks, there is no difference between them, it is
just a way to establish a connection, both the server and the
client can be the one who performs tasks and the one who sets
them, and it is also possible in a mixed mode.In other words, deciding who will be the server and who will be the
client in your system is just a way to connect and find each other
in your distributed system.Here are the ways of organizing communication between the
server and the clients.Server centric scheme exampleThis diagram describes a simple example, if there is one server
and one client exchanging messages via TCP.One client multiple servers exampleThis is an example of how a client establishes connections to a set server.Full mesh exampleFull mesh scheme, all clients are connected to all servers.AuthorizationAuthorization takes place at the start of the connection,
for this the parameterkey=(b''is by default) must contain the same keys
for client and server.It is important to understand that this is not 100% protection against
attacks like MITM etc.This approach should only be used if the client and server are on a trusted
network. In order to secure traffic as it traverses the Internet, thessl_context=parameter should be prepended to both the server and the client
(see example bellow).ExamplesThe examples below will hopefully help you figure this out.Server executing tasksfromfunctoolsimportreduceimportasynciofrompatioimportRegistryfrompatio.broker.tcpimportTCPServerBrokerfrompatio.executorimportThreadPoolExecutorrpc=Registry(project="test",auto_naming=False)defmul(*args):returnreduce(lambdax,y:x*y,args)asyncdefmain():rpc.register(mul,"mul")asyncwithThreadPoolExecutor(rpc)asexecutor:asyncwithTCPServerBroker(executor)asbroker:# Start IPv4 serverawaitbroker.listen(address='127.0.0.1')# Start IPv6 serverawaitbroker.listen(address='::1',port=12345)awaitbroker.join()if__name__=="__main__":asyncio.run(main())Client calling tasks remotelyimportasynciofrompatioimportRegistryfrompatio.broker.tcpimportTCPClientBrokerfrompatio.executorimportNullExecutorrpc=Registry(project="test",auto_naming=False)asyncdefmain():asyncwithNullExecutor(rpc)asexecutor:asyncwithTCPClientBroker(executor)asbroker:# Connect to the IPv4 addressawaitbroker.connect(address='127.0.0.1')# Connect to the IPv6 address (optional)awaitbroker.connect(address='::1',port=12345)print(awaitasyncio.gather(*[broker.call('mul',i,i)foriinrange(10)]),)if__name__=="__main__":asyncio.run(main())Examples with SSLThe task comes down to passing the ssl context to the server and to the client.Below you will see an example of how to make a couple of self-signed
certificates, and an authorization CA. Original posthere.This is just an example, and if you want to use your own certificates, just
create an ssl context as required by your security policy.Certificate authority creationAttention:this is the key used to sign the certificate requests, anyone
holding this can sign certificates on your behalf. So keep it in a safe place!opensslreq-x509\-sha256-days3650\-nodes\-newkeyrsa:2048\-subj"/CN=Patio Example CA/C=CC/L=West Island"\-keyoutCA.key-outCA.pemServer certificate creationFirst create server private key:opensslgenrsa-outserver.key2048Then create the certificate request signing this key:opensslreq\-new-sha256\-keyserver.key\-subj"/CN=server.example.net/C=CC/L=West Island"\-outserver.csrSign this request by CA:opensslx509-req\-days365-sha256\-inserver.csr\-CACA.pem\-CAkeyCA.key\-CAcreateserial\-outserver.pemThis should be enough to encrypt the traffic.Server with SSL executing tasksfromfunctoolsimportreduceimportasyncioimportsslfrompatioimportRegistryfrompatio.broker.tcpimportTCPServerBrokerfrompatio.executorimportThreadPoolExecutorrpc=Registry(project="test",auto_naming=False)defmul(*args):returnreduce(lambdax,y:x*y,args)asyncdefmain():rpc.register(mul,"mul")ssl_context=ssl.SSLContext()ssl_context.load_verify_locations("path/to/CA.pem")ssl_context.load_cert_chain("path/to/server.pem","path/to/server.key")asyncwithThreadPoolExecutor(rpc)asexecutor:asyncwithTCPServerBroker(executor,ssl_context=ssl_context)asbroker:# Start IPv4 serverawaitbroker.listen(address='127.0.0.1')# Start IPv6 serverawaitbroker.listen(address='::1',port=12345)awaitbroker.join()if__name__=="__main__":asyncio.run(main())Client calling tasks remotelyimportasyncioimportsslfrompatioimportRegistryfrompatio.broker.tcpimportTCPClientBrokerfrompatio.executorimportNullExecutorrpc=Registry(project="test",auto_naming=False)asyncdefmain():ssl_context=ssl.create_default_context(cafile="path/to/CA.pem")asyncwithNullExecutor(rpc)asexecutor:asyncwithTCPClientBroker(executor,ssl_context=ssl_context)asbroker:# Connect to the IPv4 addressawaitbroker.connect(address='127.0.0.1')# Connect to the IPv6 address (optional)awaitbroker.connect(address='::1',port=12345)print(awaitasyncio.gather(*[broker.call('mul',i,i)foriinrange(10)]),)if__name__=="__main__":asyncio.run(main())