Package: classwork
ClassworkThis is a simple Python module to help distribute you tasks across multiple brokers as microservices.Classwork usesNATSto manage communication between job schedulers and workers. Nats does most of the heavy lifting, allowing us to keep the rest of the code simple and concise.What is in name?So why ClassWork?
Well, this is simply because workers are simply"python classes"(The Class) whose"methods"become individual workers (The Students).Get startedFirst let us create a simple workerimportasynciofromclassworkimportClassWorkclassWorkerClass:# Your worker...# Notice that we expect workers to be asyncasyncdefadd(self,a,b):# the work!resp=a+b# simulate a long processawaitasyncio.sleep(3)# return resultsreturnresp# this is our nats server urlnats_url="nats://127.0.0.1:4222"# initialize ClassWorkclass_work=ClassWork(nats_url=nats_url)# init worker classworker=WorkerClass()# Ok, let us register things with asyncio# notice the method 'class_work.register' is async!asyncio.run(class_work.register(name="my-worker",worker_class=worker))This is all the code we need to set up the worker.
It is important to note the following:Worker Class methods should be asynchronousclass_work.register is also asyncYou will need NATS running withJetStreamenabled!The Job SchedulerNow we need to create the job scheduler. Below is the code we needimportasyncioimportnumpyasnpfrompprintimportpprintfromclassworkimportClassWork# init ClassWorknats_url="nats://127.0.0.1:4222"class_work=ClassWork(nats_url=nats_url)# Our callback function# This is where complete work gets reportedasyncdefreport_callback(report_card):print("We have a report!")pprint(report_card)# We need this function to create our job schedulesasyncdefschedules():# Assign a jobawaitclass_work.assign(# the task nametask="my_worker.add",# argumentsargs=[1,2],# the callback to report our resultsreport_callback=report_callback,)# Ok, let us create the schedules nowasyncio.run(schedules())This code will create ataskinto NATS and the job workers (attentive students 😂) already listening will pick the task and run it, then publish their reports which is routed via NATS back to the scheduler (teacher?).Take note of the following:class_work.assignmust be run in async mode. So we have wrapped it in an async method. You can also useasyncio.rundirectly.Naming your task is very important. This naming convection is unashamedly borrowed frommoleculer. In this case, your task is"my_worker.add". This will route to any worker class registered with thename"my_worker" and method "add".Because all this traffic is routed via NATS, your arguments must be JSON serializable. Even though we usetypto handle edge cases likesets, beware that there are limits to what you can pass in your argumentsreport_callbackmust be async. It is called with a 'report' of your task. A report card 😊 will look like the one below:argscan be passed as a list or dict. They will be treated as*argsif list and**kwargsif dict.We have a report!
{'task': 'my_worker.add',
 'duration': {'latency': {'request': '4 ms and 356 µs',
                          'response': '5 s and 4 ms'},
              'my_worker.add': '3 s and 1 ms'},
 'req_id': '8mYjJjM0kb5',
 'response': 3}Report Explanationduration:is a high precision (down to yoctoseconds) report of the time taken.latency:shows the time taken to route your "taskrequest" to the worker and "taskresponse" back to the scheduler. It is important to understand that since both worker and scheduler are disconnected, latency may also include delays of either to access the NATS network and thus does not specifically refer to network latency.req_id:is a unique id assigned to each jobresponse:is the actual value returned by the workerTry itWe have sample code inscheduler.pyandworker.py