Package: async-q
AsyncQA simple worker task queue with async supportInstallationpipinstallasync_qFeaturesSubmit asynchronously taskTask routingDistributed async worker processUse Case: Asynchronous Task Queue for I/O-Bound Task ProcessingOverviewIn scenarios where you need to execute I/O-bound tasks asynchronously within a single application, the "async_q" library provides a straightforward solution. This use case illustrates how to create and utilize the library for such purposes.PrerequisitesBefore using the "async_q" library, ensure you have Python 3.8 and Redis (Version 5.0 to current) installed, along with the required dependencies.Setting Up the Application1. Creating the Async Task Queue AppBegin by creating an instance of theAsyncTaskQueueto set up your application. In this example, we will name our app "async_q_app" and configure it to use Redis as the message broker.# main_app.pyfromasync_qimportAsyncTaskQueue,RedisBuilder# Define Async Task Queue Appasync_q_app=AsyncTaskQueue(redis_builder=RedisBuilder(port='6379',))2. Defining the Task FunctionNext, define a task function that will be submitted to the queue for processing. In this example, we have a task function namedmy_task. This function simulates I/O waiting with a specified delay.# my_task.pyimportloggingimportasynciofromasync_qimportsubmit_task# For initializing the appfrommain_appimportasync_q_appasyncdefmy_task(idx,delay=2,*args,**kwargs):# Simulate I/O waitingawaitasyncio.sleep(delay)logging.info(f'{idx}has finished the task. Task ID:{kwargs.get("task_id")}')3. Submitting TasksTo submit tasks for processing, you can use thesubmit_taskfunction. In this example, we submit 20 tasks to be processed by the queue.if__name__=='__main__':foriinrange(20):submit_task(my_task,kwargs={'idx':i,'delay':10})In the previous example, thesubmit_taskfunction also includes aqueueargument with the default value set todefault.  This offers flexibility in case you want to specify a different queue for a particular task, but if no queue is provided, it will use the default value ofdefault.if__name__=='__main__':foriinrange(20):# Submit a task with a default queue valuesubmit_task(my_task,kwargs={'idx':i,'delay':10,'queue':'default'})Running Worker Processes4. Initializing Worker ProcessesIn order to efficiently handle incoming tasks, it's crucial to set up and launch worker processes. The level of concurrency can be precisely determined by employing the-cflag. In the following example, we will initiate five worker processes that target theasync_q_appmodule specified inmain_app.py. By default, these workers will exclusively process tasks from thedefaultqueue.$python-masync_q-amain_app.py:async_q_app-c5However, you also have the flexibility to assign specific workers to process tasks from different queues, as demonstrated below:$python-masync_q-amain_app.py:async_q_app-c5-qmail_psBy making these adjustments, you can tailor the behavior of your worker processes to suit your application's specific requirements.5. Submitting Tasks for ProcessingWith the worker processes running, you can now submit tasks for processing. Use the following command to execute themy_task.pyscript, which submits tasks to the queue.$pythonmy_task.pyResultThe worker processes will asynchronously process the submitted tasks with the specified delays. You can monitor the progress and completion of tasks through log messages generated by themy_taskfunction. The "Async Queue" library is suitable for I/O-bound workloads that benefit from asynchronous processing within a single application.TodoTest and check back compatibility