Package: processq
processq packageThis library allows you to do your tasks in multiple processes easily.This is helpful when you have a lot of data to process.Assume that you have a large list of items to process. You need to write a producer to put items in the queue one by one.Workers will get data from the queue and then process it. Putting data into a queue should be quicker than processing it (worker).InstallationpipinstallprocessqUsageImport libraryfromprocessqimportProcessQueueCreate a workerCreate a worker function that gets the data as the first parameterWorker can be a normal function or a coroutine functionWorker will be called in child processesdefworker(data):passasyncdefworker2(data):passSet process for a producer
Apply the process for a producer:a. Set the number of processes and the workerb. Put data into the queueYou can also use ProcessQueue as a context managerdefproducer():# Start the queuewithProcessQueue(40,worker)aspq:...pq.put(data)You can also use it asyncasyncdefproducer():# Start the queueasyncwithProcessQueue(40,worker)aspq:...awaitpq.put(data)Run producerAsync producer:awaitproducer()orasyncio.run(producer())NoteYou can add more keyword params for all workers running in processes viaworker_paramsApart from the number of processes and the worker, you can setlog_dirto store logs to fileandworker_params_builderto generate parameters for each worker.on_process_closeis an optional param as a function that is helpful when you need to close the database connection when a process doneApart from all the above params, the rest of the keyword params will be passed to the worker.If you change the lib from the 0.0.14 version to the newer, please update the code to fix the bug:# 0.0.14withProcessQueue(num_of_processes,worker)aspq:...awaitpq.put(data)# From 0.0.15# SyncwithProcessQueue(num_of_processes,worker)aspq:...pq.put(data)# AsyncasyncwithProcessQueue(num_of_processes,worker)aspq:...awaitpq.put(data)In both sync and async cases, you can provide a worker as an async function.The async version is a little bit better in performance because it usesasyncio.sleepto wait when the queue is full compared totime.sleepin the sync version. In most cases, the difference in performance is not much.ExampleimportjsonimportpymysqlimportasynciofromprocessqimportProcessQueueNUM_OF_PROCESSES=40defget_db_connection():returnpymysql.connect(host='localhost',user='root',password='123456',database='example',cursorclass=pymysql.cursors.DictCursor)# Build params for the worker, the params will be persistent with the process# This function is called when init a new process or retrydefworker_params_builder():# Processes use db connection separatelyconn=get_db_connection()conn.autocommit(1)cursor=conn.cursor()return{"cursor":cursor,"connection":conn}# To clear resources: close database connection, ...# This function is called when the process endsdefon_close_process(cursor,connection):cursor.close()connection.close()defworker(image_info,cursor,uid:int,**kwargs):# Update image info into databasesql="UPDATE images SET width =%s, height =%s, uid =%sWHERE id =%s"cursor.execute(sql,(image_info["width"],image_info["height"],uid,image_info["id"]))defproducer(source_file:str):withProcessQueue(NUM_OF_PROCESSES,worker,log_dir=f"logs/update-images",worker_params_builder=worker_params_builder,on_close_process=on_close_process,params={"uid":123},retry_count=1)aspq:withopen(source_file,'r')asf:forlineinf:ifnotline:continuedata=json.loads(line)pq.put(data)if__name__=="__main__":producer("images.jsonl")DevelopmentBuild projectUpdate the version number in filesrc/processq/__version__.pyUpdate the Change logBuild and publish the changespython3-mbuild
python3-mtwineuploaddist/*Release InformationAddedTodoFull changelog