Package: tqueue
tqueue packageThis library allows you to do your tasks in multiple threads easily.This is helpful when you have a lot of data to process.Assume that you have a large list of items to process. You need to write a producer to put items in the queue one by one.Workers will get data from the queue and then process it. Putting data into a queue should be quicker than processing it (worker).InstallationpipinstalltqueueUsageImport libraryfromtqueueimportThreadingQueueCreate a workerCreate a worker function that gets the data as the first parameterWorker can be a normal function or a coroutine functionWorker will be called in child threadsdefworker(data):passasyncdefworker2(data):passSet threading for a producer
Apply the threading for a producer:a. Set the number of threads and the workerb. Put data into the queueYou can also use ThreadingQueue as a context managerdefproducer():# Start the queuewithThreadingQueue(40,worker)astq:...tq.put(data)You can also use it asyncasyncdefproducer():# Start the queueasyncwithThreadingQueue(40,worker)astq:...awaittq.put(data)Run producerAsync producer:awaitproducer()orasyncio.run(producer())NoteYou can add more keyword params for all workers running in threads viaworker_paramsApart from the number of threads and the worker, you can setlog_dirto store logs to fileandworker_params_builderto generate parameters for each worker.on_thread_closeis an optional param as a function that is helpful when you need to close the database connection when a thread doneApart from all the above params, the rest of the keyword params will be passed to the worker.If you change the lib from the 0.0.14 version to the newer, please update the code to fix the bug:# 0.0.14withThreadingQueue(num_of_threads,worker)astq:...awaittq.put(data)# From 0.0.15# SyncwithThreadingQueue(num_of_threads,worker)astq:...tq.put(data)# AsyncasyncwithThreadingQueue(num_of_threads,worker)astq:...awaittq.put(data)In both sync and async cases, you can provide a worker as an async function.The async version is a little bit better in performance because it usesasyncio.sleepto wait when the queue is full compared totime.sleepin the sync version. In most cases, the difference in performance is not much.ExampleimportjsonimportpymysqlimportasynciofromtqueueimportThreadingQueueNUM_OF_THREADS=40defget_db_connection():returnpymysql.connect(host='localhost',user='root',password='123456',database='example',cursorclass=pymysql.cursors.DictCursor)# Build params for the worker, the params will be persistent with the thread# This function is called when init a new thread or retrydefworker_params_builder():# Threads use db connection separatelyconn=get_db_connection()conn.autocommit(1)cursor=conn.cursor()return{"cursor":cursor,"connection":conn}# To clear resources: close database connection, ...# This function is called when the thread endsdefon_close_thread(cursor,connection):cursor.close()connection.close()defworker(image_info,cursor,uid:int,**kwargs):# Update image info into databasesql="UPDATE images SET width =%s, height =%s, uid =%sWHERE id =%s"cursor.execute(sql,(image_info["width"],image_info["height"],uid,image_info["id"]))defproducer(source_file:str):withThreadingQueue(NUM_OF_THREADS,worker,log_dir=f"logs/update-images",worker_params_builder=worker_params_builder,on_close_thread=on_close_thread,params={"uid":123},retry_count=1)astq:withopen(source_file,'r')asf:forlineinf:ifnotline:continuedata=json.loads(line)tq.put(data)if__name__=="__main__":producer("images.jsonl")DevelopmentBuild projectUpdate the version number in filesrc/tqueue/__version__.pyUpdate the Change logBuild and publish the changespython3-mbuild
python3-mtwineuploaddist/*Release InformationFixedNo exception when log to file anymoreFull changelog