Package: gy-multiprocessing
gy-multiprocessingInstallationvia Githubpipinstallgit+https://github.com/guangyu-he/gy-multiprocessingvia PyPIpipinstallgy-multiprocessingUsageinitializing multiprocessing/multithreading instanceadding your tasks into the pool, either using loop or sentencesrunning the instanceNote:the multiprocessing must work in a function or entrance, do not use it barely in the scriptmake sure the code does require multiprocessing/multithreading, wrongly using the multiprocessing may even lose
performanceplease attention to the queue implementation when using multiprocessing, details check the example belowExamplesMulti Processingimporttimeimportgy_multiprocessingasgympdefyour_func(a_string:int,queue):# NEW from v0.3! if multiprocessing is not used, queue is not a necessary parameter# NOTE! you MUST add an argument for queue and use put() method to fetch the returning value if multiprocessing is usedprint(a_string)ifa_string%5==0:time.sleep(2)# NEW from v0.3! if multiprocessing is not used, all queue relevant codes can still be kept# NEW from v0.3! and the if queue is not None: is not necessary anymore# NOTE! if you are missing this method, there will be None result returned for current process if multiprocessing is usedqueue.put(a_string)returna_stringif__name__=='__main__':# the multiprocessing must work in a function or entrance# do not use it barely"""# initializing the multiprocessing instance# the default max_process are your cpu max cores# max_process could be infinite, but performance will get suffered when the hardware is overloaded"""# NEW from v0.3! set max_process to 0 will disable multiprocessingmp=gymp.MultiProcess(max_process=8)# example for multiprocessing in the loopouter_loop_times=5# NEW from v0.3! it is possible to test beforehand if the function and args is suitable for multiprocessingtest_your_func:bool=mp.test(your_func,(1,))forcurrent_loop_indexinrange(outer_loop_times):# your running arguments, must be tupleargs=(current_loop_index,)"""# adding tasks in multiprocessing pool"""mp.add(your_func,args)# it is also possible to add task outside the loopmp.add(your_func,(10,))"""# running tasks in multiprocessing pool (returned values are optional)"""result=mp.run()print(result)Multi Threadsimportgy_multiprocessingasgympimporttimedefyour_func(a_string):# your single task functionprint(a_string)returna_string+"!"if__name__=='__main__':# the multithreading must work in a function or entrance# do not use it barely# timing (optional)start=time.time()"""# initializing the multithreading instance# the default max_threads are your cpu max cores number - 1# max_threads can not larger than your cpu max core number"""mt=gymp.MultiThread(max_threads=4)# example for multithreading in the loopouter_loop_times=5forcurrent_loop_indexinrange(outer_loop_times):args=(str(current_loop_index),)"""# adding tasks in multi threading pool"""mt.add(your_func,args)# it is also possible to work without loopargs=(str(1),)mt.add(your_func,args)args=(str(2),)mt.add(your_func,args)"""# running tasks in multithreading pool (returned values are optional)"""results=mt.run()print(results)# timing (optional)end=time.time()-startprint("done in{}s".format("%.2f"%end))Combined StructureNote: you can not use multiprocessing or sub-multithreading in the multithreading methodIf you want to use such structure, based on your needs, considering using sub-multiprocessing or multithreading in
multiprocessing structure.importgy_multiprocessingasgympdefyour_sub_func(b_string:int,queue=None):# your function that needs to multithreading/multiprocessingb_string+=1ifqueueisnotNone:queue.put(b_string)returnb_stringdefyour_mt_func(a_string:int,queue):# multithreading in multiprocessing structuremt=gymp.MultiThread()forcurrent_loop_indexinrange(a_string):# your running arguments, must be tupleargs=(current_loop_index,)mt.add(your_sub_func,args)result=mt.run()# Do not forget queue!queue.put(result)defyour_mp_func(a_string:int,queue):# sub-multiprocessing in multiprocessing structuresmp=gymp.MultiProcess()forcurrent_loop_indexinrange(a_string):# your running arguments, must be tupleargs=(current_loop_index,)smp.add(your_sub_func,args)result=smp.run()# Do not forget queue!queue.put(result)if__name__=='__main__':mp=gymp.MultiProcess()outer_loop_times=10forcurrent_loop_indexinrange(outer_loop_times):args=(current_loop_index,)mp.add(your_mt_func,args)print(mp.run())print("\n-----”\n")mp=gymp.MultiProcess()forcurrent_loop_indexinrange(outer_loop_times):args=(current_loop_index,)mp.add(your_mp_func,args)print(mp.run())Updates Logv0.2.3bug fixfixed an issue casing not adding new process to pool until all processed are done in current poolv0.2.4featurea silent mode is added to multiprocessing which is possible not showing messages in consolesolved multiprocessing not going to the end when there are internal error exceptions in input functionsolved multiprocessing not going to the end when there are missing queue.put() method in input functionv0.2.4.1 & v0.2.4.2bug fixfixed an issue from readme long desc causing installation errorimprovementsimplified duplicated codesv0.2.4.3updatespackage import improvement.PLEASE UPDATE IMPORT AS EXAMPLES ABOVEcheck a boolean type input argumentv0.3featureadded a test method to check if the function and args is suitable for multiprocessingadded a max_process=0 possibility to disable multiprocessingqueue parameter in user function has a more flexibility to be used among multiprocessing mode on and offa new multiprocessing pool logic to check if the subprocess is doneSupportfeel free to check source code inGitHub, you are welcome
to leave any comments.2022©Guangyu He, for further support please contact author.Email:me@heguangyu.net