Package: pandas-parallel-apply
pandas-parallel-applyParallel wrappers fordf.apply(fn),df[col].apply(fn),series.apply(fn), anddf.groupby([cols]).apply(fn), with tqdm progress bars included.Installationpip install pandas-parallel-applyImport with:frompandas_parallel_applyimportDataFrameParallel,SeriesParallelExamplesSeeexamples/for usage on some dummy dataframes and series.Usage# Apply on each row of a dataframedf.apply(fn)# ->DataFrameParallel(df,n_cores:int=None,pbar:bool=True).apply(fn)# Apply on a column of a dataframe (returns a Series)df[col].apply(fn,axis=1)# ->DataFrameParallel(df,n_cores:int=None,pbar:bool=True)[col].apply(fn,axis=1)# Apply on a seriesseries.apply(fn)# ->SeriesParallel(series,n_cores:int=None,pbar:bool=True).apply(fn)# GroupBy applydf.groupby([cols]).apply(fn)# ->DataFrameParallel(df,n_cores:int=None,pbar:bool=True).groupby([cols]).apply(fn)How it worksIt takes the length of your dataframe (or series, or grouper) = N and then_coresprovided to the constructors (K).
It then splits the dataframe in K chunks of N/K size and spawns K new processes, each processing the desired chunks.Only row-wise (perfect parallelable) operations are supported, sodf.apply(fn, axis=1)is okay, butdf.apply(fn, axis=0)is not because it may require rows that are on other workers.It is assumed that each row is processed in similar time, so the N/K chunks will finishe more or less at the same time.Future ImprovementNot supported but may be interesting: define also a number of chunks (C>K), so the df is actually split in N/C chunks,
and theses are passed using a round-robin approach to the K processes. Right now, C=K, so whenever one process
finishes, it will not be assigned any more work.n_cores semanticsn_cores < -1-> throws an errorn_cores == -1-> usescpu_count()- 1 coresn_cores == 0-> uses serial/standard pandas functionsn_cores == 1-> spawns a single process alongside the main onen_cores > 1-> spanws N processes and chunks the dfn_cores > cpu_cpunt()-> throws an warningn_cores > len(df)-> limits tolen(df)On CPU-bound tasks (calculations),n_cores = -1is likely to be fastest. On network-bound operations (e.g., where
threads may invoke network calls), using a very highn_coresvalue may be beneficial.DisclaimersThis is an experimental repository. It may lead to unexpected behaviour.Not all the merging semantics of pandas are supported. Pandas has weird and complex methods of converting an apply
return. For example, a series apply function may return a dataframe, a series, a dict, a list, etc. All of these are
converted in some specific way. Some cases may not be supported.Groupby apply functions aremuchslower than their serial variant currently. Still experimenting with how to make
it faster. It looks correct, just 10-100x slower for some small examples. May be better as dataframe get bigger.Usingn_cores = 1will create a multiprocessing pool of just 1 core, so the code is parallel (thus not running on
the main process), but may not yield much speed improvement, except for not blocking the main process. May be useful
in some GUI apps.You can useparallelism=multithreadofparallelism=multiprocess(2nd is default) for all constructors. Using
multiprocess requires the functions to be picklable though (lambda for example, must be global)That's all.