Package: parallel-pandas
Parallel-pandasMakes it easy to parallelize your calculations in pandas on all your CPUs.Installationpipinstall--upgradeparallel-pandasQuickstartimportpandasaspdimportnumpyasnpfromparallel_pandasimportParallelPandas#initialize parallel-pandasParallelPandas.initialize(n_cpu=16,split_factor=4,disable_pr_bar=True)# create big DataFramedf=pd.DataFrame(np.random.random((1_000_000,100)))# calculate multiple quantiles. Pandas only uses one core of CPU%%timeitres=df.quantile(q=[.25,.5,.95],axis=1)3.66 s ± 31.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)#p_quantile is parallel analogue of quantile methods. Can use all cores of your CPU.%%timeitres=df.p_quantile(q=[.25,.5,.95],axis=1)679 ms ± 10.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)As you can see thep_quantilemethod is5 times faster!UsageUnder the hood,parallel-pandasworks very simply. The Dataframe or Series is split into chunks along the first or second axis. Then these chunks are passed to a pool of processes or threads where the desired method is executed on each part. At the end, the parts are concatenated to get the final result.When initializing parallel-pandas you can specify the following options:n_cpu- the number of cores of your CPU that you want to use (defaultNone- use all cores of CPU)split_factor- Affects the number of chunks into which the DataFrame/Series is split according to the formulachunks_number = split_factor*n_cpu(default 1).show_vmem- Shows a progress bar with available RAM (defaultFalse)disable_pr_bar- Disable the progress bar for parallel tasks (defaultFalse)For exampleimportpandasaspdimportnumpyasnpfromparallel_pandasimportParallelPandas#initialize parallel-pandasParallelPandas.initialize(n_cpu=16,split_factor=4,disable_pr_bar=False)# create big DataFramedf=pd.DataFrame(np.random.random((1_000_000,100)))During initialization, we specifiedsplit_factor=4andn_cpu = 16, so the DataFrame will be split into 64 chunks (in the case of thedescribemethod, axis = 1) and the progress bar shows the progress for each chunkYou can parallelize any expression with pandas Dataframe. For example, let's do a z-score normalization of columns in a dataframe. Let's look at the execution time and memory consumption. Compare with synchronous execution and with Dask.DataFrameimportpandasaspdimportnumpyasnpfromparallel_pandasimportParallelPandasimportdask.dataframeasddfromtimeimportmonotonic#initialize parallel-pandasParallelPandas.initialize(n_cpu=16,split_factor=8,disable_pr_bar=True)# create big DataFramedf=pd.DataFrame(np.random.random((1_000_000,1000)))# create dask DataFrameddf=dd.from_pandas(df,npartitions=128)start=monotonic()res=(df-df.mean())/df.std()print(f'synchronous z-score normalization time took:{monotonic()-start:.1f}s.')synchronousz-scorenormalizationtimetook:21.7s.#parallel-pandasstart=monotonic()res=(df-df.p_mean())/df.p_std()print(f'parallel z-score normalization time took:{monotonic()-start:.1f}s.')parallelz-scorenormalizationtimetook:11.7s.#dask dataframestart=monotonic()res=((ddf-ddf.mean())/ddf.std()).compute()print(f'dask parallel z-score normalization time took:{monotonic()-start:.1f}s.')daskparallelz-scorenormalizationtimetook:12.5s.Pay attention to memory consumption.parallel-pandasanddaskuse almost half as much RAM aspandasFor some methodsparallel-pandasis faster thandask.DataFrame:#dask%%timeres=ddf.nunique().compute()Walltime:42.9s%%timeres=ddf.rolling(10).mean().compute()Walltime:19.1s#parallel-pandas%%timeres=df.p_nunique()Walltime:12.9s%%timeres=df.rolling(10).p_mean()Walltime:12.5sAPIParallel counterparts for pandas Series methodsmethodsparallel analogueexecutorpd.Series.apply()pd.Series.p_apply()threads / processespd.Series.map()pd.Series.p_map()threads / processesParallel counterparts for pandas SeriesGroupBy methodsmethodsparallel analogueexecutorpd.SeriesGroupBy.apply()pd.SeriesGroupBy.p_apply()threads / processesParallel counterparts for pandas Dataframe methodsmethodsparallel analogueexecutordf.mean()df.p_mean()threadsdf.min()df.p_min()threadsdf.max()df.p_max()threadsdf.median()df.p_max()threadsdf.kurt()df.p_kurt()threadsdf.skew()df.p_skew()threadsdf.sum()df.p_sum()threadsdf.prod()df.p_prod()threadsdf.var()df.p_var()threadsdf.sem()df.p_sem()threadsdf.std()df.p_std()threadsdf.cummin()df.p_cummin()threadsdf.cumsum()df.p_cumsum()threadsdf.cummax()df.p_cummax()threadsdf.cumprod()df.p_cumprod()threadsdf.apply()df.p_apply()threads / processesdf.applymap()df.p_applymap()processesdf.replace()df.p_replace()threadsdf.describe()df.p_describe()threadsdf.nunique()df.p_nunique()threads / processesdf.mad()df.p_mad()threadsdf.idxmin()df.p_idxmin()threadsdf.idxmax()df.p_idxmax()threadsdf.rank()df.p_rank()threadsdf.mode()df.p_mode()threads/processesdf.agg()df.p_agg()threads/processesdf.aggregate()df.p_aggregate()threads/processesdf.quantile()df.p_quantile()threads/processesdf.corr()df.p_corr()threads/processesParallel counterparts for pandas DataframeGroupBy methodsmethodsparallel analogueexecutorDataFrameGroupBy.apply()DataFrameGroupBy.p_apply()threads / processesParallel counterparts for pandas window methodsRollingmethodsparallel analogueexecutorpd.core.window.Rolling.apply()pd.core.window.Rolling.p_apply()threads / processespd.core.window.Rolling.min()pd.core.window.Rolling.p_min()threads / processespd.core.window.Rolling.max()pd.core.window.Rolling.p_max()threads / processespd.core.window.Rolling.mean()pd.core.window.Rolling.p_mean()threads / processespd.core.window.Rolling.sum()pd.core.window.Rolling.p_sum()threads / processespd.core.window.Rolling.var()pd.core.window.Rolling.p_var()threads / processespd.core.window.Rolling.sem()pd.core.window.Rolling.p_sem()threads / processespd.core.window.Rolling.skew()pd.core.window.Rolling.p_skew()threads / processespd.core.window.Rolling.kurt()pd.core.window.Rolling.p_kurt()threads / processespd.core.window.Rolling.median()pd.core.window.Rolling.p_median()threads / processespd.core.window.Rolling.quantile()pd.core.window.Rolling.p_quantile()threads / processespd.core.window.Rolling.rank()pd.core.window.Rolling.p_rank()threads / processespd.core.window.Rolling.agg()pd.core.window.Rolling.p_agg()threads / processespd.core.window.Rolling.aggregate()pd.core.window.Rolling.p_aggregate()threads / processesWindowmethodsparallel analogueexecutorpd.core.window.Window.mean()pd.core.window.Window.p_mean()threads / processespd.core.window.Window.sum()pd.core.window.Window.p_sum()threads / processespd.core.window.Window.var()pd.core.window.Window.p_var()threads / processespd.core.window.Window.std()pd.core.window.Window.p_std()threads / processesRollingGroupbymethodsparallel analogueexecutorpd.core.window.RollingGroupby.apply()pd.core.window.RollingGroupby.p_apply()threads / processespd.core.window.RollingGroupby.min()pd.core.window.RollingGroupby.p_min()threads / processespd.core.window.RollingGroupby.max()pd.core.window.RollingGroupby.p_max()threads / processespd.core.window.RollingGroupby.mean()pd.core.window.RollingGroupby.p_mean()threads / processespd.core.window.RollingGroupby.sum()pd.core.window.RollingGroupby.p_sum()threads / processespd.core.window.RollingGroupby.var()pd.core.window.RollingGroupby.p_var()threads / processespd.core.window.RollingGroupby.sem()pd.core.window.RollingGroupby.p_sem()threads / processespd.core.window.RollingGroupby.skew()pd.core.window.RollingGroupby.p_skew()threads / processespd.core.window.RollingGroupby.kurt()pd.core.window.RollingGroupby.p_kurt()threads / processespd.core.window.RollingGroupby.median()pd.core.window.RollingGroupby.p_median()threads / processespd.core.window.RollingGroupby.quantile()pd.core.window.RollingGroupby.p_quantile()threads / processespd.core.window.RollingGroupby.rank()pd.core.window.RollingGroupby.p_rank()threads / processespd.core.window.RollingGroupby.agg()pd.core.window.RollingGroupby.p_agg()threads / processespd.core.window.RollingGroupby.aggregate()pd.core.window.RollingGroupby.p_aggregate()threads / processesExpandingmethodsparallel analogueexecutorpd.core.window.Expanding.apply()pd.core.window.Expanding.p_apply()threads / processespd.core.window.Expanding.min()pd.core.window.Expanding.p_min()threads / processespd.core.window.Expanding.max()pd.core.window.Expanding.p_max()threads / processespd.core.window.Expanding.mean()pd.core.window.Expanding.p_mean()threads / processespd.core.window.Expanding.sum()pd.core.window.Expanding.p_sum()threads / processespd.core.window.Expanding.var()pd.core.window.Expanding.p_var()threads / processespd.core.window.Expanding.sem()pd.core.window.Expanding.p_sem()threads / processespd.core.window.Expanding.skew()pd.core.window.Expanding.p_skew()threads / processespd.core.window.Expanding.kurt()pd.core.window.Expanding.p_kurt()threads / processespd.core.window.Expanding.median()pd.core.window.Expanding.p_median()threads / processespd.core.window.Expanding.quantile()pd.core.window.Expanding.p_quantile()threads / processespd.core.window.Expanding.rank()pd.core.window.Expanding.p_rank()threads / processespd.core.window.Expanding.agg()pd.core.window.Expanding.p_agg()threads / processespd.core.window.Expanding.aggregate()pd.core.window.Expanding.p_aggregate()threads / processesExpandingGroupbymethodsparallel analogueexecutorpd.core.window.ExpandingGroupby.apply()pd.core.window.ExpandingGroupby.p_apply()threads / processespd.core.window.ExpandingGroupby.min()pd.core.window.ExpandingGroupby.p_min()threads / processespd.core.window.ExpandingGroupby.max()pd.core.window.ExpandingGroupby.p_max()threads / processespd.core.window.ExpandingGroupby.mean()pd.core.window.ExpandingGroupby.p_mean()threads / processespd.core.window.ExpandingGroupby.sum()pd.core.window.ExpandingGroupby.p_sum()threads / processespd.core.window.ExpandingGroupby.var()pd.core.window.ExpandingGroupby.p_var()threads / processespd.core.window.ExpandingGroupby.sem()pd.core.window.ExpandingGroupby.p_sem()threads / processespd.core.window.ExpandingGroupby.skew()pd.core.window.ExpandingGroupby.p_skew()threads / processespd.core.window.ExpandingGroupby.kurt()pd.core.window.ExpandingGroupby.p_kurt()threads / processespd.core.window.ExpandingGroupby.median()pd.core.window.ExpandingGroupby.p_median()threads / processespd.core.window.ExpandingGroupby.quantile()pd.core.window.ExpandingGroupby.p_quantile()threads / processespd.core.window.ExpandingGroupby.rank()pd.core.window.ExpandingGroupby.p_rank()threads / processespd.core.window.ExpandingGroupby.agg()pd.core.window.ExpandingGroupby.p_agg()threads / processespd.core.window.ExpandingGroupby.aggregate()pd.core.window.ExpandingGroupby.p_aggregate()threads / processesExponentialMovingWindowmethodsparallel analogueexecutorpd.core.window.ExponentialMovingWindow.mean()pd.core.window.ExponentialMovingWindow.p_mean()threads / processespd.core.window.ExponentialMovingWindow.sum()pd.core.window.ExponentialMovingWindow.p_sum()threads / processespd.core.window.ExponentialMovingWindow.var()pd.core.window.ExponentialMovingWindow.p_var()threads / processespd.core.window.ExponentialMovingWindow.std()pd.core.window.ExponentialMovingWindow.p_std()threads / processesExponentialMovingWindowGroupbymethodsparallel analogueexecutorpd.core.window.ExponentialMovingWindowGroupby.mean()pd.core.window.ExponentialMovingWindowGroupby.p_mean()threads / processespd.core.window.ExponentialMovingWindowGroupby.sum()pd.core.window.ExponentialMovingWindowGroupby.p_sum()threads / processespd.core.window.ExponentialMovingWindowGroupby.var()pd.core.window.ExponentialMovingWindowGroupby.p_var()threads / processespd.core.window.ExponentialMovingWindowGroupby.std()pd.core.window.ExponentialMovingWindowGroupby.p_std()threads / processes