Package: sk-dist
What is it?sk-distis a Python package for machine learning built on top ofscikit-learnand is
distributed under theApache 2.0 software
license. Thesk-distmodule can be thought of as “distributed scikit-learn” as
its core functionality is to extend thescikit-learnbuilt-injoblibparallelization of meta-estimator training tospark. A popular use case is the
parallelization of grid search as shown here:Check out theblog postfor more information on the motivation and use cases ofsk-dist.Main FeaturesDistributed Training-sk-distparallelizes the training ofscikit-learnmeta-estimators with PySpark. This allows
distributed training of these estimators without any constraint on
the physical resources of any one machine. In all cases, spark
artifacts are automatically stripped from the fitted estimator. These
estimators can then be pickled and un-pickled for prediction tasks,
operating identically at predict time to theirscikit-learncounterparts. Supported tasks are:Grid Search:Hyperparameter optimization
techniques,
particularlyGridSearchCVandRandomizedSeachCV,
are distributed such that each parameter set candidate is trained
in parallel.Multiclass Strategies:Multiclass classification
strategies,
particularlyOneVsRestClassifierandOneVsOneClassifier,
are distributed such that each binary probelm is trained in
parallel.Tree Ensembles:Decision tree
ensemblesfor classification and regression, particularlyRandomForestandExtraTrees,
are distributed such that each tree is trained in parallel.Distributed Prediction-sk-distprovides a prediction module
which buildsvectorized
UDFsforPySparkDataFramesusing fittedscikit-learnestimators. This distributes thepredictandpredict_probamethods ofscikit-learnestimators, enabling large scale prediction withscikit-learn.Feature Encoding-sk-distprovides a flexible feature
encoding utility calledEncoderizerwhich encodes mix-typed
feature spaces using either default behavior or user defined
customizable settings. It is particularly aimed at text features, but
it additionally handles numeric and dictionary type feature spaces.InstallationDependenciessk-distrequires:Python(>= 3.5)scikit-learn(>=0.20.0)pandas(>=0.17.0)numpyscipyjoblibDependency Notesversions ofnumpy,scipyandjoblibthat are compatible with any supported version ofscikit-learnshould be sufficient forsk-distsk-distis not supported with Python 2Spark DependenciesMostsk-distfunctionality requires a spark installation as well as
PySpark. Some functionality can run without spark, so spark related
dependencies are not required. The connection between sk-dist and spark
relies solely on asparkContextas an argument to varioussk-distclasses upon instantiation.A variety of spark configurations and setups will work. It is left up to
the user to configure their own spark setup. The testing suite runsspark 2.3andspark 2.4, though anyspark 2.0+versions
are expected to work.Additional spark related dependecies arepyarrow, which is used only
forskdist.predictfunctions. This uses vectorized pandas UDFs which
requirepyarrow>=0.8.0, tested withpyarrow==0.15.0.
Depending on the spark version, it may be necessary to setspark.conf.set("spark.sql.execution.arrow.enabled","true")in the
spark configuration.User InstallationThe easiest way to installsk-distis withpip:pip install --upgrade sk-distYou can also download the source code:git clone https://github.com/Ibotta/sk-dist.gitTestingWithpytestinstalled, you can run tests locally:pytest sk-distExamplesThe package contains numerousexampleson how to usesk-distin practice. Examples of note are:Grid Search with XGBoostSpark ML Benchmark ComparisonEncoderizer with 20 NewsgroupsOne-Vs-Rest vs One-Vs-OneLarge Scale Sklearn Prediction with PySpark UDFsGradient Boostingsk-disthas been tested with a number of popular gradient boosting packages that conform to thescikit-learnAPI. This
includesxgboostandcatboost. These will need to be installed in addition tosk-diston all nodes of the spark
cluster via a node bootstrap script. Version compatibility is left up to the user.Support forlightgbmis not guaranteed, as it requiresadditional installationson all
nodes of the spark cluster. This may work given proper installation but has not beed tested withsk-dist.BackgroundThe project was started atIbotta
Inc.on the machine learning
team and open sourced in 2019.It is currently maintained by the machine learning team at Ibotta. Special
thanks to those who contributed tosk-distwhile it was initially
in development at Ibotta:Evan HarrisNicole WoytarowiczMike LewisBobby CrimiThanks toJames Foleyfor logo artwork.