Package: elephas
Elephas: Distributed Deep Learning with Keras & SparkElephas is an extension ofKeras, which allows you to run distributed deep learning models at
scale withSpark. Elephas currently supports a number of
applications, including:Data-parallel training of deep learning modelsDistributed inference and evaluation of deep learning modelsDistributed training of ensemble models(removed as of 3.0.0)Distributed hyper-parameter optimization(removed as of 3.0.0)Schematically, elephas works as follows.Table of content:Elephas: Distributed Deep Learning with Keras & SparkIntroductionGetting startedBasic Spark integrationDistributed Inference and EvaluationSpark MLlib integrationSpark ML integrationHadoop integrationDistributed hyper-parameter optimizationDistributed training of ensemble modelsDiscussionLiteratureIntroductionElephas brings deep learning withKerastoSpark. Elephas intends to
keep the simplicity and high usability of Keras, thereby allowing for fast prototyping of distributed models, which
can be run on massive data sets. For an introductory example, see the followingiPython notebook.ἐλέφας is Greek forivoryand an accompanying project to κέρας, meaninghorn. If this seems weird mentioning, like
a bad dream, you should confirm it actually is at theKeras documentation.
Elephas also meanselephant, as in stuffed yellow elephant.Elephas implements a class of data-parallel algorithms on top of Keras, using Spark's RDDs and data frames.
Keras Models are initialized on the driver, then serialized and shipped to workers, alongside with data and broadcasted
model parameters. Spark workers deserialize the model, train their chunk of data and send their gradients back to the
driver. The "master" model on the driver is updated by an optimizer, which takes gradients either synchronously or
asynchronously.Getting startedJust install elephas from PyPI with, Spark will be installed throughpysparkfor you.pip install elephasThat's it, you should now be able to run Elephas examples.Basic Spark integrationAfter installing both Elephas, you can train a model as follows. First, create a local pyspark contextfrompysparkimportSparkContext,SparkConfconf=SparkConf().setAppName('Elephas_App').setMaster('local[8]')sc=SparkContext(conf=conf)Next, you define and compile a Keras modelfromtensorflow.keras.modelsimportSequentialfromtensorflow.keras.layersimportDense,Dropout,Activationfromtensorflow.keras.optimizersimportSGDmodel=Sequential()model.add(Dense(128,input_dim=784))model.add(Activation('relu'))model.add(Dropout(0.2))model.add(Dense(128))model.add(Activation('relu'))model.add(Dropout(0.2))model.add(Dense(10))model.add(Activation('softmax'))model.compile(loss='categorical_crossentropy',optimizer=SGD())and create an RDD from numpy arrays (or however you want to create an RDD)fromelephas.utils.rdd_utilsimportto_simple_rddrdd=to_simple_rdd(sc,x_train,y_train)The basic model in Elephas is theSparkModel. You initialize aSparkModelby passing in a compiled Keras model,
an update frequency and a parallelization mode. After that you can simplyfitthe model on your RDD. Elephasfithas the same options as a Keras model, so you can passepochs,batch_sizeetc. as you're used to from tensorflow.keras.fromelephas.spark_modelimportSparkModelspark_model=SparkModel(model,frequency='epoch',mode='asynchronous')spark_model.fit(rdd,epochs=20,batch_size=32,verbose=0,validation_split=0.1)Your script can now be run using spark-submitspark-submit--driver-memory1G./your_script.pyIncreasing the driver memory even further may be necessary, as the set of parameters in a network may be very large
and collecting them on the driver eats up a lot of resources. See the examples folder for a few working examples.Distributed Inference and EvaluationTheSparkModelcan also be used for distributed inference (prediction) and evaluation. Similar to thefitmethod,  thepredictandevaluatemethods
conform to the Keras Model API.fromelephas.spark_modelimportSparkModel# create/train the model, similar to the previous section (Basic Spark Integration)model=...spark_model=SparkModel(model,...)spark_model.fit(...)x_test,y_test=...# load test datapredictions=spark_model.predict(x_test)# perform inferenceevaluation=spark_model.evaluate(x_test,y_test)# perform evaluation/scoringThe paradigm is identical to the data parallelism in training, as the model is serialized and shipped to the workers and used to evaluate a chunk of the testing data. The predict method will take either a numpy array or an RDD.Spark MLlib integrationFollowing up on the last example, to use Spark's MLlib library with Elephas, you create an RDD of LabeledPoints for
supervised training as followsfromelephas.utils.rdd_utilsimportto_labeled_pointlp_rdd=to_labeled_point(sc,x_train,y_train,categorical=True)Training a given LabeledPoint-RDD is very similar to what we've seen alreadyfromelephas.spark_modelimportSparkMLlibModelspark_model=SparkMLlibModel(model,frequency='batch',mode='hogwild')spark_model.train(lp_rdd,epochs=20,batch_size=32,verbose=0,validation_split=0.1,categorical=True,nb_classes=nb_classes)Spark ML integrationTo train a model with a SparkML estimator on a data frame, use the following syntax.df=to_data_frame(sc,x_train,y_train,categorical=True)test_df=to_data_frame(sc,x_test,y_test,categorical=True)estimator=ElephasEstimator(model,epochs=epochs,batch_size=batch_size,frequency='batch',mode='asynchronous',categorical=True,nb_classes=nb_classes)fitted_model=estimator.fit(df)Fitting an estimator results in a SparkML transformer, which we can use for predictions and other evaluations by
calling the transform method on it.prediction=fitted_model.transform(test_df)pnl=prediction.select("label","prediction")pnl.show(100)importnumpyasnpprediction_and_label=pnl.rdd.map(lambdarow:(row.label,float(np.argmax(row.prediction))))metrics=MulticlassMetrics(prediction_and_label)print(metrics.weightedPrecision)print(metrics.weightedRecall)If the model utilizes custom activation function, layer, or loss function, that will need to be supplied using theset_custom_objectsmethod:defcustom_activation(x):...classCustomLayer(Layer):...model=Sequential()model.add(CustomLayer(...))estimator=ElephasEstimator(model,epochs=epochs,batch_size=batch_size)estimator.set_custom_objects({'custom_activation':custom_activation,'CustomLayer':CustomLayer})Hadoop IntegrationIn addition to saving locally, models may be saved directly into a network-accessible Hadoop cluster.spark_model.save('/absolute/file/path/model.h5',to_hadoop=True)Models saved on a network-accessible Hadoop cluster may be loaded as follows.fromelephas.spark_modelimportload_spark_modelspark_model=load_spark_model('/absolute/file/path/model.h5',from_hadoop=True)Distributed hyper-parameter optimizationUPDATE: As of 3.0.0, Hyper-parameter optimization features have been removed, since Hyperas is no longer active and was causing versioning compatibility issues. To use these features, install version 2.1 or below.Hyper-parameter optimization with elephas is based onhyperas, a convenience
wrapper for hyperopt and keras. Each Spark worker executes a number of trials, the results get collected and the best
model is returned. As the distributed mode in hyperopt (using MongoDB), is somewhat difficult to configure and error
prone at the time of writing, we chose to implement parallelization ourselves. Right now, the only available
optimization algorithm is random search.The first part of this example is more or less directly taken from the hyperas documentation. We define data and model
as functions, hyper-parameter ranges are defined through braces. See the hyperas documentation for more on how
this works.fromhyperoptimportSTATUS_OKfromhyperas.distributionsimportchoice,uniformdefdata():fromtensorflow.keras.datasetsimportmnistfromtensorflow.keras.utilsimportto_categorical(x_train,y_train),(x_test,y_test)=mnist.load_data()x_train=x_train.reshape(60000,784)x_test=x_test.reshape(10000,784)x_train=x_train.astype('float32')x_test=x_test.astype('float32')x_train/=255x_test/=255nb_classes=10y_train=to_categorical(y_train,nb_classes)y_test=to_categorical(y_test,nb_classes)returnx_train,y_train,x_test,y_testdefmodel(x_train,y_train,x_test,y_test):fromtensorflow.keras.modelsimportSequentialfromtensorflow.keras.layersimportDense,Dropout,Activationfromtensorflow.keras.optimizersimportRMSpropmodel=Sequential()model.add(Dense(512,input_shape=(784,)))model.add(Activation('relu'))model.add(Dropout({{uniform(0,1)}}))model.add(Dense({{choice([256,512,1024])}}))model.add(Activation('relu'))model.add(Dropout({{uniform(0,1)}}))model.add(Dense(10))model.add(Activation('softmax'))rms=RMSprop()model.compile(loss='categorical_crossentropy',optimizer=rms)model.fit(x_train,y_train,batch_size={{choice([64,128])}},nb_epoch=1,show_accuracy=True,verbose=2,validation_data=(x_test,y_test))score,acc=model.evaluate(x_test,y_test,show_accuracy=True,verbose=0)print('Test accuracy:',acc)return{'loss':-acc,'status':STATUS_OK,'model':model.to_json()}Once the basic setup is defined, running the minimization is done in just a few lines of code:fromelephas.hyperparamimportHyperParamModelfrompysparkimportSparkContext,SparkConf# Create Spark contextconf=SparkConf().setAppName('Elephas_Hyperparameter_Optimization').setMaster('local[8]')sc=SparkContext(conf=conf)# Define hyper-parameter model and run optimizationhyperparam_model=HyperParamModel(sc)hyperparam_model.minimize(model=model,data=data,max_evals=5)Distributed training of ensemble modelsUPDATE: As of 3.0.0, Hyper-parameter optimization features have been removed, since Hyperas is no longer active and was causing versioning compatibility issues. To use these features, install version 2.1 or below.Building on the last section, it is possible to train ensemble models with elephas by means of running hyper-parameter
optimization on large search spaces and defining a resulting voting classifier on the top-n performing models.
Withdataandmodeldefined as above, this is a simple as runningresult=hyperparam_model.best_ensemble(nb_ensemble_models=10,model=model,data=data,max_evals=5)In this example an ensemble of 10 models is built, based on optimization of at most 5 runs on each of the Spark workers.DiscussionPremature parallelization may not be the root of all evil, but it may not always be the best idea to do so. Keep in
mind that more workers mean less data per worker and parallelizing a model is not an excuse for actual learning.
So, if you can perfectly well fit your data into memoryandyou're happy with training speed of the model consider
just using keras.One exception to this rule may be that you're already working within the Spark ecosystem and want to leverage what's
there. The above SparkML example shows how to use evaluation modules from Spark and maybe you wish to further process
the outcome of an elephas model down the road. In this case, we recommend to use elephas as a simple wrapper by setting
num_workers=1.Note that right now elephas restricts itself to data-parallel algorithms for two reasons. First, Spark simply makes it
very easy to distribute data. Second, neither Spark nor Theano make it particularly easy to split up the actual model
in parts, thus making model-parallelism practically impossible to realize.Having said all that, we hope you learn to appreciate elephas as a pretty easy to setup and use playground for
data-parallel deep-learning algorithms.Literature[1] J. Dean, G.S. Corrado, R. Monga, K. Chen, M. Devin, QV. Le, MZ. Mao, M’A. Ranzato, A. Senior, P. Tucker, K. Yang, and AY. Ng.Large Scale Distributed Deep Networks.[2] F. Niu, B. Recht, C. Re, S.J. WrightHOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent[3] C. Noel, S. Osindero.Dogwild! — Distributed Hogwild for CPU & GPUMaintainers / ContributionsThis great project was started by Max Pumperla, and is currently maintained by Daniel Cahall (https://github.com/danielenricocahall). If you have any questions, please feel free to open up an issue or send an email todanielenricocahall@gmail.com. If you want to contribute, feel free to submit a PR, or start a conversation about how we can go about implementing something.Star History