Package: raydp
RayDPRayDP provides simple APIs for running Spark onRayand integrating Spark with AI libraries, making it simple to build distributed data and AI pipeline in a single python program.INTRODUCTIONProblem StatementA large-scale AI workflow usually involves multiple systems, for example Spark for data processing and PyTorch or Tensorflow for distributed training. A common setup is to use two separate clusters and stitch together multiple programs using glue code or a workflow orchestrator such as AirFlow or KubeFlow. However, in many cases this adds costs in terms of system efficiency and operations. The setup overhead of the workflow tasks adds latency. Data exchange among frameworks has to rely on external storage system which also adds latency. On operation side, managing two separate clusters introduces additional cost. Writing the pipeline using workflow orchestrator usually is also more complex than writing a single python program.Solution with Ray and RayDPTo solve the above challenges, more and more companies have adopted Ray as a single substrate for data processing, model training, serving and more. Ray makes it simple to build the data and AI pipeline in a single python program and scale from laptop to a cluster seamlessly. Ray has built a rich ecosystem by providing high quality libraries and integrating with other popular ones.Spark as a popular big data framework plays an important role in data and AI pipelines. RayDP brings Spark to the Ray ecosystem by supporting running Spark on top of Ray. By using RayDP, you can easily write PySpark code together with other Ray libraries in the same python program which improves productivity and expressivity. RayDP makes it simple to build distributed end-to-end data analytics and AI pipeline. RayDP supports exchanging data between Spark and other frameworks using Ray's in-memory object to provide best performance.Who will use RayDPML infrastructure team can build a modern ML platform on top of Ray, utilize RayDP to run Spark on Ray and unify with other AI components.Data scientists can use RayDP to write PySpark code together with other AI libraries, scale from laptop to cloud seamlessly.Data engineers can use RayDP to run on-demand Spark job in cloud without a need to setup a Spark cluster manually. The Ray cluster launcher helps to start a Ray cluster in cloud and RayDP allows you to run Spark in that cluster with auto scaling.PresentationsData + AI Summit 2021:Build Large-Scale Data Analytics and AI Pipeline Using RayDPRay Summit 2021:RayDP: Build Large-scale End-to-end Data Analytics and AI Pipelines Using Spark and RayARCHITECTURERayDP provides simple APIs for running Spark on Ray and APIs for converting a Spark DataFrame to a Ray Dataset which can be consumed by XGBoost, Ray Train, Horovod on Ray, etc. RayDP also provides high level scikit-learn style Estimator APIs for distributed training with PyTorch or Tensorflow.RayDP supports Ray as a Spark resource manager and runs Spark executors in Ray actors. The communication between Spark executors still uses Spark's internal protocol.QUICK STARTInstallationYou can install latest RayDP using pip. RayDP requires Ray and PySpark. Please also make sure java is installed and JAVA_HOME is set properly.pipinstallraydpOr you can install RayDP nightly build:pipinstall--preraydpNOTICE: formerly usedraydp-nightlywill no longer be updated.If you'd like to build and install the latest master, use the following command:./build.sh
pipinstalldist/raydp*.whlSpark on RayRayDP provides an API for starting a Spark job on Ray. To create a Spark session, call theraydp.init_sparkAPI. After that, you can use any Spark API as you want. For example:importrayimportraydp# connect to ray clusterray.init(address='auto')# create a Spark cluster with specified resource requirementsspark=raydp.init_spark(app_name='RayDP Example',num_executors=2,executor_cores=2,executor_memory='4GB')# normal data processesing with Sparkdf=spark.createDataFrame([('look',),('spark',),('tutorial',),('spark',),('look',),('python',)],['word'])df.show()word_count=df.groupBy('word').count()word_count.show()# stop the spark clusterraydp.stop_spark()Spark features such as dynamic resource allocation, spark-submit script, etc are also supported. Please refer toSpark on Rayfor more details.Spark + AI Pipeline on RayRayDP provides APIs for converting a Spark DataFrame to a Ray Dataset which can be consumed by XGBoost, Ray Train, Horovod on Ray, etc. RayDP also provides high level scikit-learn style Estimator APIs for distributed training with PyTorch or Tensorflow. To get started with end-to-end Spark + AI pipeline, the easiest way is to run the following tutorials on Google Collab. More examples are also available in theexamplesfolder.Spark + Ray Train Tutorial on Google CollabSpark + TorchEstimator Tutorial on Google CollabSpark DataFrame & Ray Dataset conversionYou can useray.data.from_sparkandds.to_sparkto convert between Spark DataFrame and Ray Dataset.importrayimportraydpray.init()spark=raydp.init_spark(app_name="RayDP Example",num_executors=2,executor_cores=2,executor_memory="4GB")# Spark Dataframe to Ray Datasetdf1=spark.range(0,1000)ds1=ray.data.from_spark(df1)# Ray Dataset to Spark Dataframeds2=ray.data.from_items([{"id":i}foriinrange(1000)])df2=ds2.to_spark(spark)Ray dataset converted from Spark dataframe this way will be no longer accessible afterraydp.stop_spark(). If you want to access the data after spark is shutdown, please useraydp.stop_spark(cleanup_data=False).Please refer toSpark+XGBoost on Rayfor a full example.Estimator APIThe Estimator APIs allow you to train a deep neural network directly on a Spark DataFrame, leveraging Ray’s ability to scale out across the cluster. The Estimator APIs are wrappers of Ray Train and hide the complexity of converting a Spark DataFrame to a PyTorch/Tensorflow dataset and distributing the training. RayDP providesraydp.torch.TorchEstimatorfor PyTorch andraydp.tf.TFEstimatorfor Tensorflow. The following is an example of using TorchEstimator.importrayimportraydpfromraydp.torchimportTorchEstimatorray.init(address="auto")spark=raydp.init_spark(app_name="RayDP Example",num_executors=2,executor_cores=2,executor_memory="4GB")# Spark DataFrame Codedf=spark.read.parquet(…)train_df=df.withColumn(…)# PyTorch Codemodel=torch.nn.Sequential(torch.nn.Linear(2,1))optimizer=torch.optim.Adam(model.parameters())estimator=TorchEstimator(model=model,optimizer=optimizer,...)estimator.fit_on_spark(train_df)raydp.stop_spark()Please refer toNYC Taxi PyTorch EstimatorandNYC Taxi Tensorflow Estimatorfor full examples.Fault ToleranceThe ray dataset converted from spark dataframe like above is not fault-tolerant. This is because we implement it usingRay.putcombined with sparkmapPartitions. Objects created byRay.putis not recoverable in Ray.RayDP now supports converting data in a way such that the resulting ray dataset is fault-tolerant. This feature is currentlyexperimental. Here is how to use it:importrayimportraydpray.init(address="auto")# set fault_tolerance_mode to True to enable the feature# this will connect pyspark driver to ray clusterspark=raydp.init_spark(app_name="RayDP Example",num_executors=2,executor_cores=2,executor_memory="4GB",fault_tolerance_mode=True)# df should be large enough so that result will be put into plasmadf=spark.range(100000)# use this API instead of ray.data.from_sparkds=raydp.spark.from_spark_recoverable(df)# ds is now fault-tolerant.Notice thatfrom_spark_recoverablewill persist the converted dataframe. You can provide the storage level through keyword parameterstorage_level. In addition, this feature is not available in ray client mode. If you need to use ray client, please wrap your application in a ray actor, as described in the ray client chapter.Getting InvolvedTo report bugs or request new features, please open a github issue.