Package: tensorflowonspark
TensorFlowOnSparkTensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark
clusters.By combining salient features from theTensorFlowdeep learning framework withApache SparkandApache Hadoop, TensorFlowOnSpark enables distributed
deep learning on a cluster of GPU and CPU servers.It enables both distributed TensorFlow training and
inferencing on Spark clusters, with a goal to minimize the amount
of code changes required to run existing TensorFlow programs on a
shared grid.  Its Spark-compatible API helps manage the TensorFlow
cluster with the following steps:Startup- launches the Tensorflow main function on the executors, along with listeners for data/control messages.Data ingestionInputMode.TENSORFLOW- leverages TensorFlow's built-in APIs to read data files directly from HDFS.InputMode.SPARK- sends Spark RDD data to the TensorFlow nodes via aTFNode.DataFeedclass.  Note that we leverage theHadoop Input/Output Formatto access TFRecords on HDFS.Shutdown- shuts down the Tensorflow workers and PS nodes on the executors.Table of ContentsBackgroundInstallUsageAPIContributeLicenseBackgroundTensorFlowOnSpark was developed by Yahoo for large-scale distributed
deep learning on our Hadoop clusters in Yahoo's private cloud.TensorFlowOnSpark provides some important benefits (seeour
blog)
over alternative deep learning solutions.Easily migrate existing TensorFlow programs with <10 lines of code change.Support all TensorFlow functionalities: synchronous/asynchronous training, model/data parallelism, inferencing and TensorBoard.Server-to-server direct communication achieves faster learning when available.Allow datasets on HDFS and other sources pushed by Spark or pulled by TensorFlow.Easily integrate with your existing Spark data processing pipelines.Easily deployed on cloud or on-premise and on CPUs or GPUs.InstallTensorFlowOnSpark is provided as a pip package, which can be installed on single machines via:# for tensorflow>=2.0.0
pip install tensorflowonspark

# for tensorflow<2.0.0
pip install tensorflowonspark==1.4.4For distributed clusters, please see ourwiki sitefor detailed documentation for specific environments, such as our getting started guides forsingle-node Spark Standalone,YARN clustersandAWS EC2.  Note: the Windows operating system is not currently supported due tothis issue.UsageTo use TensorFlowOnSpark with an existing TensorFlow application, you can follow ourConversion Guideto describe the required changes.  Additionally, ourwiki sitehas pointers to some presentations which provide an overview of the platform.Note: since TensorFlow 2.x breaks API compatibility with TensorFlow 1.x, the examples have been updated accordingly.  If you are using TensorFlow 1.x, you will need to checkout thev1.4.4tag for compatible examples and instructions.APIAPI Documentationis automatically generated from the code.ContributePlease join theTensorFlowOnSpark user groupfor discussions and questions.  If you have a question, please review ourFAQbefore posting.Contributions are always welcome.  For more information, please see ourguide for getting involved.LicenseThe use and distribution terms for this software are covered by the Apache 2.0 license.
SeeLICENSEfile for terms.