Before revision:  In this paper, we propose HyPar-Flow : a framework for scalable and user-transparent parallel training of very large DNNs (up to 5, 000 layers). We exploit TensorFlow's Eager Execution features and KerasAPIs for model definition and distribution .
After revision:  In this paper, we propose HyPar-Flow : a framework for scalable and user-transparent system for hybrid-parallel training by exploiting MPI, Keras, and TensorFlow .