Before revision:  While data-parallelism has been extensively studied and developed,  model-parallelism has received less attention as it is non-trivial to split the model across processes . In this paper, we propose HyPar-Flow : a framework for scalable and user-transparent parallel training of very large DNNs (up to 5, 000 layers).
After revision:  While data-parallelism has been extensively studied and developed, model-parallelism has received less attention as it is non-trivial to split the model across processes , 2) implementing forward/back-propagation across process boundaries that requires explicit communication, 3) obtaining parallel speedup on an inherently sequential task, and 4) achieving scalability without losing out on a model's accuracy. To address these problems, we create HyPar-Flow : a framework for scalable and user-transparent parallel training of very large DNNs (up to 5, 000 layers).