A computer-implemented method for parallelizing training of a deep neural network "DNN" model, comprising: generating a profile of the DNN model, the DNN model comprising a plurality of layers and the profile of the DNN model including data describing the performance characteristics of the DNN model, by: computing total computation time for each layer of the plurality of layers in the DNN; computing size of output activations from each layer to the next layer in the DNN; and determining size of weights for each layer in the DNN; partitioning the layers of the DNN model into a plurality of stages based on the profile, wherein each of the plurality of stages comprises one or more of the layers of the DNN model, and wherein the partitioning is optimized to minimize a time to train the DNN model; and assigning the plurality of stages to a plurality of worker computing devices based on the partitioning; training the DNN model using a one-forward one-backward scheduling policy, wherein the one-forward one-backward scheduling policy interleaves forward and backward minibatch processing on each of the plurality of worker computing devices, whereby in a steady state each of the plurality of worker computing devices performs forward processing and backward processing for the layers of the DNN model in a stage of the plurality of stages, one or more of the plurality of worker computing devices alternating between performing forward processing and backward processing of the same minibatches on the DNN training data; and one or more of the plurality of worker computing devices alternating between performing forward processing and backward processing of different minibatches of training data, wherein at least one of the plurality of stages is configured for model parallel processing and wherein at least one of the plurality of stages is configured for data parallel processing. generating a profile of the DNN model, the DNN model comprising a plurality of layers and the profile of the DNN model including data describing the performance characteristics of the DNN model, by: computing total computation time for each layer of the plurality of layers in the DNN; computing size of output activations from each layer to the next layer in the DNN; and determining size of weights for each layer in the DNN; computing total computation time for each layer of the plurality of layers in the DNN; computing size of output activations from each layer to the next layer in the DNN; and determining size of weights for each layer in the DNN; partitioning the layers of the DNN model into a plurality of stages based on the profile, wherein each of the plurality of stages comprises one or more of the layers of the DNN model, and wherein the partitioning is optimized to minimize a time to train the DNN model; and assigning the plurality of stages to a plurality of worker computing devices based on the partitioning; training the DNN model using a one-forward one-backward scheduling policy, wherein the one-forward one-backward scheduling policy interleaves forward and backward minibatch processing on each of the plurality of worker computing devices, whereby in a steady state each of the plurality of worker computing devices performs forward processing and backward processing for the layers of the DNN model in a stage of the plurality of stages, one or more of the plurality of worker computing devices alternating between performing forward processing and backward processing of the same minibatches on the DNN training data; and one or more of the plurality of worker computing devices alternating between performing forward processing and backward processing of different minibatches of training data, wherein at least one of the plurality of stages is configured for model parallel processing and wherein at least one of the plurality of stages is configured for data parallel processing.