Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In PALEO the authors propose a simple model of execution of deep neural networks. It turns out that even this simple model allows to quite accurately predict the computation time for image recognition networks both in single-machine and distributed settings.

The ability to predict network running time is very useful, and the paper shows that even a simple model does it reasonably, which is a strength. But the tests are only performed on a few networks of very similar type (AlexNet, Inception, NiN) and only in a few settings. Much broader experiments, including a variety of models (RNNs, fully connected, adversarial, etc.) in a variety of settings (different batch sizes, layer sizes, node placement on devices, etc.) would probably reveal weaknesses of the proposed very simplified model. This is why this reviewer considers this paper borderline -- it's a first step, but a very basic one and without sufficiently large experimental underpinning.

More experiments were added, so I'm updating my score.
review_title: Final review: Sound paper but a very simple model, few experiments at start but more added.

final_score: rating: 7: Good paper, accept
review_content: This paper introduces an analytical performance model to estimate the training and evaluation time of a given network for different software, hardware and communication strategies. 
The paper is very clear.  The authors included many freedoms in the variables while calculating the run-time of a network such as the number of workers, bandwidth, platform, and parallelization strategy. Their results are consistent with the reported results from literature.
Furthermore, their code is open-source and the live demo is looking good. 
The authors mentioned in their comment that they will allow users to upload customized networks and model splits in the coming releases of the interface, then the tool can become very useful.
It would be interesting to see some newer network architectures with skip connections such as ResNet, and DenseNet.


review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper is technically sound. It highlights well the strengths and weaknesses of the proposed simplified model.

In terms of impact, its novelty is limited, in the sense that the authors did seemingly the right thing and obtained the expected outcomes. The idea of modeling deep learning computation is not in itself particularly novel. As a companion paper to an open source release of the model, it would meet my bar of acceptance in the same vein as a paper describing a novel dataset, which might not provide groundbreaking insights, yet be generally useful to the community.

In the absence of released code, even if the authors promise to release it soon, I am more ambivalent, since that's where all the value lies. It would also be a different story if the authors had been able to use this framework to make novel architectural decisions that improved training scalability in some way, and incorporated such new insights in the paper.

UPDATED: code is now available. Revised review accordingly.
review_title: Technically sound. Only useful under the assumption that the code is released.

final_score: rating: null
review_content: I found the paper clear enough, had no questions, but want this task to go away.
review_title: No question

final_score: rating: null
review_content: Can you explain what is "one step time" in Table 2? Also again in Table 2, there are estimates by PALEO, can you also add the reported results from the publications for these estimations? 
review_title: Clarification - Table 2

final_score: rating: null
review_content: Or better: incorporate it into TensorFlow so that the framework is able to make better decisions.
review_title: Any plans to open-source the model?

Meta-review: The reviewers were consistent in their praise of the paper. They asked for newer architectures, e.g. ResNet, DenseNet. The authors released an update with a Caffe converter which provides access to a wide range of CNNs and residual networks (ResNet-50 and DenseNet examples are provided). This seems like an incredibly useful tool and very glad it is open source. Paper is a clear accept.
Decision: Accept (Poster)