Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I don't understand the distinction being drawn between "networks" and "functions" in the "network reuse" subsection (Section 3).   To me it seems like this is exactly how Theano and TensorFlow (and possibly others) already work.  e.g., if I have a "relu" function and a "conv" function, I could compose these into a two-layer "network" function as follows (in untested Python, with '.'s used for indentation):

def compose(*funcs):
..def f(x):
....for func in funcs:
......x = func(x)
....return x
..return f

network = compose(conv, relu, conv, relu)

Could you expand more on the distinction you're making?
review_title: Distinction between "networks" and "functions"

final_score: rating: null
review_content: Does DeepDSL allow to manipulate symbolic gradients (e.g. to allow the symbolic representation of second-order gradients)?
review_title: Gradients question

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a domain specific language for the specification of deep learning models. The intermediate representation offers many possibilities for optimization and to focus on speed or runtime.

The paper is well-written and makes conclusive statements and comparisons. The experiments cover five fundamentally differenct CNN architectures, each evaluated for two batch sizes. They include the two competing frameworks Tensorflow and Caffe and show convincing performance. Overall, the paper is well-written and structured.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: This paper presents and evaluates a Scala-based deep learning framework called “DeepDSL,” describing the language’s syntactic and performance benefits with respect to existing frameworks.


Pros:

The use of Scala is unique among deep learning frameworks, to my knowledge, making this framework interesting for Scala users.  The fact that Scala compiles to Java and therefore cross-platform support comes for free is also nice.

The ability to inspect memory information as shown in Figure 3 is interesting and potentially useful for large networks or situations where memory is limited.

DeepDSL compares favorably with existing frameworks in terms of memory use and speed for many common convolutional network architectures.


Cons:

There appears to be special privileged handling of parameters, gradients, and updates in the compilation process itself (as in Caffe), rather than having gradients/updates as a normal part of the full user-defined computation graph (as in Theano + TensorFlow).  This makes certain applications, such as RNNs (which require parameter sharing) and GANs (which require gradients wrt multiple objectives), impossible to implement in DeepDSL without further extension of the underlying API.  (Note: I might be wrong about this -- and please correct me if I am -- but all the examples in the paper are nets trained by gradient descent on a single objective, and do not share parameters or access gradients directly.)

The paper repeatedly refers to line counts from the verbose Protobuf-based low-level representation of networks in Caffe to demonstrate the compactness of its own syntax.  This is misleading as Caffe has officially supported a compact network definition style called “NetSpec” for years -- see a ~15 line definition of AlexNet [1].  Given that, Protobuf is essentially an intermediate representation for Caffe (as with TensorFlow), which happens to have a human-readable text format.

DeepDSL is not especially novel when compared with existing frameworks, which is not a problem in and of itself, but some statements misleadingly or incorrectly oversell the novelty of the framework.  Some examples:

“This separation between network definition and training is an unique advantage of DeepDSL comparing to other tools.”  This separation is not unique -- it’s certainly a feature of Caffe where the network definition is its own file, and can be attained in TensorFlow as well (though it’s not the default workflow there).

“The difference [between our framework and Theano, TensorFlow, etc.] is that we do not model deep networks as ‘networks’ but as abstract ‘functions’.” There is no notion of a “network” in Theano or TensorFlow (not sure about the others) either -- there are only functions, like in DeepDSL.  I asked about this statement, and the response didn’t convince me otherwise.  The counterexample given was that in TensorFlow the number of input channels needs to be specified separately for each convolution.  This is only true using the low-level API and can easily be worked around with higher-level wrappers like TensorFlow Slim -- e.g., see the definition of AlexNet [2].  It may be true that DeepDSL is more “batteries included” for writing compact network definitions than these other frameworks, but the paper’s claims seem to go beyond this.


Overall, the DeepDSL framework seems to have real value in its use of Scala and its memory/speed efficiency as demonstrated by the experiments, but the current version of the paper contains statements that overclaim novelty in ways that are misleading and unfair to existing frameworks. I will consider upgrading my rating if these statements are removed or amended to be more technically precise.


[1] https://github.com/BVLC/caffe/blob/master/examples/pycaffe/caffenet.py#L24
[2] https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/slim/python/slim/nets/alexnet.py#L92

=====================

Update: the authors have revised their paper to address the concerns that I considered grounds for rejection in my review.  I've upgraded my rating from 5 (below threshold) to 7 (good paper, accept).
review_title: null

final_score: rating: null
review_content: As far as I understand, computation graph optimization in Theano does not differ much (at least on first sight) to your optimization approach. Could you detail more the differences?

In Figure 5, you detail the GPU memory usage. A first comment concerns the statement "Since Tensorflow allocates all GPU memory regardless of actual usage, we only compare DeepDSL and Caffe" (caption Fig. 5). I want to point out this behaviour is the default, but can be changed. You can get the session configuration by using, e.g., `sess_config = tf.ConfigProto()` before creating the session and then specify `sess_config.gpu_options.allow_growth = True`. In that case, Tensorflow will not preallocate the full memory for its pool.

In any case, it would be important to also have the values for the memory efficient, not only the runtime efficient mode of DeepDSL in Figure 5!
review_title: Evaluation comment and question

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents DeepDSL, a "domain specific language (DSL) embedded in
Scala, that compiles deep networks written in DeepDSL to Java source code". It
introduces its syntax and the key concepts which differentiate it
from other existing frameworks, such as Torch7, Theano, Caffe, TensorFlow,
CNTK, Chainer and MXNet. It also benchmarks speed and memory usage against
TensorFlow and Caffe on a variety of convolutional neural network architectures.

The paper is clear and well written and it does a good job of presenting DeepDSL
in the context of existing deep learning frameworks.

However, I don't think ICLR is the right venue for this type of work. Some of
the ideas it presents are interesting, but overall the paper lacks novelty and
potential impact and stays firmly within the realm of deep learning framework
whitepapers such as [1,2,3,4], which to my knowledge don't have a precedent of
being accepted at venues like ICLR.

[1]: Bergstra, James, et al. "Theano: A CPU and GPU math compiler in Python."
Proc. 9th Python in Science Conf. 2010.

[2]: Bastien, Frédéric, et al. "Theano: new features and speed improvements."
arXiv preprint arXiv:1211.5590 (2012).

[3]: Abadi, Martın, et al. "Tensorflow: Large-scale machine learning on
heterogeneous distributed systems." arXiv preprint arXiv:1603.04467 (2016).

[4]: The Theano Development Team et al. "Theano: A Python framework for fast
computation of mathematical expressions." arXiv preprint arXiv:1605.02688
(2016).

UPDATE: The rating has been revised to a 6 following the authors' reply.
review_title: ICLR is not the right venue for this paper.

Meta-review: All reviewers find value in the contributions. 
Decision: Accept (Poster)