Title: DeepDSL: A Compilation-based Domain-Specific Language for Deep Learning
Authors: Tian Zhao
Xiao Bing Huang
Yu Cao
Venue: ICLR 2017
Keywords: Deep learning
Applications
Optimization

Abstract: In recent years, Deep Learning (DL) has found great success in domains such as multimedia understanding. However, the complex nature of multimedia data makes it difficult to develop DL-based software. The state-of-the-art tools, such as Caffe, TensorFlow, Torch7, and CNTK, while are successful in their applicable domains, are programming libraries with fixed user interface, internal representation, and execution environment. This makes it difficult to implement portable and customized DL applications.

In this paper, we present DeepDSL, a domain specific language (DSL) embedded in Scala, that compiles deep networks written in DeepDSL to Java source code. Deep DSL provides 

(1) intuitive constructs to support compact encoding of deep networks; 
(2) symbolic gradient derivation of the networks; 
(3) static analysis for memory consumption and error detection; and 
(4) DSL-level optimization to improve memory and runtime efficiency. 

DeepDSL programs are compiled into compact, efficient, customizable, and portable Java source code, which operates the CUDA and CUDNN interfaces running on NVIDIA GPU via a Java Native Interface (JNI) library. We evaluated DeepDSL with a number of popular DL networks. Our experiments show that the compiled programs have very competitive runtime performance and memory efficiency compared to the existing libraries.

TL;DR: DeepDSL(a DSL embedded in Scala) that compiles deep learning networks written in DeepDSL to Java source code, which runs on any GPU equipped machines with competitive efficiency as existing state-of-the-art tools (e.g. Caffe and Tensorflow)

Review discussion:
Does DeepDSL allow to manipulate symbolic gradients (e.g. to allow the symbolic representation of second-order gradients)?
Hi, glad to answer your question!
First, regarding the second-order gradient support:
1. We already support second-order gradients for scalar values;
2. We do not support second-order gradients for tensor values currently but supporting it should not require extensive changes since everything is abstract.
Now we describe a bit details with the below examples.
Suppose e is a scalar expression, DeepDSL supports all the below already:
A. if x is a scalar variable, d e / d x is still a scalar
B. if v is a tensor, d e / d v is a tensor
C. (d e / d v) ( d v' ) will be a tensor gradient
D d v' / d v is a tensor gradient
But we don't have d (d v1 / d v2) / d v3 since we don't see a use for it.
Everything is currently handled internally and transparent to the user.
However, we certainly can expose the gradient handling to allow users to manipulate symbolic gradients. It is just that we don't see a use case yet for this functionality. Maybe you can explain a bit on why this is needed and under what circumstance so we can add the support in the next version.
As far as I understand, computation graph optimization in Theano does not differ much (at least on first sight) to your optimization approach. Could you detail more the differences?
In Figure 5, you detail the GPU memory usage. A first comment concerns the statement "Since Tensorflow allocates all GPU memory regardless of actual usage, we only compare DeepDSL and Caffe" (caption Fig. 5). I want to point out this behaviour is the default, but can be changed. You can get the session configuration by using, e.g., `sess_config = tf.ConfigProto()` before creating the session and then specify `sess_config.gpu_options.allow_growth = True`. In that case, Tensorflow will not preallocate the full memory for its pool.
In any case, it would be important to also have the values for the memory efficient, not only the runtime efficient mode of DeepDSL in Figure 5!
Thank you for your comments and suggestions. Below is our reply:
1. Using your excellent suggestion (greatly appreciated!), we are able to measure the memory use of Tensorflow for our tests. The detailed information is included in the revised PDF file. A quick summary is that DeepDSL uses less memory than Tensorflow in all but Vgg.
2. We also included runtime and memory performance for DeepDSL in memory efficient mode. In this mode, DeepDSL uses 10~30%  less memory with similar runtime overhead except Googlenet and Vgg where higher percentage of memory saving is achieved with relatively less runtime penalty.
3.
About similarity to Theano's graph optimization, we had added a few paragraphs towards the end of the review section.
DeepDSL does share some similarity in the optimization steps such as redundancy removal and simplification strategy.
However, DeepDSL does not have a computation graph specifically since DeepDSL expressions are actually just expressions. There is no explicit or implicit graph of any kind. DeepDSL expressions are optimized locally for each expression first.
Then the expressions are broken down into simple assignment expressions (SSA transformation) where global optimization such as common subexpression elimination, code scheduling, in-place computation, and deallocation are implemented.
Another difference is in code generation, where Theano generates C code from its low-level functions while DeepDSL code generation is through a simple single-pass code generator that goes through the optimized DSL expressions (in single-assignment form) and print Java code as strings.
I don't understand the distinction being drawn between "networks" and "functions" in the "network reuse" subsection (Section 3).   To me it seems like this is exactly how Theano and TensorFlow (and possibly others) already work.  e.g., if I have a "relu" function and a "conv" function, I could compose these into a two-layer "network" function as follows (in untested Python, with '.'s used for indentation):
def compose(*funcs):
..def f(x):
....for func in funcs:
......x = func(x)
....return x
..return f
network = compose(conv, relu, conv, relu)
Could you expand more on the distinction you're making?
Thanks for your comments. Below is our clarification of the difference between network and DeepDSL function.
In DeepDSL, tensor functions are DSL entities. There is no network concept in DeepDSL and a network is just a tensor function (composed one).
The example you gave is similar to the "o" operation in DeepDSL though it is a Python function.
The difference may seems superficial but considering that a Python function does not expose its argument before it is applied to the argument.
This poses a problem.
For example, in Tensorflow, you may have trouble extracting the output channel size of a convolution layer before calling
the tf.nn.conv2d function to form the convolution layer.
As a result, when you compose two convolution layers, you may have to specify the input and output channel sizes for the second convolution layer.
That is, before calling "compose(conv1, relu, conv2, relu)", you may have to fix the in_channel size for conv2 using the out_channel size of conv1.
Of course, there is always a way to get around this but having DSL abstractions for tensor functions makes it explicit.
For example, given
val cv1 = CudaLayer.convolv("cv1", 5, 20) // 5 x 5 kernel, output channel 20, stride 1, padding 0
val cv2 = CudaLayer.convolv("cv2", 5, 50) // 5 x 5 kernel, output channel 50, stride 1, padding 0
The tensor type of cv1 is List(N, K, M1, M2) -> List(N, 20, (M1-5)+1, (M2-5)+1)).
The tensor type of cv2 is List(N1, K1, M3, M4) -> List(N1, 50, (M3-5)+1, (M4-5)+1).
The tensor type of cv1 o cv2 is
List(N, K, M1, M2) -> List(N, 50, (((M1-5)+1)-5)+1, ((M2-5)+1)-5)+1)
where N, K, M1, M2 are dimension variables and (((M1-5)+1)-5)+1, ((M2-5)+1)-5)+1 are dimension expressions that won't be evaluated until code generation.
During composition of cv1 and cv2, the dimension variable K1 is substituted by 20, N1 is replaced by N, M3 and M4 are replaced by (M1-5)+1 and (M2-5)+1 respectively.
Note that "K" remains a dimension variable and may be substituted by other dimension expressions when "cv1 o cv2" is composed with other tensor functions.
This paper presents a domain specific language for the specification of deep learning models. The intermediate representation offers many possibilities for optimization and to focus on speed or runtime.
The paper is well-written and makes conclusive statements and comparisons. The experiments cover five fundamentally differenct CNN architectures, each evaluated for two batch sizes. They include the two competing frameworks Tensorflow and Caffe and show convincing performance.
Overall, the paper is well-written and structured.
The paper presents DeepDSL, a "domain specific language (DSL) embedded in
Scala, that compiles deep networks written in DeepDSL to Java source code". It
introduces its syntax and the key concepts which differentiate it
from other existing frameworks, such as Torch7, Theano, Caffe, TensorFlow,
CNTK, Chainer and MXNet. It also benchmarks speed and memory usage against
TensorFlow and Caffe on a variety of convolutional neural network architectures.
The paper is clear and well written and it does a good job of presenting DeepDSL
in the context of existing deep learning frameworks.
However, I don't think ICLR is the right venue for this type of work. Some of
the ideas it presents are interesting, but overall the paper lacks novelty and
potential impact and stays firmly within the realm of deep learning framework
whitepapers such as [1,2,3,4], which to my knowledge don't have a precedent of
being accepted at venues like ICLR.
[1]: Bergstra, James, et al. "Theano: A CPU and GPU math compiler in Python."
Proc. 9th Python in Science Conf. 2010.
[2]: Bastien, Frédéric, et al. "Theano: new features and speed improvements."
arXiv preprint arXiv:1211.5590 (2012).
[3]: Abadi, Martın, et al. "Tensorflow: Large-scale machine learning on
heterogeneous distributed systems." arXiv preprint arXiv:1603.04467 (2016).
[4]: The Theano Development Team et al. "Theano: A Python framework for fast
computation of mathematical expressions." arXiv preprint arXiv:1605.02688
UPDATE: The rating has been revised to a 6 following the authors' reply.
In the list of suggested topics for this conference, there is
- Implementation issues, parallelization, software platforms, hardware
which we understood to include software platforms such as DeepDSL.
While the main theme of the conference is on learning representation, software tools for learning deserve some attention from users such as the audience of this conference. Many researchers use existing libraries for their deep learning research but picking the right tool, using them correctly, and knowing how to extend them can save a lot time and energy.
As to novelty and potential impact, DeepDSL is the first DL tool that compiles to Java source code with very few dependencies, which greatly improves the portability of the tool. Running it on Windows is as easy as running it on Linux or Unix. DeepDSL is also significantly more efficient than Tensorflow and Caffe in complex networks such as Googlenet and Resnet.
While researchers focus on algorithms to improve accuracy, speed, and memory usage, tools like DeepDSL are able to improve speed and memory usage at implementation level. Through venue like ICLR, we hope to reach the target audience.
Thank you for pointing out this specific section of the conference instructions to me. Given this and the fact that the other reviewers are ready to accept the paper, I'm updating my rating to 6 to reflect my opinion of the paper's quality.
This paper presents and evaluates a Scala-based deep learning framework called “DeepDSL,” describing the language’s syntactic and performance benefits with respect to existing frameworks.
Pros:
The use of Scala is unique among deep learning frameworks, to my knowledge, making this framework interesting for Scala users.  The fact that Scala compiles to Java and therefore cross-platform support comes for free is also nice.
The ability to inspect memory information as shown in Figure 3 is interesting and potentially useful for large networks or situations where memory is limited.
DeepDSL compares favorably with existing frameworks in terms of memory use and speed for many common convolutional network architectures.
Cons:
There appears to be special privileged handling of parameters, gradients, and updates in the compilation process itself (as in Caffe), rather than having gradients/updates as a normal part of the full user-defined computation graph (as in Theano + TensorFlow).  This makes certain applications, such as RNNs (which require parameter sharing) and GANs (which require gradients wrt multiple objectives), impossible to implement in DeepDSL without further extension of the underlying API.  (Note: I might be wrong about this -- and please correct me if I am -- but all the examples in the paper are nets trained by gradient descent on a single objective, and do not share parameters or access gradients directly.)
The paper repeatedly refers to line counts from the verbose Protobuf-based low-level representation of networks in Caffe to demonstrate the compactness of its own syntax.  This is misleading as Caffe has officially supported a compact network definition style called “NetSpec” for years -- see a ~15 line definition of AlexNet [1].  Given that, Protobuf is essentially an intermediate representation for Caffe (as with TensorFlow), which happens to have a human-readable text format.
DeepDSL is not especially novel when compared with existing frameworks, which is not a problem in and of itself, but some statements misleadingly or incorrectly oversell the novelty of the framework.  Some examples:
“This separation between network definition and training is an unique advantage of DeepDSL comparing to other tools.”  This separation is not unique -- it’s certainly a feature of Caffe where the network definition is its own file, and can be attained in TensorFlow as well (though it’s not the default workflow there).
“The difference [between our framework and Theano, TensorFlow, etc.] is that we do not model deep networks as ‘networks’ but as abstract ‘functions’.” There is no notion of a “network” in Theano or TensorFlow (not sure about the others) either -- there are only functions, like in DeepDSL.  I asked about this statement, and the response didn’t convince me otherwise.  The counterexample given was that in TensorFlow the number of input channels needs to be specified separately for each convolution.  This is only true using the low-level API and can easily be worked around with higher-level wrappers like TensorFlow Slim -- e.g., see the definition of AlexNet [2].  It may be true that DeepDSL is more “batteries included” for writing compact network definitions than these other frameworks, but the paper’s claims seem to go beyond this.
Overall, the DeepDSL framework seems to have real value in its use of Scala and its memory/speed efficiency as demonstrated by the experiments, but the current version of the paper contains statements that overclaim novelty in ways that are misleading and unfair to existing frameworks. I will consider upgrading my rating if these statements are removed or amended to be more technically precise.
[1] https://github.com/BVLC/caffe/blob/master/examples/pycaffe/caffenet.py#L24
[2] https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/slim/python/slim/nets/alexnet.py#L92
Update: the authors have revised their paper to address the concerns that I considered grounds for rejection in my review.  I've upgraded my rating from 5 (below threshold) to 7 (good paper, accept).
Thank you for the comments. Below is our reply.
> There appears to be special privileged handling of parameters, gradients, and updates in the compilation process itself (as in Caffe), rather than
> having gradients/updates as a normal part of the full user-defined computation graph (as in Theano + TensorFlow).  This makes certain applications,
> such as RNNs (which require parameter sharing) and GANs (which require gradients wrt multiple objectives), impossible to implement in DeepDSL without
> further extension of the underlying API.
We do provide API for gradient derivation and updates. For the examples we encoded gradient updates inside the "Loop" class since it is the same for the networks we tested.
To clarify this, we have added some text within Section 5 (compilation).
The gradient of a loss expression w.r.t. a parameter p can be implemented as " loss.grad(p) ", which is a tensor expression.
Gradient update is written using expression of the form " Update(p, gradient, alpha, beta) " which represents the computation:
p = p * beta + gradient * alpha.
The gradient updates for all parameters are put into a list and then passed to optimization functions to obtain a list of optimized IR expressions ready for code generation.
We have not implemented RNN or GAN, which remains our future work though we do not think gradient update is a problem.
> The paper repeatedly refers to line counts from the verbose Protobuf-based low-level representation of networks in Caffe to demonstrate the
> compactness of its own syntax.  This is misleading as Caffe has officially supported a compact network definition style called “NetSpec” for years --
> see a ~15 line definition of AlexNet [1].  Given that, Protobuf is essentially an intermediate representation for Caffe (as with TensorFlow), which
> happens to have a human-readable text format.
You are right. The compact encoding is not unique to DeepDSL and we removed all mentioning of this.
> DeepDSL is not especially novel when compared with existing frameworks, which is not a problem in and of itself, but some statements misleadingly or > incorrectly oversell the novelty of the framework.  Some examples:
> “This separation between network definition and training is an unique advantage of DeepDSL comparing to other tools.”  This separation is not unique
> -- it’s certainly a feature of Caffe where the network definition is its own file, and can be attained in TensorFlow as well (though it’s not the
> default workflow there).
Our sentence is perhaps misleading and we replaced it.
We meant to say that the compiled Java source code (used for training) is independent of the network definition and DeepDSL compiler.
The benefit is that the compiled Java code only depends on a small Java library (that calls cuDNN) -- not the rest of the DeepDSL or Scala.
> “The difference [between our framework and Theano, TensorFlow, etc.] is that we do not model deep networks as ‘networks’ but as abstract
> ‘functions’.” There is no notion of a “network” in Theano or TensorFlow (not sure about the others) either -- there are only functions, like in
> DeepDSL.  I asked about this statement, and the response didn’t convince me otherwise.  The counterexample given was that in TensorFlow the number of
> input channels needs to be specified separately for each convolution.  This is only true using the low-level API and can easily be worked around with
> higher-level wrappers like TensorFlow Slim -- e.g., see the definition of AlexNet [2].  It may be true that DeepDSL is more “batteries included” for
> writing compact network definitions than these other frameworks, but the paper’s claims seem to go beyond this.
You are right that the difference syntax-wise is really minor. We removed the text regarding this.
> Overall, the DeepDSL framework seems to have real value in its use of Scala and its memory/speed efficiency as demonstrated by the experiments, but
> the current version of the paper contains statements that overclaim novelty in ways that are misleading and unfair to existing frameworks. I will
> consider upgrading my rating if these statements are removed or amended to be more technically precise.
We have updated our paper with the above changes and we wish you would view it in a positive light.
thank you.
Thanks for the quick response!  With your latest revisions, the paper looks good to me.  I've upgraded my rating to accept.

Meta-review:
role: PC
data: comment: All reviewers find value in the contributions. 

Decision: Accept (Poster)