Title: Deep Learning with Dynamic Computation Graphs
Authors: Moshe Looks
Marcello Herreshoff
DeLesley Hutchins
Peter Norvig
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Neural networks that compute over graph structures are a natural fit for problems in a variety of domains, including natural language (parse trees) and cheminformatics (molecular graphs). However, since the computation graph has a different shape and size for every input, such networks do not directly support batched training or inference. They are also difficult to implement in popular deep learning libraries, which are based on static data-flow graphs. We introduce a technique called dynamic batching, which not only batches together operations between different input graphs of dissimilar shape, but also between different nodes within a single input graph. The technique allows us to create static graphs, using popular libraries, that emulate dynamic computation graphs of arbitrary shape and size. We further present a high-level library of compositional blocks that simplifies the creation of dynamic graph models. Using the library, we demonstrate concise and batch-wise parallel implementations for a variety of models from the literature.

TL;DR: We make batching effective and easy to use for neural nets where every input may have a different shape (e.g. TreeRNNs).

Review discussion:
In the paper, you state on p. 3, last paragraph, that "Dynamic batching instantiates each operation only once, and invokes it once for each depth, so the number of kernel invocations is log(n), rather than n, where n is tree size". Probably I just did not understand a detail, but I do not see how the number of kernel invocations could scale with log(n) in arbitrary cases. Couldn't it (hypothetically) happen that at each tree level a different operation is invoked? In that case, it would still scale linearly with tree depth, or do I misunderstand?
Hi, thanks for reviewing!  If we take m to be the number of possible operations then scaling is O(m * lg(n)), so the answer to your question is that we are treating m as a constant; e.g the experiments we are describing in this section m=2; the operations being LSTM (internal nodes) and embedding lookup (leaves). Certainly if e.g. every node is a different operation then there's no way to do any batching at all!
Treating the number of possible operations as a constant seems reasonable, esp. noting that an operation can be composed of multiple ops e.g. a fully connected layer h(w*x + b) counts as one operation rather than 3 (h, *, +), and standard optimizations like operator fusion are possible as well.
The paper describes a novel technique to improve the efficiency of computation graphs in deep learning frameworks. An impressive speedup can be observed in their implementation within TensorFlow. The content is presented with sufficient clarity, although some more graphical illustrations could be useful. This work is relevant in order to achieve highest performance in neural network training.
Pros:
- significant speed improvements through dynamic batching
- source code provided
Cons:
- the effect on a large real-world (ASR, SMT) would allow the reader to put the improvements better into context
- presentation/vizualisation can be improved
Thanks for the kind review.
In terms of improving the presentation / visualization, did you have anything specific in mind? One thing that we could add would be an illustration showing the unrolled wiring diagram that you get for a batch of inputs with different shapes - we've put one together at https://goo.gl/chKVtZ . My original thought was that it had too many boxes and arrows and would be more confusing than enlightening, but if you or the other reviewers think that it is useful then we could update the paper to include it.
Regarding showing the effect of dynamic batching on a large real-world problem, I agree that this would help put the improvements in context. One caveat is that I don't think that dynamic batching has much to offer for classic seq-to-seq models where the computation graph is essentially a linear chain with no variation in structure from one example to the next. Were there particular ASR/SMT models that you had in mind where the shape of the computation graph varies from example to example? (parenthetically, applying dynamic batching and the TensorFlow Fold library to large real-world problem is *exactly* what our team is most focused on right now, although not currently ASR or SMT ... and not quite ready for publication yet, unfortunately)
Yes I agree that the diagram might be confusing and it is good to hear that you are planning to apply this to ASR. I understand that seq2seq models are not perfectly suitable for this task, but a hybrid HMM system (with frame-wise labeled observation) on a task similar to Switchboard should show a significant speedup.
The paper presents a novel strategy to deal with dynamic computation graphs. They arise, when the computation is dynamically influenced by the input data, such as in LSTMs. The authors propose an `unrolling' strategy over the operations done at every step, which allows a new kind of batching of inputs.
The presented idea is novel and the results clearly indicate the potential of the approach. For the sake of clarity of the presentation I would drop parts of Section 3 ("A combinator library for neural networks") which presents technical details that are in general interesting, but do not help the understanding of the core idea of the paper. The presented experimental results on the "Stanford Sentiment Treebank" are in my opinion not supporting the claim of the paper, which is towards speed, than a little bit confusing. It is important to point out that even though the presented ensemble "[...] variant sets a new state-of-the-art on both subtasks" [p. 8], this is not due to the framework, not even due to the model (comp. lines 4 and 2 of Tab. 2), but probably, and this can only be speculated about, due to the ensemble averaging. I would appreciate a clearer argumentation in this respect.
Update on Jan. 17th:
after the authors update for their newest revision, I increase my rating to 8 due to the again improved, now very clear argumentation.
Thanks for the kind review. You are right that some of sec. 3 and especially the SST experimental results are a bit of a tangent. My motivation for including them was that, while the core of the paper is certainly dynamic batching and the speed results, we are also claiming that dynamic batching enables implementing deep learning models (which are growing ever more complex) at a higher level of abstraction (vs. manual batching) and thus a more rapid feedback loop for trying out novel model variants (and thus obtaining superior results).
Besides the SST experiments, are there any parts of Section 3 in particular that you'd suggest dropping?
Thank you for your explanation of the paper structure, that explains for me better why you decided to include the experiment!
I totally see your point  "[...] that dynamic batching enables implementing deep learning models (which are growing ever more complex) at a higher level of abstraction (vs. manual batching) and thus a more rapid feedback loop for trying out novel model variants". This will be super hard to profoundly argue for, but the SST experiment could give an example for it. Maybe even "just" a clearer line of argumentation on why the experiment is included and what should be shown with it and that actually Tab. 3 is at least as important as Tab. 2 could already help!
I understand that space is very scarce, but both points:
* the aim of the SST experiment,
* the importance of Tab. 3 over Tab. 2,
* and the source of improvement in Tab. 2,
should really receive more attention. Maybe some space could be gained dropping (a) code example(s)? All of Section 3 argues for more compact representations and easier design of complex architectures, maybe this can be summarized in a more condensed way?
Authors describe implementation of TensorFlow Fold which allows one to run various computations without modifying computation graph. They achieve this by creating a generic scheduler as a TensorFlow computation graph, which can accept graph description as input and execute it.
They show clear benefits to this approach for tasks where computation changes for each datapoint, such as the case with TreeRNN.
In the experiments, they compare against having static batch (same graph structure repeated many times) and batch size 1.
The reason my score is 7 and not higher is because they do not provide comparison to the main alternative of their method -- someone could create a new TensorFlow graph for each dynamic batch.
In other words, instead of using their graph as the scheduling algorithm, one could explicitly create each non-uniform batch as a TensorFlow graph, and run that using standard TensorFlow.
Thanks for the kind review. The reason that we don't compare to explicitly creating TF graphs for non-uniform batches is that this would entail modifying the graph in the course of training. There are two problems with this:
1. TF graph creation/modification currently happens in pure Python, and is thus super slow
2. In distributed training, TF does not allow the graph to be modified once training has commenced (so even if #1 were resolved in the future and graph creation were fast, it still wouldn't work at scale).
Because of #1 we were careful not to measure the cost of graph creation in any of our experiments. If/when #1 is resolved (i.e. with a C/C++ API for graph creation) then it will be feasible to perform the experiments you suggest within TF (and it might be possible to perform them today using some other framework), but I don't expect #2 to change anytime soon; the assumption of an entirely static graph for distributed training seems to be baked in pretty deeply to the framework.
To put some numbers on Moshe's claim of "super slow", it takes about 17 seconds on our test machine to build and differentiate a graph using TensorFlow's single-threaded python API for the trees in the paper, vs 0.5 seconds on CPU or 0.17 seconds on GPU to run it.
Thus, building a new graph for each tree would cause a further slowdown of 30x to 100x over the numbers in the paper.
However, this slowdown is entirely due to the current implementation of the TensorFlow graph construction API, which is not optimized for this use case.  Future versions of TensorFlow, or other libraries, could potentially reduce this overhead to negligible amounts.
Thus, in the paper we measure only the intrinsic cost of the batching itself, which is caused by hardware issues such as cache, memory bandwidth, and GPU utilization, and thus applies to any library implementation.
I've just uploaded a minor revision in response to reviewer comments, in particular clarifying the intent of sections 3 and 3.5.1 with new/revised initial paragraphs. This pushed us a bit over the page limit so I moved the code example from 3.4 into an appendix per AnonReviewer1's suggestion. Thanks again AnonReviewers for your time and attention, I think this version is certainly an improvement over the original draft.
Adds \iclrfinalcopy and a link to the github repo which is now live

Meta-review:
role: PC
data: comment: All reviewers viewed the paper favorably as a nice/helpful contribution to the implementation of this important class of methods.

Decision: Accept (Poster)