Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: While it is interesting that this can be done, and it will be useful for some, it does seem like the audience is not really the mainstream ICLR audience, who will not be afraid to use a conventional ML toolkit. 
There is no new algorithm here, nor is there any UI/meta-design improvement to make it easier for non-experts to design and train neural network systems. 

I think there will be relatively little interest at ICLR in such a paper that doesn't really advance the state of the art. 
I have no significant objection to the presentation or methodology of the paper. 

review_title: Javascript wrapper for DNN code allows training within a web browser, even using GPU. 

final_score: rating: null
review_content: What do you see as the use case for this work?

Training of models by researchers is typically initiated via a central server, so the web's client/server division is the wrong-way round for doing big compute jobs.

Fast WebCL for *deploying* trained models to be used client-side rather than server-side seems like a practical technology, but this work seems to focus specifically on training time.

The use case of distributing javascript back to web servers (node.js running on central GPU-equipped machines with data slices etc.) to do experiments would be interesting, but then the discussion of training on low-power computers in the introduction seems off-track.

Am I understanding things well?
review_title: use cases

final_score: rating: 7: Good paper, accept
review_content: This paper presents a JavaScript framework including WebCL components for training and deploying deep neural networks. The authors show that it is possible to reach competitive speeds with this technology, even higher speed than a compiled application with ViennaCL on AMD GPUs. While remaining a little more than factor three slower than compiled high performance software on NVIDIA GPUs, it offers compelling possibilities for easily deployable training and application settings for deep learning.

My main points of criticism are:
1. In Tab. 4 different batch sizes are used. Even if this is due to technical limits for the Javascript library, it would only be fair to use the smaller batch sizes for the other frameworks as well (on the GPUs probably in favor of the presented framework).

2. In Fig. 6, why not include more information in the graphs? Especially, as stated in the question, why not include the node.js values? While I do see the possible application with one server and many "low performance" clients, the setting of having a few dedicated high performance servers is quite likely. Even if not, these are good values to compare with. For the sake of consistency, please include in both subfigures Firefox, Chrome, node.js.

Apart from these points, well-written, understandable and conclusive.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Validity:
The presented work seems technically valid. Code for matrix library sushi2 and DL library sukiyaki2 are on github, including live demos that run in your browser.

https://mil-tokyo.github.io/sukiyaki2/examples/mnist/ was fun, but seemed very slow (5 mnist images per second). The demo page would be more interesting if it showed what model was being trained, which implementation was being used (pure js or webcl?), which hardware was being used for the computation, and how that compared with other people who logged into the page. As it is, the demo is kind of unclear as to what is happening.

Relevance:

The grand vision of a DLTraining@Home is exciting. While much work remains, having a solid WebCL foundation seems valuable. The big advantage of javascript is that it runs everywhere, especially on idle desktops and laptops around the world. However, these sorts of computers do not (with probability 1) have K80 or S9120 video cards. Instead, they have a wide variety of every consumer-grade card ever sold, which call for different blocking, tiling, and looping strategies in the computational kernels that underpin deep learning inference and training algorithms (hence, autotuning), which isn't discussed.

Sushi2 and Sukiyaki2 seem relatively young as projects. They are not widely followed on github, there is no tutorial-style documentation for Sukiyaki2, and the implementations of e.g. convolution do not seem to have seen much engineering work. Speed of evaluation seems to be one of the main focal points of the paper, but it’s not a major selling point to the ICLR audience because it seems about ¼ as fast as e.g. cuDNN on standard (e.g. AWS nodes) NVidia hardware. The performance of sukiyaki2 vs AMD's Caffe port is impressive.

Benchmarking on high-end compute server hardware is an interesting point of reference, but the questions that come to mind for me when reading this paper are

(1) How would this fit into a live-video processing application on a mobile device
(2) What kind of a “cluster” would this present to someone trying to do distributed deep learning in the wild by drawing on idle graphics cards: how much memory do they have, how might we handle data for training on such computers, what is the compute speed vs. communication latency and bandwidth.

Answers to these questions are out of scope for this paper, but it would have been interesting to see at least some preliminary discussion.

Novelty:
I’m not aware of a more mature WebCL-based HPC library.

Presentation:
Table 1 is hard to read because it is actually two tables with different formatting, and the numbers (speeds?) aren’t labeled with units.
review_title: null

final_score: rating: null
review_content: In Figure 6, you provide results on the "speed of training VGG16 in clients with NVIDIA K80 (Firefox browser)". Could you explain why you used the Firefox browser in this experiment and not node.js, for which you should that it is orders of magnitudes faster before? Is it, because it is harder to manage the distributed clients in that case?
review_title: Evaluation question

Meta-review: A summary of the reviews and discussion is as follows:
 
 Strengths
 Code for matrix library sushi2 and DL library sukiyaki2 are on Github, including live demos -- work is reproduceable (R2)
 Work/vision is exciting (R2)
 
 Weaknesses
 Projects preliminary (documentation, engineering of convolutions, speed, etc.) (R2)
 Perhaps not the right fit for ICLR? (R3) AC comment: ICLR specifically lists *implementation issues, parallelization, software platforms, hardware* as one of the topics of interest
 Doesn’t advance the state-of-the-art in performance (e.g. no new algorithm or UI/UX improvement) (R3)
 
 The authors responded to the pre-review questions and also the official reviews; they updated their demo and paper accordingly.
 
 Looking at the overall sentiment of the reviews, the extensive feedback from the authors, and the openness of the project I feel that it is a valuable contribution to the community. 
 
 However, given that the paper doesn't clearly advance the state of the art, the PCs believe it would be more appropriate to present it as part of the Workshop Track.
Decision: Invite to Workshop Track