Title: Development of JavaScript-based deep learning platform and application to distributed training
Authors: Masatoshi Hidaka
Ken Miura
Tatsuya Harada
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Deep learning is increasingly attracting attention for processing big data.
Existing frameworks for deep learning must be set up to specialized computer systems. Gaining sufficient computing resources therefore entails high costs of deployment and maintenance.
In this work, we implement a matrix library and deep learning framework that uses JavaScript. It can run on web browsers operating on ordinary personal computers and smartphones.
Using JavaScript, deep learning can be accomplished in widely diverse environments without the necessity for software installation. Using GPGPU from WebCL framework, our framework can train large scale convolutional neural networks such as VGGNet and ResNet.
In the experiments, we demonstrate their practicality by training VGGNet in a distributed manner using web browsers as the client.


TL;DR: Development of JavaScript-based matrix library and deep learning library which uses GPGPU. VGGNet is trained distributedly using web browsers.

Review discussion:
In Figure 6, you provide results on the "speed of training VGG16 in clients with NVIDIA K80 (Firefox browser)". Could you explain why you used the Firefox browser in this experiment and not node.js, for which you should that it is orders of magnitudes faster before? Is it, because it is harder to manage the distributed clients in that case?
Our objective is to make install-free distributed computing system for deep learning.
node.js is not considered as common software for personal computers.
Therefore, it is far from our objective to use node.js environment as computing client.
However, we need to install special software on at least one computer for providing training data and gathering trained model.
Single computer experiment using node.js assumes the situation that if we use the computer for server role as the standalone computing node.
What do you see as the use case for this work?
Training of models by researchers is typically initiated via a central server, so the web's client/server division is the wrong-way round for doing big compute jobs.
Fast WebCL for *deploying* trained models to be used client-side rather than server-side seems like a practical technology, but this work seems to focus specifically on training time.
The use case of distributing javascript back to web servers (node.js running on central GPU-equipped machines with data slices etc.) to do experiments would be interesting, but then the discussion of training on low-power computers in the introduction seems off-track.
Am I understanding things well?
The framework we suppose is:
1. the computation clients connect to the server before a concrete job is created.
2. A researcher uploads job (model structure, dataset) to the server.
3. The computation clients load the job dynamically (i.e. Ajax) and start computation.
This concept is demonstrated in sashimi framework https://github.com/mil-tokyo/sashimi in our repository.
Existing works on using multiple computers for deep learning assume there are many high-end GPU servers.
In contrast, the use case we assume is that the researcher have access to few high-end servers and many ordinary computers.
In schools and offices, there are many ordinary computers which are not used at night.
In SETI@home project, a lot of personal computers participated via internet.
We are interested in using these resources for machine learning.
Although our current experiment used eight K80 GPUs, our motivation is to utilize much more computers with less computational power.
This paper presents a JavaScript framework including WebCL components for training and deploying deep neural networks. The authors show that it is possible to reach competitive speeds with this technology, even higher speed than a compiled application with ViennaCL on AMD GPUs.
While remaining a little more than factor three slower than compiled high performance software on NVIDIA GPUs, it offers compelling possibilities for easily deployable training and application settings for deep learning.
My main points of criticism are:
1. In Tab. 4 different batch sizes are used.
Even if this is due to technical limits for the Javascript library, it would only be fair to use the smaller batch sizes for the other frameworks as well (on the GPUs probably in favor of the presented framework).
2. In Fig. 6, why not include more information in the graphs? Especially, as stated in the question, why not include the node.js values?
While I do see the possible application with one server and many "low performance" clients, the setting of having a few dedicated high performance servers is quite likely.
Even if not, these are good values to compare with. For the sake of consistency, please include in both subfigures Firefox, Chrome, node.js.
Apart from these points, well-written, understandable and conclusive.
Thank you for the review.
In Table 4 (speed comparison between our system and Caffe using single computer), we measured the speed with same batch size for all software and updated the table.
In Figure 6 (speed of distributed computing), we experimented with the situation in which node.js is the computing client as well as Firefox and updated the chart.
While it is interesting that this can be done, and it will be useful for some, it does seem like the audience is not really the mainstream ICLR audience, who will not be afraid to use a conventional ML toolkit.
There is no new algorithm here, nor is there any UI/meta-design improvement to make it easier for non-experts to design and train neural network systems.
I think there will be relatively little interest at ICLR in such a paper that doesn't really advance the state of the art.
I have no significant objection to the presentation or methodology of the paper.
Thank you for the review.
ICLR specifically lists ”Implementation issues, parallelization, software platforms, hardware” as one of their topics of interest. We think our work matches this topic.
The objective of our system is to provide accelerated deep learning system to non-experts who are not able to maintain server cluster. We are proposing a unique design for non-experts, which is different from using a cloud platform. Our system can desterilize idle resources which non-experts have. WebCL is a specification that is independent of OS and browser. The key point of our system is that when WebCL is implemented on a platform, our system immediately works on it without installation.
Additionally, although there are some matrix libraries for JavaScript, our proposed library has unique design in which basic matrix manipulation features have similar interfaces to MATLAB. It allows new users to create application easily.
The reason that the calculation speed is not reaching the state-of-the-art is due to the optimization level of low-level library (OpenCL / CUDA + cuDNN), as noted in the paper. This level of optimization is not main focus of this paper. CUDA only works with NVIDIA GPU, but OpenCL is hardware-independent, so it works with most personal computers. Caffe with OpenCL and our system achieved similar performance, so we can expect that the performance will be improved by further updates of low-level libraries.
Validity:
The presented work seems technically valid. Code for matrix library sushi2 and DL library sukiyaki2 are on github, including live demos that run in your browser.
https://mil-tokyo.github.io/sukiyaki2/examples/mnist/ was fun, but seemed very slow (5 mnist images per second). The demo page would be more interesting if it showed what model was being trained, which implementation was being used (pure js or webcl?), which hardware was being used for the computation, and how that compared with other people who logged into the page. As it is, the demo is kind of unclear as to what is happening.
Relevance:
The grand vision of a DLTraining@Home is exciting.
While much work remains, having a solid WebCL foundation seems valuable. The big advantage of javascript is that it runs everywhere, especially on idle desktops and laptops around the world.
However, these sorts of computers do not (with probability 1) have K80 or S9120 video cards.
Instead, they have a wide variety of every consumer-grade card ever sold, which call for different blocking, tiling, and looping strategies in the computational kernels that underpin deep learning inference and training algorithms (hence, autotuning), which isn't discussed.
Sushi2 and Sukiyaki2 seem relatively young as projects. They are not widely followed on github, there is no tutorial-style documentation for Sukiyaki2, and the implementations of e.g. convolution do not seem to have seen much engineering work. Speed of evaluation seems to be one of the main focal points of the paper, but it’s not a major selling point to the ICLR audience because it seems about ¼ as fast as e.g. cuDNN on standard (e.g. AWS nodes) NVidia hardware. The performance of sukiyaki2 vs AMD's Caffe port is impressive.
Benchmarking on high-end compute server hardware is an interesting point of reference, but the questions that come to mind for me when reading this paper are
(1) How would this fit into a live-video processing application on a mobile device
(2) What kind of a “cluster” would this present to someone trying to do distributed deep learning in the wild by drawing on idle graphics cards: how much memory do they have, how might we handle data for training on such computers, what is the compute speed vs. communication latency and bandwidth.
Answers to these questions are out of scope for this paper, but it would have been interesting to see at least some preliminary discussion.
Novelty:
I’m not aware of a more mature WebCL-based HPC library.
Presentation:
Table 1 is hard to read because it is actually two tables with different formatting, and the numbers (speeds?) aren’t labeled with units.
Thank you for the review.
We updated the representation of Table 1.
We updated the demo (training of MNIST).
For calculation speed in ordinary personal computers, we made estimations and added them to the end of experiments section.
For real-time processing of video in mobile devices, currently it is difficult because no mobile device supports WebCL.
Instead, primitive real-time image recognition demo using pure JavaScript is published. https://mil-tokyo.github.io/sukiyaki2/ This demo actually runs on web browsers on Android devices. Unfortunately, the browser of iPhone does not support real-time camera capture (WebRTC), so the demo cannot work on iPhone.

Meta-review:
role: PC
data: comment: A summary of the reviews and discussion is as follows:
 
 Strengths
 Code for matrix library sushi2 and DL library sukiyaki2 are on Github, including live demos -- work is reproduceable (R2)
 Work/vision is exciting (R2)
 
 Weaknesses
 Projects preliminary (documentation, engineering of convolutions, speed, etc.) (R2)
 Perhaps not the right fit for ICLR? (R3) AC comment: ICLR specifically lists *implementation issues, parallelization, software platforms, hardware* as one of the topics of interest
 Doesn’t advance the state-of-the-art in performance (e.g. no new algorithm or UI/UX improvement) (R3)
 
 The authors responded to the pre-review questions and also the official reviews; they updated their demo and paper accordingly.
 
 Looking at the overall sentiment of the reviews, the extensive feedback from the authors, and the openness of the project I feel that it is a valuable contribution to the community. 
 
 However, given that the paper doesn't clearly advance the state of the art, the PCs believe it would be more appropriate to present it as part of the Workshop Track.

Decision: Invite to Workshop Track