Title: Near-Data Processing for Machine Learning
Authors: Hyeokjun Choe
Seil Lee
Hyunha Nam
Seongsik Park
Seijoon Kim
Eui-Young Chung
Sungroh Yoon
Venue: ICLR 2017

Abstract: In computer architecture, near-data processing (NDP) refers to augmenting the memory or the storage with processing power so that it can process the data stored therein. By offloading the computational burden of CPU and saving the need for transferring raw data in its entirety, NDP exhibits a great potential for acceleration and power reduction. Despite this potential, specific research activities on NDP have witnessed only limited success until recently, often owing to performance mismatches between logic and memory process technologies that put a limit on the processing capability of memory. Recently, there have been two major changes in the game, igniting the resurgence of NDP with renewed interest. The first is the success of machine learning (ML), which often demands a great deal of computation for training, requiring frequent transfers of big data. The second is the advent of NAND flash-based solid-state drives (SSDs) containing multicore processors that can accommodate extra computation for data processing. Sparked by these application needs and technological support, we evaluate the potential of NDP for ML using a new SSD platform that allows us to simulate in-storage processing (ISP) of ML workloads. Our platform (named ISP-ML) is a full-fledged simulator of a realistic multi-channel SSD that can execute various ML algorithms using the data stored in the SSD. For thorough performance analysis and in-depth comparison with alternatives, we focus on a specific algorithm: stochastic gradient decent (SGD), which is the de facto standard for training differentiable learning machines including deep neural networks. We implement and compare three variants of SGD (synchronous, Downpour, and elastic averaging) using ISP-ML, exploiting the multiple NAND channels for parallelizing SGD. In addition, we compare the performance of ISP and that of conventional in-host processing, revealing the advantages of ISP. Based on the advantages and limitations identified through our experiments, we further discuss directions for future research on ISP for accelerating ML.


Review discussion:
Bringing computation as close to data as possible is certainly an interesting and highly relevant topic.
As a simple pre-review question, I'd like to ask how you see this (SSD-based) method evolving alongside other approaches based on, say, technological advances in data interconnects?
For example, HP just released some details about the hyped "The Machine" which is based on photonic interconnects.
Thank you for your comment. It is to the point and appropriate.
The two approaches (“the Machine” and our method) are complementary.
“The Machine” is a combination of NVM (non-volatile memory) + disaggregated memory + optical interconnect, providing a very interesting disruptive technology. The scope of “the Machine” is thus wider than that of SSD-based approaches.
In short, the SSD-based methodology (such as our method) can evolve together with other inventions for data-centric architectures (such as “the Machine”), complementing each other.
Here are more detailed comparisons and reasoning:
1) The concept of near-data processing includes “PIM (Processing in Memory)” approaches for giving intelligence to main memory (DRAM) and “ISP (In-Storage Processing)” approaches for making secondary storage (HDD/SSD) smart. As stated above, “the Machine” aims at more than PIM, but if we focus on its memory subsystem, we may categorize it as a PIM methodology. According to the specification of “the Machine” available on the web, its memory size is 240 GB.
By contrast, our proposal belongs to the category of ISP, targeting secondary storage devices (SSDs) with terabytes or more capacities. From the whole system point of view, we believe that “the Machine” and ISP techniques including ours can be placed together working in harmony.
2) In terms of the compatibility with legacy systems, as far as our understanding of “the Machine” goes, a substantial amount of new designs and developments in terms of both hardware and software will be needed to deploy “the Machine” in practice. According to the presentation material (“The Machine: An Architecture for Memory-centring Computing” presented by HP engineers at the ROSS workshop, June 2015; available at http://www.mcs.anl.gov/events/workshops/ross/2015/slides/ross2015-keeton.pdf), “the Machine” will need efforts from operating systems, data stores, analytics platforms, programming models and tools, and algorithms (page 43 of the slides).
In other words, so as to successfully unleash the full power of “the Machine” in practice, we will need to develop various new components and software.
By contrast, the proposed method aims at seamless deployment in legacy systems by “drop-in” replacements of existing SSDs and limited updates of device drivers.
3) The silicon photonics technology “the Machine” relies on can provide 1.2 terabits/second transfer data rates, which is approximately 10 times the speed of an 100G InfiniBand.
That is, there is an order of performance advantage.
However, the data growth rate in many domains exceeds this performance gain by employing silicon photonics technology. To store big data and to maintain the advantage as a memory-centric system, “the Machine” based system will still need a large amount of additional memory. Given the data rate of silicon photonics, however, it would be impossible to connect all the installed memory at once at a single flat level without creating a bottleneck, and there will thus be probably a hierarchy of memory, and we can consider incorporating ISP technology like our method into the lower hierarchy of the memory architecture. Once again, we expect that “the Machine” type systems will evolve along with ISP methods including ours, as long as a large volume of NVM is used.
4) The speed-up gain may be greater by employing “the Machine” because it is to replace most of the system not just secondary storage.
However, in terms of cost, time-to-market, backward compatibility, and many other practical factors, employing ISP approaches including ours can provide a very appealing solution to system acceleration before the era of radically different computing systems actually comes.
To summarize:
1) “The Machine” covers a wider range of a computer system than the proposed method described in this paper.
Consequently, our approach can be incorporated into memory-centric systems such as “the Machine.”
2) Our proposed method is more focused and practical with cost-effective performance gains, whereas “the Machine” is far futuristic but potentially more powerful.
To reflect your comment into the paper, I will revise it soon with a summary of the above.
Dear Authors,
My understanding is that you implemented MNIST as linear logistic regression: what error rate do you observe?
Most very large scale applications where linear classifiers are still better than DNNs, and were data transfer are more costly than weight updates involve sparse data and text: have you considered them?
Thank you for your clarifying questions.
Q1) What was the observed error rate with linear logistic regression?
A1) The test error rates we observed were approximately 0.09 for ISP (In-Storage Processing) with EASGD and 0.1 for ISP with synchronous SGD and Downpour SGD, as shown in Figure 4. For IHP (In-Host Processing), the test error rate observed was approximately 0.09, as shown in FIgure 5.
We might lower the test error rate slightly by employing better regularization methods;
however, according to the MNIST website (http://yann.lecun.com/exdb/mnist/), the test error rate of optimized single-layer logistic regression methods was reported approximately at the same level as our method. This indicates the functional validity of our logistic regression implementation.
Please refer to the last paragraph on page 7 for more details of our experiments carried out to measure the test error (or 1 - test accuracy).
Q2) Did you consider that (1) for large-scale applications data transfer takes more time than weight updates, often making linear classifiers better than DNNs and (2) often such large-scale data are sparse (like texts)?
A2) Yes, we considered them. Your point (1) is exactly the reason why we propose to use ISP, which can significantly reduce the amount of data transfers for analytics workloads, and your point (1) also reveals our rationale behind implementing logistic regression as our first proof-of-concept target for testing our platform. Your point (2) is related to our discussion in the paper regarding expanding the current platform to support DNNs, through which we will be able to handle sparse text data more effectively, for example, by implementing sparse autoencoders/word embedding for text representation and analysis.
While the idea of moving the processing for machine learning into silicon contained within the (SSD) data storage devices is intriguing and offers the potential for low-power efficient computation, it is a rather specialized topic, so I don't feel it will be of especially wide interest to the ICLR audience. The paper describes simulation results, rather than actual hardware implementation, and describes implementations of existing algorithms.
The comparisons of algorithms' train/test performance does not seem relevant (since there is no novelty in the algorithms) and the use of a single layer perceptron on MNIST calls into question the practicality of the system, since this is a tiny neural network by today's standards. I did not understand from the paper how it was thought that this could scale to contemporary scaled networks, in terms of numbers of parameters for both storage and bandwidth.
I am not an expert in this area, so have not evaluated in depth.
Thank you for your review and evaluation efforts.
It is unfortunate that some of the paper contents were not clearly presented to you, probably due to the interdisciplinary nature of this paper.
Below I provide a point-by-point response to each of your comments. I hope that my responses will be helpful for addressing the points you raised and for making the paper more understandable.
Comment 1: "it is a rather specialized topic, so I don't feel it will be of especially wide interest to the ICLR audience"
Response 1: In recent machine learning conferences (such as NIPS and ICML), ML researchers’ interest in hardware-related topics is clearly growing, and a number of companies are also developing solutions for accelerating ML (e.g., Google’s Tensor Processing Units). At this year’s NIPS that took place last week, there were multiple papers and workshops related to this topic (e.g., the "Efficient Methods for Machine Learning" workshop, where I could find a large number of audience). The list of this year’s ICLR topics available at the website (http://www.iclr.cc) also clearly includes "Implementation issues, parallelization, software platforms, hardware" (the second line from the bottom). Given this observation, I believe that there is a growing interest of the ML community in the topic this paper is about.
Comment 2: "The paper describes simulation results, rather than actual hardware implementation"
Response 2: By "hardware implementation" do you mean chip manufacturing, given that what we are proposing is a new chip architecture (SSD controller capable of ML)? Please note that we did everything we could from the designers' side (such as architecture exploration/design, functional verification, and low-level simulation for timing). Actual chip manufacturing is definitely beyond the scope of academic research and infeasible, mainly due to its prohibitively high cost. The platform we created utilizes Synopsys Platform Architect (PA), which is a professional tool actually used in chip companies to manufacture a chip. Using PA we can perform very detailed and realistic hardware design activities far better than using academic, open-source system-level simulators available. Please understand that our workflow is a standard one; no company would manufacture a semiconductor chip directly without carrying out rigorous low-level simulation. Unfortunately, asking us to show an actual hardware chip would be a request that is impossible to address.
Comment 3: "and describes implementations of existing algorithms"
Response 3: As already clearly indicated in the paper, the core contribution of this paper is our implementation of a fully functional, realistic, multi-channel intelligent SSD platform that can test the effectiveness of in-storage processing (ISP). To compare the performance of ISP with conventional in-host processing (IHP), we used widely used optimization algorithms (especially SGD and its variants that are popular). If we had used our own algorithm tailored for hardware, then the review would have been something like "the comparison is unfair, since the authors used custom algorithms optimized for hardware; the authors should use common workloads that can run in conventional machines."
Comment 4: "the use of a single layer perceptron,...how this could scale to contemporary networks.."
Response 4: Please note that our platform is general and can run other workloads than SGD and single-layer perceptron for MNIST. In this paper, we wanted to report our development of this versatile platform for accelerating machine learning, which took multi-year team efforts for completion and, I believe, deserves a publication in the ML community. For proof of concept and also for unbiased comparison with the conventional in-host processing, we thus chose to use the specific algorithms (SGD) in the current version of the paper. Through this work, we think that we have confirmed the effectiveness and potential of ISP, and the next step would be testing more diverse sets of ML algorithms.
Regarding our future work, the related explanation is already provided in the Discussion section with details. We have started extending the current work to larger scale tests including deep neural networks. This extension suggests various intriguing research directions including IHP-ISP collaboration, new memory balancing in ISP, and parallel algorithms that have distinct characteristics (such as negligible on-chip communication latency between computing nodes, unlike conventional parallel and distributed systems).
Once again, thank you for your reviewing this manuscript. Please post your additional comments if there remain uncertain points.
Combining storage and processing capabilities is an interesting research topic because data transfer is a major issue for many machine learning tasks.
The paper itself is well-written, but unfortunately addresses a lot of things only to medium depth (probably due length constraints).
My opinion is that a journal with an in-depth discussion of the technical details would be a better target for this paper.
Even though the researchers took an interesting approach to evaluate the performance of the system, it's difficult for me to grasp the expected practical improvements of this approach.
With such a big focus on GPU (and more specialized hardware such as TPUs), the one question that comes to mind: By how much does this - or do you expect it to - beat the latest and greatest GPU on a real task?
I don't consider myself an expert on this topic even though I have some experience with SystemC.
Thank you for your comments and suggestions.
As a storage device (not a computing device), intelligent SSDs are meant to complement GPUs and TPUs, not to replace them. [Please also refer to my response to your very first comment to this paper below, for more information and comparisons.] In order to utilize GPUs, we have to pump up the data stored in HDDs or SSDs first. As you pointed out correctly, a massive amount of data transfers involved in today's machine learning are slowing down the whole ML pipeline. The idea behind in-storage processing (ISP) that includes intelligent SSDs is to reduce this data transfer, sending up data upwards in the memory hierarchy selectively.
As such, please understand that we are NOT proposing to use our approach in lieu of GPUs (we cannot do that). What we are proposing is, by adopting ISP technology, the current GPU-based computing infrastructure can get improved further, by reducing the amount of the data that need to be handled by GPUs eventually.
Regarding the venue of publication, I appreciate your suggestion about sending this paper to a (probably HW-related) journal. I did consider that option, but I think that a conference would be a better place for presenting a new idea in a timely manner. The ICLR call for papers soliciting papers on hardware-related topics also guided my decision: the list of this year’s ICLR topics available at the website (http://www.iclr.cc) clearly includes "Implementation issues, parallelization, software platforms, hardware" (the second line from the bottom). Most importantly, there are contributions and findings in this paper that would be more valuable to the ML community than to the HW community.
For instance, there are different characteristics in the multiple NAND channels that can be exploited in an intriguing manner in terms of devising a parallel training algorithm (e.g., negligible on-chip communication latency, using which we can devise a new parallel SGD algorithm).
Another example is the possibility of the cross-layer optimization between in-storage processing (ISP) and in-host processing (IHP) in terms of deploying a complex models with many parameters (such as deep neural nets). Evidently, there are components that can easily be accelerated by taking advantage of intelligent storage devices, while there are components that are more suitable for the conventional processing. Implementing and optimizing such cross-layer optimization requires a set of new ML algorithms. In this regard, the Discussion section of this paper presents more discussions on future directions of this research, not only in terms of system design, but also in terms of ML algorithm development. I strongly believe that this paper is within the scope of the topics the ML community can and should understand and appreciate.
Thank you very much for your evaluation efforts. Please post your comments and questions if additional ones arise.
For more than a decade, near data processing has been a key requirement for large scale linear learning platforms, as the time to load the data exceeds the learning time, and this has justified the introduction of approaches such as Spark
Deep learning usually deals with the data that can be contained in a single machine and the bottleneck is often the CPU-GPU bus or the GPU-GPU-bus, so a method that overcomes this bottleneck could be relevant.
Unfortunately, this work is still very preliminary and limited to linear training algorithms, so of little interest yet to ICLR readership. I would recommend publication to a conference where it can reach the large-scale linear ML audience first, such as ICML. This paper is clear and well written in the present form and would probably mostly need a proper benchmark on a large scale linear task. Obviously, when the authors have convincing DNN learning simulations, they are welcome to target ICLR, but can the flash memory FPGA handle it?
For experiments, the choice of MNIST is somewhat bizarre: this task is small and performance is notoriously terrible when using linear approaches (the authors do not even report it)
Thank you for your constructive comments.
Given that most of the mainstream DNNs are based on (variants of) gradient descent, we strongly believe that the proposed framework can successfully accelerate many forms of deep learning applications. We already showed how the logistic regression (in a sense, the basic unit of DNN) could be accelerated by our approach. We also presented how parallel SGD algorithms should be modified for better results, since the multi-channel architectures inside ISP-supporting SSDs have different communication characteristics compared with conventional distributed systems.
The authors thought (and still think) that this paper would come within the scope of ICLR, because of the phrases "Implementation issues, parallelization, software platforms, hardware" included in the call for paper of ICLR.
Once again, thank you for your feedback. In the future release of our framework, we will make sure to incorporate it!

Meta-review:
role: PC
data: comment: This paper is well motivated and clearly written, and is representative of the rapidly growing interdisciplinary area of hardware-software co-design for handling large-scale Machine Learning workloads. In particular, the paper develops a detailed simulator of SSDs with onboard multicore processors so that ML computations can be done near where the data resides.
 
 Reviewers are however unanimously unconvinced about the potential impact of the simulator, and more broadly the relevance to ICLR. The empirical section of the paper is largely focused on benchmarking logistic regression models on MNIST, which reviewers find underwhelming. It is conceivable that the results reflect performance on real hardware, but the ICLR community would atleast expect to see realistic deep learning workloads on larger datasets such as Imagenet, where scalability challenges have been throughly studied. Without such results, the impact of the contribution is hard to evaluate and the claimed gains are bit of a leap of faith. 
 
 The authors make several good points in their response about the paper - that their method is expected to scale, that high quality simulations can given insights that can inform hardware manufacturing, and that their approach complements other hardware and algorithmic acceleration strategies. They are encouraged to resubmit the paper with a stronger empirical section, e.g., benchmarking training and inference of Inception-like models on ImageNet.

Decision: Reject