A Tale of Two Convergences : Applications and Computing Platforms

There are two important types of convergence that will shape the near term future of computing sciences. The first is the convergence between HPC and Cloud platforms for science. The second is the integration between Simulations and Big Data applications. We believe understanding these trends is not just a matter of ideal speculation but is important in particular to conceptualize and design future computing platforms for Science. This paper presents our analysis of the convergence between simulations and big-data applications as well as selected research about managing the convergence between HPC and Cloud platforms. I. APPLICATIONS AND PLATFORMS: STATUS QUO On the one hand, there is a march towards the exascale computing platforms. On the other hand, there is a proliferation of software systems to support dataintensive applications with an accompanying cloud infrastructure of well publicized dramatic and increasing size and sophistication. Traditional simulations involve applications of differential equation based models that need very fine space and time steps and this leads to numerical formulations that need the memory and compute power of traditional HPC resources to solve individual problems (capability computing). A big data application does not typically need a full HPC system for However there are different types of parallelism that can/need to be exploited in order for data-intensive analytics to scale. Ref. [1] was an initial attempt to try to understand the convergence in high-performance computing and data-intensive computing. It was by necessity both high-level and broad reaching. In this paper we build upon initial work in Ref. [1] and we revisit the convergence problem by decomposing along two different trends: platforms (high-performance and cloud platforms) and applications (simulations and bigdata). Understanding these trends is important: (i) to engineer the platforms of the future, that might support both HPC and data-intensive problems, (ii) allow efficient sharing of large scale resources running simulations and data analytics; (iii) the need for higher performance Big Data algorithms; (iv) a richer software environment for research community building on many ”big data” tools, and (v) Facilitate a sustainability model for HPC, as it does not have resources to build and maintain a full software stack. II. UNDERSTANDING APPLICATIONS Needless to say there are many similarities between between data-intensive and simulation applications. Some high-level differences worth a brief mention are: • Classic Non-iterative MapReduce is major paradigm in data-intensive sciences, but it is not a common simulation paradigm except where ”reduce” summarizes pleasingly parallel execution as in some Monte Carlo simulations • Data intensive applications often have large collective communication, whereas classic simulation has a lot of smallish point-to-point messages which motivates the MapCollective model • Simulations tend to need high precision and very accurate results (partly because of differential operators), however, dataintensive problems often don’t need high accuracy as seen in trend to low precision (16 or 32 bit) deep learning networks, as there are no derivatives and the data has inevitable errors. In order to understand and analyze systematically these differences we examined extensively the landscape of applications across the HPC and dataintensive spectrum. For example in Ref. [? ], [2] on examining applications with common characteristics, we introduced the concept of Ogres, and 64 Convergence Diamonds (features). Ogres provide a means of understanding and characterizing the most common application characteristics found across the two paradigms. Ogres provide a classification and structure including, (i) classic MPI-based simulations, (ii) pleasingly parallel and workflow systems, and (iii) data-intensive applications epitomized by deep learning. Full details of Ogres and their facets can be found in Ref. [2]. We introduce four Ogres views — classification dimensions or features. These views are: 1) Problem Architecture: Related to the machine architecture needed to support application and describes properties of problem such as Pleasing Parallel or Uses Collective Communication. 2) Execution View: Describes issues such as I/O versus compute rates, iterative nature and regularity of computation and the classic Vs of Big Data defining problem size, rate of change, etc. Execution facets allow the separation of ”Data” and ”Model” for both simulations and data-intensive applications. 3) Data Source and Style views include specifying how the data is collected, stored and accessed. For example: Streaming, files versus objects, HDFS vs. Lustre 4) Processing view describe types of processing steps including nature of algorithms and kernels used by model e.g. Linear Programming, Learning, Maximum Likelihood, Spectral methods, Mesh type. It incoporates aspects of key simulation kernels and in particular includes facets seen in NAS Parallel Benchmarks and Berkeley Dwarfs Of course there are other ways of looking at the Ogres and our work should be treated as an initial suggestion for further discussion. Comparison between Data Intensive and Simulation Problems It is useful to understand the aspects of data-intensive applications that are unique and those that are similar to traditional compute-intensive simulations. In general, data-intensive applications are generally more heterogeneous than computeintensive simulation problems. Typically a data pipeline (or workflows) comprises of multiple steps: data ingest, transfer, pre-processing, several rounds of processing (e. g. for cleaning, fusing, computation of summary statistics) and advanced analytics. Each step of the pipeline can be characterized according to computational characteristics facet: (i) by the size of the input, intermediate and output data, (ii) data access pattern (sequential, random) and (iii) computational characteristics (e.g. the parallelisms deployed). The analytics part of the pipeline is compute-intensive and thus, resemble many characteristics of traditional simulations problems. For example, many analytics and machine learning problems can be formulated with linear algebra or n-body (see seven giants [3]). Thus, analytical kernels (e. g. linear algebra libraries, such as BLAS, SCALAPACK) provide the basis for data analytics. For example, machine learning algorithm, such as SVM or principal component analysis (PCA) rely on dense and sparse linear algebra. Often these analytical kernels are implemented using low-level libraries using finegrained, tightly coupled parallelism often implemented with MPI, which yield into better performance than shoehorning the problem into a rigid MapReduce programming model. However, there is also a lack of scalable analytics algorithms that are able to operate on high-dimensional, sparse datasets. We use Ogres (facets) to facilitate this comparision. There are some clear similarities: Embarassingly parallel, BSP and SPMD are common in both arenas. However, the Classic MapReduce architecture is a major Big Data paradigm, but has much less common in simulations with one example between the execution of multiple simulations (as in Quantum Monte Carlo) followed by a reduce operation to collect the results of different simulations. The Iterative Map-Collective architecture is common in Big Data analytics, such as in clustering where there is no local graph structure and the parallel algorithms involve large-scale collectives but no point to point communication. The same structure is seen in N-body (long range force) or other “all-pairs” simulations without the locality typical from discretizing differential operators. Many simulation problems have the Map-Communication architecture with numerous small point-to-point messages coming from local interactions between points defining system to be simulated. The importance of sparse data structures and algorithms is well understood in simulations and is seen in some Big Data problems such as PageRank, which calculates the leading eigenvector of the sparse matrix formed by internet site links. Other Big Data sparse data structures are seen in user-item ratings and bags of words problems. Most items are rated by few users and many documents contain a small fraction of the word vocabulary. However important data analytics involve full matrix algorithms; for example recent papers [4], [5] on a new MultiDimensional Scaling method use conjugate gradient solvers with full matrices. Note that there are similarities between some Big Data graph problems and particle simulations with an unusual potential defined by the graph node connectivity. Both use the Map-Communication architecture and the links in a Big Data graph are equivalent to strength of force between the graph nodes considered as particles. In this analogy, many Big Data problems are “long range force” corresponding to a graph where all nodes are linked to each other. As in simulation cases, these O(N2) problems are typically very compute intense but straightforward to parallelize efficiently. It is interesting to consider the analogue of the “fast multipole” methods for the fully connected Big Data problems which can dramatically improve the performance to O(N) or O(N log N). Finally note the network connections used in deep learning are sparse but in recent image interpretation studies [? ], the network weights are block sparse (corresponding to links to pixel blocks) and can be formulated as full matrix operations with GPUs and MPI running efficiently with these blocks. The above discussion focuses on a qualitative comparison of Big Data applications with traditional simulation (HPC) applications visualization, comparing the structure. As shown here there are similarities as well as points of distinction. It is likely however, that there will be significant differences in the “computational feature” fa