Query: spark

title: Spark: Cluster Computing with Working Sets
abstract: MapReduce and its variants have been highly successful in implementing large scale data intensive applications on commodity clusters. However, most of these systems are built around an acyclic data flow model that is not suitable for other popular applications. This paper focuses on one such class of applications: those that reuse a working set of data across multiple parallel operations. This includes many iterative machine learning algorithms, as well as interactive data analysis tools. We propose a new framework called Spark that supports these applications while retaining the scalability and fault tolerance of MapReduce. To achieve these goals, Spark introduces an abstraction called resilient distributed datasets (RDDs) An RDD is a read only collection of objects partitioned across a set of machines that can be rebuilt if a partition is lost. Spark can outperform Hadoop by 10x in iterative machine learning jobs, and can be used to interactively query a 39 GB dataset with sub second response time.

title: MLlib: Machine Learning in Apache Spark
abstract: Apache Spark is a popular open source platform for large scale data processing that is well suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high level API that leverages Spark's rich ecosystem to simplify the development of end to end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

title: Apache Spark

title: Preparation of mullite TiB2 CNTs hybrid composite through spark plasma sintering
abstract: A near fully dense mullite TiB2 CNTs hybrid composite was prepared successfully trough spark plasma sintering. 1 wt%CNT and 10 wt%TiB2 were mixed with nano sized mullite powders using a high energy mixer mill. Spark plasma sintering was carried out at 1350 degC under the primary and final pressure of 10 MPa and 30 MPa, respectively. XRD results showed mullite and TiB2 as dominant crystalline phases accompanied by tiny peaks of alumina. The microstructure of prepared composites demonstrated uniform distribution of TiB2 reinforcements in mullite matrix without any pores and porosities as a result of near fully densified spark plasma sintered composite. The fracture surface of composite revealed a proper bonding of TiB2 with mullite matrix and also areas with CNTs tunneling and superficies as a result of pulling out phenomenon. The flexural strength of 531 28 MPa, Vickers harness of 18.31 0.3 GPa, and fracture toughness of 5.46 0.12 MPa m 1/2 were achieved for prepared composites as the measured mechanical properties.

title: Spark SQL: Relational Data Processing in Spark
abstract: Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g. declarative queries and optimized storage) and lets SQL users call complex analytics libraries in Spark (e.g. machine learning) Compared to previous systems, Spark SQL makes two main additions. First, it offers much tighter integration between relational and procedural processing, through a declarative DataFrame API that integrates with procedural Spark code. Second, it includes a highly extensible optimizer, Catalyst, built using features of the Scala programming language, that makes it easy to add composable rules, control code generation, and define extension points. Using Catalyst, we have built a variety of features (e.g. schema inference for JSON, machine learning types, and query federation to external databases) tailored for the complex needs of modern data analysis. We see Spark SQL as an evolution of both SQL on Spark and of Spark itself, offering richer APIs and optimizations while keeping the benefits of the Spark programming model.

title: Apache Spark
abstract: This open source computing framework unifies streaming, batch, and interactive big data workloads to unlock new applications.

title: Field Assisted Sintering Technology/Spark Plasma Sintering: Mechanisms, Materials, and Technology Developments
abstract: Field assisted sintering technology/Spark plasma sintering is a low voltage, direct current (DC) pulsed current activated, pressure assisted sintering, and synthesis technique, which has been widely applied for materials processing in the recent years. After a description of its working principles and historical background, mechanical, thermal, electrical effects in FAST/SPS are presented along with the role of atmosphere. A selection of successful materials development including refractory materials, nanocrystalline functional ceramics, graded, and non equilibrium materials is then discussed. Finally, technological aspects (advanced tool concepts, temperature measurement, finite element simulations) are covered.

title: Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming
abstract: Streaming data processing has been gaining attention due to its application into a wide range of scenarios. To serve the booming demands of streaming data processing, many computation engines have been developed. However, there is still a lack of real world benchmarks that would be helpful when choosing the most appropriate platform for serving real time streaming needs. In order to address this problem, we developed a streaming benchmark for three representative computation engines: Flink, Storm and Spark Streaming. Instead of testing speed of light event processing, we construct a full data pipeline using Kafka and Redis in order to more closely mimic the real world production scenarios. Based on our experiments, we provide a performance comparison of the three data engines in terms of 99th percentile latency and throughput for various configurations.

title: Knocking combustion in spark ignition engines
abstract: Knocking combustion research is crucially important because it determines engine durability, fuel consumption, and power density, as well as noise and emission performance. Current spark ignition (SI) engines suffer from both conventional knock and super knock. Conventional knock limits raising the compression ratio to improve thermal efficiency due to end gas auto ignition, while super knock limits the desired boost to improve the power density of modern gasoline engines due to detonation. Conventional combustion has been widely studied for many years. Although the basic characteristics are clear, the correlation between the knock index and fuel chemistry, pressure oscillations and heat transfer, and auto ignition front propagation, are still in early stages of understanding. Super knock combustion in highly boosted spark ignition engines with random pre ignition events has been intensively studied in the past decade in both academia and industry. These works have mainly focused on the relationship between pre ignition and super knock, source analyses of pre ignition, and the effects of oil/fuel properties on super knock. The mechanism of super knock has been recently revealed in rapid compression machines (RCM) under engine like conditions. It was found that detonation can occur in modern internal combustion engines under high energy density conditions. Thermodynamic conditions and shock waves influence the combustion wave and detonation initiation modes. Three combustion wave modes in the end gas have been visualized as deflagration, sequential auto ignition and detonation. The most frequently observed detonation initiation mode is shock wave reflection induced detonation (SWRID) Compared to the effect of shock compression and negative temperature coefficient (NTC) combustion on ignition delay, shock wave reflection is the main cause of near wall auto ignition/detonation. Finally, suppression methods for conventional knock and super knock in SI engines are reviewed, including use of exhaust gas recirculation (EGR) the injection strategy, and the integration of a high tumble high EGR Atkinson/Miller cycle. This paper provides deep insights into the processes occurring during knocking combustion in spark ignition engines. Furthermore, knock control strategies and combustion wave modes are summarized, and future research directions, such as turbulence shock reaction interaction theory, detonation suppression and utilization, and super knock solutions, are also discussed.

title: GraphX: a resilient distributed graph system on Spark
abstract: From social networks to targeted advertising, big graphs capture the structure in data and are central to recent advances in machine learning and data mining. Unfortunately, directly applying existing data parallel tools to graph computation tasks can be cumbersome and inefficient. The need for intuitive, scalable tools for graph computation has lead to the development of new graph parallel systems (e.g. Pregel, PowerGraph) which are designed to efficiently execute graph algorithms. Unfortunately, these new graph parallel systems do not address the challenges of graph construction and transformation which are often just as problematic as the subsequent computation. Furthermore, existing graph parallel systems provide limited fault tolerance and support for interactive data mining. We introduce GraphX, which combines the advantages of both data parallel and graph parallel systems by efficiently expressing graph computation within the Spark data parallel framework. We leverage new ideas in distributed graph representation to efficiently distribute graphs as tabular data structures. Similarly, we leverage advances in data flow systems to exploit in memory computation and fault tolerance. We provide powerful new operations to simplify graph construction and transformation. Using these primitives we implement the PowerGraph and Pregel abstractions in less than 20 lines of code. Finally, by exploiting the Scala foundation of Spark, we enable users to interactively load, transform, and compute on massive graphs.