With the spreading prevalence of Big Data, many advances have recently been made in this field. Frameworks such as Apache Hadoop and Apache Spark have gained a lot of traction over the past decades and have become massively popular, especially in industries. It is becoming increasingly evident that effective big data analysis is key to solving artificial intelligence problems. Thus, a multi-algorithm library was implemented in the Spark framework, called MLlib. While this library supports multiple machine learning algorithms, there is still scope to use the Spark setup efficiently for highly time-intensive and computationally expensive procedures like deep learning. In this paper, we propose a novel framework that combines the distributive computational abilities of Apache Spark and the advanced machine learning architecture of a deep multi-layer perceptron (MLP), using the popular concept of Cascade Learning. We conduct empirical analysis of our framework on two real world datasets. The results are encouraging and corroborate our proposed framework, in turn proving that it is an improvement over traditional big data analysis methods that use either Spark or Deep learning as individual elements.
The architecture and utility of Apache Spark was first presented to the community by the authors of @cite_4 . It gives a brief overview regarding the programming model, which includes RDDs, parallel computing etc. It also introduces a few implementations in the environment. @cite_8 introduces Apache Spark’s machine learning library, MLlib. This includes its core features as well as the different components constituting the MLlib library. The work done in @cite_3 analyzes Spark’s primary framework by running a sample ML instance on it. The authors present comprehensive results based on their evaluations that highlight Spark’s advantages. The authors of @cite_1 present a similar usage of Spark by analyzing twitter data using Spark’s framework. The massive size of the twitter dataset used highlights the significance of using a tool like Spark. The authors of @cite_11 have performed similar twitter sentiment analysis using Apache Spark.
@cite_4 @cite_8 @cite_1 @cite_3 @cite_11
2189465200 2963288913 2579583823 2406975861
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLLIB provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLLIB supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLLIB has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed. Apache Spark is a distributed memory-based computing framework which is natural suitable for machine learning. Compared to Hadoop, Spark has a better ability of computing. In this paper, we analyze Spark's primary framework, core technologies, and run a machine learning instance on it. Finally, we will analyze the results and introduce our hardware equipment. Apache Spark enables fast computations and greatly accelerates analytics applications by efficiently utilizing the main memory and caching data for later use. At its core Apache Spark uses data structures called RDDs (Resilient Distributed Datasets) to give a unified view to the distributed data. However, the data represented in the RDDs remain unencrypted which can result in leakage of confidential data produced or processed by applications. Apache Spark persists (unencrypted) RDDs to the disk storage under various circumstances including but not limited to caching, RDD checkpointing and data spill during the data shuffling operations, etc. This lack of security makes Apache Spark unsuitable for processing of sensitive information that should be secured at all times. Moreover, RDDs stored in the main memory are prone to main-memory attacks such as RAM-scrapping. In this paper, we propose and develop solutions to fill-up such security lapses in the current Apache Spark framework. We present three different approaches to incorporate security in the Apache Spark framework. These approaches are designed to limit the exposure of unencrypted data during data processing, caching and data spill to disk. We use combination of cryptographic splitting and encryption to secure data stored and spilled by Apache Spark, both to the disk as well as to the main memory. Our approaches provide strong security by incorporating combination of Information Dispersal Algorithm (IDA) and Shamir's Perfect Secret Sharing (PSS). Extensive experimentation show that with appropriately chosen parameters our security approaches provide high security at a performance penalty between 10 –25 . This paper introduces G eo S park an in-memory cluster computing framework for processing large-scale spatial data. G eo S park consists of three layers: Apache Spark Layer, Spatial RDD Layer and Spatial Query Processing Layer. Apache Spark Layer provides basic Spark functionalities that include loading storing data to disk as well as regular RDD operations. Spatial RDD Layer consists of three novel Spatial Resilient Distributed Datasets (SRDDs) which extend regular Apache Spark RDDs to support geometrical and spatial objects. G eo S park provides a geometrical operations library that accesses Spatial RDDs to perform basic geometrical operations (e.g., Overlap, Intersect). System users can leverage the newly defined SRDDs to effectively develop spatial data processing programs in Spark. The Spatial Query Processing Layer efficiently executes spatial query processing algorithms (e.g., Spatial Range, Join, KNN query) on SRDDs. G eo S park also allows users to create a spatial index (e.g., R-tree, Quad-tree) that boosts spatial data processing performance in each SRDD partition. Preliminary experiments show that G eo S park achieves better run time performance than its Hadoop-based counterparts (e.g., SpatialHadoop).