Query: spark memory management

title: Spark Memory Management
abstract: In order to obtain detailed information about Spark framework and realize fine grained monitoring of cluster operation information, a performance analysis system is designed. Therefore, the problems of Spark1.6 memory management scheme are researched in depth and improved. The experimental results show that the original memory management scheme is inconsistent with the requirements of Spark's official website. However, the improved memory management scheme not only meets the requirements of Spark's official website, but also makes the application run successfully under the condition of small memory capacity.

title: Spark Performance Optimization Analysis In Memory Management with Deploy Mode In Standalone Cluster Computing
abstract: As data is growing in different dimensions, it is difficult to get appropriate data analytic tools. Spark is one of high speed "in memory computing" big data analytic tool designed to improve the efficiency of data computing in both batch and realtime data analytic. Spark is memory bottleneck problem which degrades the performance of applications due to in memory computation and uses of storing intermediate and output result in memory. Investigating how performance is increased in relation to spark executor memory, number of executors, number of cores, and deploy mode parameters configuration in a standalone cluster model is our primary goal. Three representative spark applications are used as workloads to evaluates performance in relation to changing these parameters value. Experimental result show, submitting the job in cluster deploy mode is faster to finish than a submitting job in client deploy mode under two workloads. This implies spark performance does not depend on deploy mode rather it depends on types of application. However, increasing number of executor per worker, a number of core per executor and memory fraction will increase spark performance under all workloads in any deploy mode.

title: Memory Management Approaches in Apache Spark: A Review
abstract: In the era of Big Data, processing large amounts of data through data intensive applications, is presenting a challenge. An in memory distributed computing system; Apache Spark is often used to speed up big data applications. It caches intermediate data into memory, so there is no need to repeat the computation or reload data from disk when reusing these data later. This mechanism of caching data in memory makes Apache Spark much faster than other systems. When the memory used for caching data is full, the cache replacement policy used by Apache Spark is the Least Recently Used (LRU) however LRU algorithm performs poorly in some workloads. This review is going to give an insight about different replacement algorithms used to address the LRU problems, categorize the different selection factors and provide a comparison between the algorithms in terms of selection factors, performance and the benchmarks used in the research.

title: DSMM: A Dynamic Setting for Memory Management in Apache Spark
abstract: Apache Spark (Spark) is a unified analytics engine for large scale data processing. Unlike traditional data processing engines like Hadoop, Spark is a framework that caches data in memory. Therefore, memory management in Spark is importance. However, there are several factors that interfere with memory management. First, if users want to cache data in memory, they need to choose their own storage level. In this case, if they do not select the optimal storage level, Spark will be put a heavy burden on memory. Next, users need to select the ratio for spark memory directly within Spark. If they do not choose optimal ratio for spark memory, garbage collection overheads will be incurred. In this poster, we propose DSMM that dynamically select the above factors on the system for memory management. Our experimental result shows 13% execution time improvement as compared to standard Spark.

title: Safe Dynamic Memory Management in Ada and SPARK
abstract: Handling memory in a correct and efficient way is a step toward safer, less complex, and higher performing software intensive systems. However, languages used for critical software development such as Ada, which supports formal verification with its SPARK subset, face challenges regarding any use of pointers due to potential pointer aliasing. In this work, we introduce an extension to the Ada language, and to its SPARK subset, to provide pointer types "access types" in Ada) that provide provably safe, automatic storage management without any asynchronous garbage collection, and without explicit deallocation by the user. Because the mechanism for these safe pointers relies on strict control of aliasing, it can be used in the SPARK subset for formal verification, including both information flow analysis and proof of safety and correctness properties. In this paper, we present this proposal (which has been submitted for inclusion in the next version of Ada) and explain how we are able to incorporate these pointers into formal analyses.

title: SparkNei Cun Guan Li Ji Huan Cun Ce Lue Yan Jiu (Research on Memory Management and Cache Replacement Policies in Spark)

title: A Memory Management Strategy Based on Task Requirement for In Memory Computing
abstract: For in memory computing framework, the memory allocation category cannot adjust the proportion of memory according to different data processing scenarios, and there are limitations in terms of memory usage and performance. In order to solve the issue, we analyzes the spark memory model, defines the concepts of available memory amount, job execution time, data inclination, execution area demand, execution area allocation amount, execution area overflow amount, task fairness, task demand degree, etc. presents the theoretical analysis, proposes the memory optimization algorithm based on the execution area on demand allocation, and determines the memory allocation amount according to the size of task input data. On the basis of slightly modifying spark source code, the memory allocation of storage area minimization and execution area for tasks is optimized to reduce cache loss and memory overflow and improve task execution efficiency.

title: Panthera: holistic memory management for big data processing over hybrid memories
abstract: Modern data parallel systems such as Spark rely increasingly on in memory computing that can significantly improve the efficiency of iterative algorithms. To process real world datasets, modern data parallel systems often require extremely large amounts of memory, which are both costly and energy inefficient. Emerging non volatile memory (NVM) technologies offers high capacity compared to DRAM and low energy compared to SSDs. Hence, NVMs have the potential to fundamentally change the dichotomy between DRAM and durable storage in Big Data processing. However, most Big Data applications are written in managed languages (e.g. Scala and Java) and executed on top of a managed runtime (e.g. the Java Virtual Machine) that already performs various dimensions of memory management. Supporting hybrid physical memories adds in a new dimension, creating unique challenges in data replacement and migration. This paper proposes Panthera, a semantics aware, fully automated memory management technique for Big Data processing over hybrid memories. Panthera analyzes user programs on a Big Data system to infer their coarse grained access patterns, which are then passed down to the Panthera runtime for efficient data placement and migration. For Big Data applications, the coarse grained data division is accurate enough to guide GC for data layout, which hardly incurs data monitoring and moving overhead. We have implemented Panthera in OpenJDK and Apache Spark. An extensive evaluation with various datasets and applications demonstrates that Panthera reduces energy by 32 52% at only a 1 9% execution time overhead.

title: Speeding up AutoTuning of the Memory Management Options in Data Analytics
abstract: Many solutions used towards building autonomous (or, self driving) data processing systems today are trying to leverage the "black box" algorithm of Bayesian Optimization BO both due to its wider applicability and the theoretical guarantees provided on the quality of results produced. The black box approach, however, could be time and labor intensive; or otherwise get stuck in a local minima. We study an important problem of auto tuning the memory allocation for applications running on modern distributed data processing systems. A simple "white box" model is developed which can quickly separate good configurations from bad ones. To combine the benefits of the two approaches to tuning, we build a framework called Guided Bayesian Optimization GBO that uses the white box model as a guide during the Bayesian Optimization exploration process. An evaluation carried out on Apache Spark using industry standard benchmark applications shows that GBO consistently provides performance speedups across the application workload with the magnitude of savings being close to 2x.

title: Transparent In memory Cache Management in Apache Spark based on Post Mortem Analysis
abstract: This paper proposes an extension to Apache Spark that provides automated and efficient in memory cache management based on post mortem dependency graph analysis. This extension allows programmers to focus on algorithmic issues without code modification for caching decision. We realized this extension with two techniques: (1) a selection algorithm for intermediate data to be cached and (2) a cache replacement algorithm looking ahead to the whole execution. For avoiding all recalculations in case of no cache replacement, the selection algorithm implicitly activates the necessary cache directives. When cache replacement is unavoidable, the cache replacement algorithm prevents frequent cache replacement that came from excessive cache directives. Experimental results demonstrate that machine learning applications on the extended runtime achieved competitive or higher performance up to 1.3 times compared to manually optimized programs, except for very simple cases. We expect that the proposed method is useful to reduce the cache management burden on programmers without regard to the amount of processing data.