Exploring the Distinctions in Characteristics, Framework Designs, and Toolsets of Big Data Systems

Big Data Analytics is gaining increased recognition as a valuable tool for analyzing vast amounts of data on demand.The most common Big Data processing frameworks include Apache Flink, Apache Hadoop, Apache Storm, and Apache Spark.Each of these frameworks varies in its approach and supporting architecture, yet all facilitate Big Data processing.Numerous studies have devoted time and effort to compare these Big Data frameworks by evaluating them based on specific Key Performance Indicators (KPIs).Upon comparing all four, Apache Spark emerged as the superior choice across all the identified KPIs, such as CPU usage, task performance, execution time, processing time, and scalability for non-real-time data, compared to Apache Storm and Apache Hadoop frameworks.However, Apache Flink excelled in stream processing regarding processing time, CPU usage, latency, throughput, execution time, task performance, scalability, and fault tolerance when compared to Apache Storm and Apache Spark frameworks.This paper summarizes previous work on Apache Flink by identifying a shared set of KPIs, including processing time, CPU usage, latency, throughput, execution time, sustainable concurrency level, task performance, scalability, and fault tolerance, and compares all Big Data frameworks along these KPIs through a literature review.This comprehensive analysis of the widely-used Big Data frameworks -Apache Flink, Apache Hadoop, Apache Storm, and Apache Sparkaims to provide a clearer understanding of their respective strengths and weaknesses in various use cases.The identified KPIs serve as a guide for both researchers and practitioners when selecting the appropriate framework for their specific needs.To ensure a thorough comparison, we conducted an extensive literature review to gather relevant information and empirical data related to the KPIs.This review included various benchmarking studies, performance evaluations, and comparisons of the frameworks in different application scenarios.Our analysis revealed that, while Apache Spark outperforms Apache Storm and Apache Hadoop frameworks in non-real-time data processing, Apache Flink offers superior performance in stream processing tasks, exhibiting advantages in latency, throughput, and fault tolerance.Moreover, our findings suggest that each framework has unique features that make them suitable for particular applications.For instance, Apache Hadoop is ideal for batch processing tasks and large-scale data storage, while Apache Storm excels in real-time data processing, and Apache Spark provides a versatile platform for both batch and stream processing with a focus on in-memory data processing.In conclusion, this paper offers valuable insights into the characteristics, performance, and applicability of popular Big Data frameworks, assisting stakeholders in making informed decisions when choosing the most suitable framework for their projects.Furthermore, our comparison lays the groundwork for future research on optimizing and enhancing these frameworks to better cater to the evolving needs of Big Data processing and analytics.