# CommC: A Multi-Purpose COMModity Hardware Cluster
###### Abstract
However, acquiring and maintaining specialized hardware, such as supercomputers, remains financially challenging for many medium-sized organizations and educational institutions [5]. This limitation has led to a growing interest in utilizing commodity hardware and virtualization technologies to achieve scalable and cost-effective computing solutions. We developed a system that integrates cutting-edge open-source virtualization and cluster management technologies to address this need. Our approach involves deploying a hypervisor stack, virtualization management software, and a desktop environment on standard personal computers. This setup enables the creation of virtual machines (VMs) on each physical host, which users can seamlessly access through a VNC-based application. This ensures that users can perform their daily tasks without realizing they are interacting with a virtualized system rather than one running on physical hardware. This configuration also empowers system administrators with precise control over resource allocation for VMs on each host, allowing them to run additional tasks on the same machines. The use of virtual machines instead of physical machines provides significant security enhancements by isolating workloads effectively [6]. Each VM operates within its own sandboxed environment, ensuring that a compromise in one VM does not impact others or the host system. This isolation reduces the risk of malware propagation or unauthorized access. Additionally, VMs enable secure snapshots and rollbacks, allowing administrators to quickly restore systems to a known safe state in the event of a security breach. Virtualized environments also support advanced monitoring and access control mechanisms, simplifying real-time threat detection and response. Features like virtual network segmentation and controlled resource sharing further enhance the security and manageability of the infrastructure compared to traditional physical systems. For enhanced flexibility and easier maintenance, the physical machines are interconnected to form a cluster. This cluster setup provides numerous benefits for managing virtualized environments efficiently. By pooling resources such as CPU, RAM, and storage across multiple nodes, the cluster enhances scalability and ensures optimal utilization [7]. It facilitates seamless VM migration between nodes with minimal downtime, ensuring high availability and robust disaster recovery capabilities to maintain service continuity even during hardware failures [8]. Centralized cluster management simplifies administration by offering a unified interface to monitor and control the entire infrastructure. Moreover, shared storage solutions improve data redundancy and accessibility. With integrated tools for backup, restoration, and resource allocation, clustering enables organizations to maintain highly efficient, flexible, and reliable virtualized infrastructures. This cluster, as mentioned before, can also be exploited to run big data engines like Apache Spark. However, this cannot be achieved without a specialized cluster manager for these kinds of jobs. Our solution is to host (in the same physical machines) some additional virtual machines that form a Kubernetes [9] cluster. Using Apache Spark with Kubernetes (K8s) offers powerful advantages for managing big data workloads. Kubernetes provides a flexible, scalable, and containerized environment that simplifies deploying and orchestrating Spark applications. Unlike Spark's Standalone mode, which lacks the breadth of ecosystem integrations and advanced scheduling features, Kubernetes provides fine-grained control over resource allocation and integrates natively with DevOps workflows, CI/CD pipelines, and monitoring tools. By leveraging Kubernetes' built-in resource management, Spark clusters can scale dynamically to handle varying workloads efficiently. The integration allows seamless containerization, ensuring consistency across development, testing, and production environments.