Chapter 4. Server and rack solutions Hardware vendors have responded to the enthusiasm around Ceph by providing both optimized server-level and rack-level solution SKUs. Validated through joint testing with Red Hat, these solutions offer predictable price-to-performance ratios for Ceph deployments, with a convenient modular approach to expand Ceph storage for specific workloads. Typical rack-level solutions include: Network switching: Redundant network switching interconnects the cluster and provides access to clients. A minimum of one network switch is recommended. For redundancy purpose, two network switches are used, with each switch partitioned to support two network segments. Ceph MON nodes: The Ceph monitor is a datastore for the health of the entire cluster, and contains the cluster log. A minimum of three monitor nodes are strongly recommended for a cluster quorum in production. Ceph OSD hosts: Ceph OSD hosts house the storage capacity for the cluster, with one or more OSDs running per individual storage device if the device is HDD or SSD. For NVME device, Red Hat recommends to have two or more OSDs running per individual storage device. OSD hosts are selected and configured differently depending on both workload optimization and the data devices installed: HDDs, SSDs, or NVMe SSDs. Red Hat Ceph Storage: Many vendors provide a capacity-based subscription for Red Hat Ceph Storage bundled with both server and rack-level solution SKUs. Note Red Hat recommends to review the Red Hat Ceph Storage:Supported Configurations article prior to committing to any server and rack solution. Contact Red Hat support for any additional assistance. IOPS-optimized solutions With the growing use of flash storage, organizations increasingly host IOPS-intensive workloads on Ceph storage clusters to let them emulate high-performance public cloud solutions with private cloud storage. These workloads commonly involve structured data from MySQL-, MariaDB-, or PostgreSQL-based applications. Typical servers include the following elements: CPU: 6 cores per NVMe SSD, assuming a 2 GHz CPU. RAM: 16 GB baseline, plus 5 GB per OSD. Networking: 10 Gigabit Ethernet, GbE, per 2 OSDs. OSD media: High-performance, high-endurance enterprise NVMe SSDs. OSDs: Two per NVMe SSD. Bluestore WAL/DB : High-performance, high-endurance enterprise NVMe SSD, co-located with OSDs. Controller: Native PCIe bus. Note For Non-NVMe SSDs, for CPU , use two cores per SSD OSD. Table 4.1. Solutions SKUs for IOPS-optimized Ceph Workloads, by cluster size. Vendor Small (250TB) Medium (1PB) Large (2PB+) SuperMicro [a] SYS-5038MR-OSD006P N/A N/A [a] See Supermicro(R) Total Solution for Ceph for details. Throughput-optimized Solutions Throughput-optimized Ceph solutions are usually centered around semi-structured or unstructured data. Large-block sequential I/O is typical. Storage media on OSD hosts is commonly HDDs with write journals on SSD-based volumes. Typical server elements include: CPU: 0.5 cores per HDD, assuming a 2 GHz CPU. RAM: 16 GB baseline, plus 5 GB per OSD. Networking: 10 GbE per 12 OSDs each for client and cluster-facing networks. OSD media: 7,200 RPM enterprise HDDs. OSDs: One per HDD. Bluestore WAL/DB: High-endurance, high-performance enterprise serial-attached SCSI (SAS) or NVMe SSDs. Host bus adapter (HBA): Just a bunch of disks (JBOD). Several vendors provide pre-configured server and rack-level solutions for throughput-optimized Ceph workloads. Red Hat has conducted extensive testing and evaluation of servers from Supermicro and Quanta Cloud Technologies (QCT). Table 4.2. Rack-level SKUs for Ceph OSDs, MONs, and top-of-rack (TOR) switches. Vendor Small (250TB) Medium (1PB) Large (2PB+) SuperMicro [a] SRS-42E112-Ceph-03 SRS-42E136-Ceph-03 SRS-42E136-Ceph-03 Table 4.3. Individual OSD Servers Vendor Small (250TB) Medium (1PB) Large (2PB+) SuperMicro [a] SSG-6028R-OSD072P SSG-6048-OSD216P SSG-6048-OSD216P QCT [a] QxStor RCT-200 QxStor RCT-400 QxStor RCT-400 [a] See QCT: QxStor Red Hat Ceph Storage Edition for details. Table 4.4. Additional Servers Configurable for Throughput-optmized Ceph OSD Workloads. Vendor Small (250TB) Medium (1PB) Large (2PB+) Dell PowerEdge R730XD [a] DSS 7000 [b] , twin node DSS 7000, twin node Cisco UCS C240 M4 UCS C3260 [c] UCS C3260 [d] Lenovo System x3650 M5 System x3650 M5 N/A [a] See Dell PowerEdge R730xd Performance and Sizing Guide for Red Hat Ceph Storage - A Dell Red Hat Technical White Paper for details. [b] See Dell EMC DSS 7000 Performance & Sizing Guide for Red Hat Ceph Storage for details. [c] See Red Hat Ceph Storage hardware reference architecture for details. [d] See UCS C3260 for details Cost and capacity-optimized solutions Cost- and capacity-optimized solutions typically focus on higher capacity, or longer archival scenarios. Data can be either semi-structured or unstructured. Workloads include media archives, big data analytics archives, and machine image backups. Large-block sequential I/O is typical. For greater cost effectiveness, OSDs are usually hosted on HDDs with Ceph write journals co-located on the HDDs. Solutions typically include the following elements: CPU: 0.5 cores per HDD, assuming a 2 GHz CPU. RAM: 16 GB baseline, plus 5 GB per OSD. Networking: 10 GbE per 12 OSDs (each for client- and cluster-facing networks). OSD media: 7,200 RPM enterprise HDDs. OSDs: One per HDD. Bluestore WAL/DB: Co-located on the HDD. HBA: JBOD. Supermicro and QCT provide pre-configured server and rack-level solution SKUs for cost- and capacity-focused Ceph workloads. Table 4.5. Pre-configured Rack-level SKUs for Cost- and Capacity-optimized Workloads Vendor Small (250TB) Medium (1PB) Large (2PB+) SuperMicro [a] N/A SRS-42E136-Ceph-03 SRS-42E172-Ceph-03 Table 4.6. Pre-configured Server-level SKUs for Cost- and Capacity-optimized Workloads Vendor Small (250TB) Medium (1PB) Large (2PB+) SuperMicro [a] N/A SSG-6048R-OSD216P [a] SSD-6048R-OSD360P QCT N/A QxStor RCC-400 [a] QxStor RCC-400 [a] [a] See Supermicro's Total Solution for Ceph for details. Table 4.7. Additional Servers Configurable for Cost- and Capacity-optimized Workloads Vendor Small (250TB) Medium (1PB) Large (2PB+) Dell N/A DSS 7000, twin node DSS 7000, twin node Cisco N/A UCS C3260 UCS C3260 Lenovo N/A System x3650 M5 N/A Additional Resources Red Hat Ceph Storage on Samsung NVMe SSDs Deploying MySQL Databases on Red Hat Ceph Storage Intel(R) Data Center Blocks for Cloud - Red Hat OpenStack Platform with Red Hat Ceph Storage Red Hat Ceph Storage on QCT Servers Red Hat Ceph Storage on Servers with Intel Processors and SSDs Red Hat Data Services Solutions

Commands: