Chapter 4. Telco RAN DU reference design specifications The telco RAN DU reference design specifications (RDS) describes the configuration for clusters running on commodity hardware to host 5G workloads in the Radio Access Network (RAN). It captures the recommended, tested, and supported configurations to get reliable and repeatable performance for a cluster running the telco RAN DU profile. 4.1. Reference design specifications for telco RAN DU 5G deployments Red Hat and certified partners offer deep technical expertise and support for networking and operational capabilities required to run telco applications on OpenShift Container Platform 4.18 clusters. Red Hat's telco partners require a well-integrated, well-tested, and stable environment that can be replicated at scale for enterprise 5G solutions. The telco core and RAN DU reference design specifications (RDS) outline the recommended solution architecture based on a specific version of OpenShift Container Platform. Each RDS describes a tested and validated platform configuration for telco core and RAN DU use models. The RDS ensures an optimal experience when running your applications by defining the set of critical KPIs for telco 5G core and RAN DU. Following the RDS minimizes high severity escalations and improves application stability. 5G use cases are evolving and your workloads are continually changing. Red Hat is committed to iterating over the telco core and RAN DU RDS to support evolving requirements based on customer and partner feedback. The reference configuration includes the configuration of the far edge clusters and hub cluster components. The reference configurations in this document are deployed using a centrally managed hub cluster infrastructure as shown in the following image. Figure 4.1. Telco RAN DU deployment architecture 4.2. Reference design scope The telco core and telco RAN reference design specifications (RDS) capture the recommended, tested, and supported configurations to get reliable and repeatable performance for clusters running the telco core and telco RAN profiles. Each RDS includes the released features and supported configurations that are engineered and validated for clusters to run the individual profiles. The configurations provide a baseline OpenShift Container Platform installation that meets feature and KPI targets. Each RDS also describes expected variations for each individual configuration. Validation of each RDS includes many long duration and at-scale tests. Note The validated reference configurations are updated for each major Y-stream release of OpenShift Container Platform. Z-stream patch releases are periodically re-tested against the reference configurations. 4.3. Deviations from the reference design Deviating from the validated telco core and telco RAN DU reference design specifications (RDS) can have significant impact beyond the specific component or feature that you change. Deviations require analysis and engineering in the context of the complete solution. Important All deviations from the RDS should be analyzed and documented with clear action tracking information. Due diligence is expected from partners to understand how to bring deviations into line with the reference design. This might require partners to provide additional resources to engage with Red Hat to work towards enabling their use case to achieve a best in class outcome with the platform. This is critical for the supportability of the solution and ensuring alignment across Red Hat and with partners. Deviation from the RDS can have some or all of the following consequences: It can take longer to resolve issues. There is a risk of missing project service-level agreements (SLAs), project deadlines, end provider performance requirements, and so on. Unapproved deviations may require escalation at executive levels. Note Red Hat prioritizes the servicing of requests for deviations based on partner engagement priorities. 4.4. Engineering considerations for the RAN DU use model The RAN DU use model configures an OpenShift Container Platform cluster running on commodity hardware for hosting RAN distributed unit (DU) workloads. Model and system level considerations are described below. Specific limits, requirements and engineering considerations for individual components are detailed in later sections. Note For details of the RAN DU KPI test results, see the Telco RAN DU reference design specification KPI test results for OpenShift 4.18 . This information is only available to customers and partners. Workloads DU workloads are described in "Telco RAN DU application workloads". DU worker nodes are Intel 3rd Generation Xeon (IceLake) 2.20 GHz or better with host firmware tuned for maximum performance. Resources The maximum number of running pods in the system, inclusive of application workload and OpenShift Container Platform pods, is 120. Resource utilization OpenShift Container Platform resource utilization varies depending on many factors such as the following application workload characteristics: Pod count Type and frequency of probes Messaging rates on the primary or secondary CNI with kernel networking API access rate Logging rates Storage IOPS Resource utilization is measured for clusters configured as follows: The cluster is a single host with single-node OpenShift installed. The cluster runs the representative application workload described in "Reference application workload characteristics". The cluster is managed under the constraints detailed in "Hub cluster management characteristics". Components noted as "optional" in the use model configuration are not included. Note Configuration outside the scope of the RAN DU RDS that do not meet these criteria requires additional analysis to determine the impact on resource utilization and ability to meet KPI targets. You might need to allocate additional cluster resources to meet these requirements. Reference application workload characteristics Uses 15 pods and 30 containers for the vRAN application including its management and control functions Uses an average of 2 ConfigMap and 4 Secret CRs per pod Uses a maximum of 10 exec probes with a frequency of not less than 10 seconds Incremental application load on the kube-apiserver is less than or equal to 10% of the cluster platform usage Note You can extract CPU load can from the platform metrics. For example: USD query=avg_over_time(pod:container_cpu_usage:sum{namespace="openshift-kube-apiserver"}[30m]) Application logs are not collected by the platform log collector Aggregate traffic on the primary CNI is less than 8 MBps Hub cluster management characteristics RHACM is the recommended cluster management solution and is configured to these limits: Use a maximum of 5 RHACM configuration policies with a compliant evaluation interval of not less than 10 minutes. Use a minimal number (up to 10) of managed cluster templates in cluster policies. Use hub-side templating. Disable RHACM addons with the exception of the policyController and configure observability with the default configuration. The following table describes resource utilization under reference application load. Table 4.1. Resource utilization under reference application load Metric Limits Notes OpenShift platform CPU usage Less than 4000mc - 2 cores (4HT) Platform CPU is pinned to reserved cores, including both hyper-threads of each reserved core. The system is engineered to 3 CPUs (3000mc) at steady-state to allow for periodic system tasks and spikes. OpenShift Platform memory Less than 16G 4.5. Telco RAN DU application workloads Develop RAN DU applications that are subject to the following requirements and limitations. Description and limits Develop cloud-native network functions (CNFs) that conform to the latest version of Red Hat best practices for Kubernetes . Use SR-IOV for high performance networking. Use exec probes sparingly and only when no other suitable options are available. Do not use exec probes if a CNF uses CPU pinning. Use other probe implementations, for example, httpGet or tcpSocket . When you need to use exec probes, limit the exec probe frequency and quantity. The maximum number of exec probes must be kept below 10, and frequency must not be set to less than 10 seconds. Exec probes cause much higher CPU usage on management cores compared to other probe types because they require process forking. Note Startup probes require minimal resources during steady-state operation. The limitation on exec probes applies primarily to liveness and readiness probes. Note A test workload that conforms to the dimensions of the reference DU application workload described in this specification can be found at openshift-kni/du-test-workloads . 4.6. Telco RAN DU reference design components The following sections describe the various OpenShift Container Platform components and configurations that you use to configure and deploy clusters to run RAN DU workloads. Figure 4.2. Telco RAN DU reference design components Note Ensure that additional components you include that are not specified in the telco RAN DU profile do not affect the CPU resources allocated to workload applications. Important Out of tree drivers are not supported. 5G RAN application components are not included in the RAN DU profile and must be engineered against resources (CPU) allocated to applications. 4.6.1. Host firmware tuning New in this release No reference design updates in this release Description Tune host firmware settings for optimal performance during initial cluster deployment. For more information, see "Recommended single-node OpenShift cluster configuration for vDU application workloads". Apply tuning settings in the host firmware during initial deployment. See "Managing host firmware settings with GitOps ZTP" for more information. The managed cluster host firmware settings are available on the hub cluster as individual BareMetalHost custom resources (CRs) that are created when you deploy the managed cluster with the ClusterInstance CR and GitOps ZTP. Note Create the ClusterInstance CR based on the provided reference example-sno.yaml CR. Limits and requirements You must enable Hyper-Threading in the host firmware settings Engineering considerations Tune all firmware settings for maximum performance. All settings are expected to be for maximum performance unless tuned for power savings. You can tune host firmware for power savings at the expense of performance as required. Enable secure boot. When secure boot is enabled, only signed kernel modules are loaded by the kernel. Out-of-tree drivers are not supported. Additional resources Managing host firmware settings with GitOps ZTP Configuring host firmware for low latency and high performance Provisioning real-time and low latency workloads 4.6.2. CPU partitioning and performance tuning New in this release No reference design updates in this release Description The RAN DU use model includes cluster performance tuning via PerformanceProfile CRs for low-latency performance. The PerformanceProfile CRs are reconciled by the Node Tuning Operator. The RAN DU use case requires the cluster to be tuned for low-latency performance. For more details about node tuning with the PerformanceProfile CR, see "Tuning nodes for low latency with the performance profile". Limits and requirements The Node Tuning Operator uses the PerformanceProfile CR to configure the cluster. You need to configure the following settings in the telco RAN DU profile PerformanceProfile CR: Set a reserved cpuset of 4 or more, equating to 4 hyper-threads (2 cores) for either of the following CPUs: Intel 3rd Generation Xeon (IceLake) 2.20 GHz or better CPUs with host firmware tuned for maximum performance AMD EPYC Zen 4 CPUs (Genoa, Bergamo, or newer) Note AMD EPYC Zen 4 CPUs (Genoa, Bergamo, or newer) are fully supported. Power consumption evaluations are ongoing. It is recommended to evaluate features, such as per-pod power management, to determine any potential impact on performance. Set the reserved cpuset to include both hyper-thread siblings for each included core. Unreserved cores are available as allocatable CPU for scheduling workloads. Ensure that hyper-thread siblings are not split across reserved and isolated cores. Ensure that reserved and isolated CPUs include all the threads for all cores in the CPU. Include Core 0 for each NUMA node in the reserved CPU set. Set the huge page size to 1G. Only pin OpenShift Container Platform pods which are by default configured as part of the management workload partition to reserved cores. Engineering considerations Meeting the full performance metrics requires use of the RT kernel. If required, you can use the non-RT kernel with corresponding impact to performance. The number of hugepages you configure depends on application workload requirements. Variation in this parameter is expected and allowed. Variation is expected in the configuration of reserved and isolated CPU sets based on selected hardware and additional components in use on the system. The variation must still meet the specified limits. Hardware without IRQ affinity support affects isolated CPUs. To ensure that pods with guaranteed whole CPU QoS have full use of allocated CPUs, all hardware in the server must support IRQ affinity. When workload partitioning is enabled by setting cpuPartitioningMode to AllNodes during deployment, you must allocate enough CPUs to support the operating system, interrupts, and OpenShift Container Platform pods in the PerformanceProfile CR. Additional resources Finding the effective IRQ affinity setting for a node 4.6.3. PTP Operator New in this release No reference design updates in this release Description Configure PTP in cluster nodes with PTPConfig CRs for the RAN DU use case with features like Grandmaster clock (T-GM) support via GPS, ordinary clock (OC), boundary clocks (T-BC), dual boundary clocks, high availability (HA), and optional fast event notification over HTTP. PTP ensures precise timing and reliability in the RAN environment. Limits and requirements Limited to two boundary clocks for nodes with dual NICs and HA Limited to two Westport channel NIC configurations for T-GM Engineering considerations RAN DU RDS configurations are provided for ordinary clocks, boundary clocks, grandmaster clocks, and highly available dual NIC boundary clocks. PTP fast event notifications use ConfigMap CRs to persist subscriber details. Hierarchical event subscription as described in the O-RAN specification is not supported for PTP events. Use the PTP fast events REST API v2. The PTP fast events REST API v1 is deprecated. The REST API v2 is O-RAN Release 3 compliant. 4.6.4. SR-IOV Operator New in this release No reference design updates in this release Description The SR-IOV Operator provisions and configures the SR-IOV CNI and device plugins. Both netdevice (kernel VFs) and vfio (DPDK) devices are supported and applicable to the RAN DU use models. Limits and requirements Use devices that are supported for OpenShift Container Platform. See "Supported devices". SR-IOV and IOMMU enablement in host firmware settings: The SR-IOV Network Operator automatically enables IOMMU on the kernel command line. SR-IOV VFs do not receive link state updates from the PF. If link down detection is required you must configure this at the protocol level. Engineering considerations SR-IOV interfaces with the vfio driver type are typically used to enable additional secondary networks for applications that require high throughput or low latency. Customer variation on the configuration and number of SriovNetwork and SriovNetworkNodePolicy custom resources (CRs) is expected. IOMMU kernel command line settings are applied with a MachineConfig CR at install time. This ensures that the SriovOperator CR does not cause a reboot of the node when adding them. SR-IOV support for draining nodes in parallel is not applicable in a single-node OpenShift cluster. You must include the SriovOperatorConfig CR in your deployment; the CR is not created automatically. This CR is included in the reference configuration policies which are applied during initial deployment. In scenarios where you pin or restrict workloads to specific nodes, the SR-IOV parallel node drain feature will not result in the rescheduling of pods. In these scenarios, the SR-IOV Operator disables the parallel node drain functionality. NICs which do not support firmware updates under secure boot or kernel lockdown must be pre-configured with sufficient virtual functions (VFs) to support the number of VFs needed by the application workload. For Mellanox NICs, the Mellanox vendor plugin must be disabled in the SR-IOV Network Operator. For more information, see "Configuring an SR-IOV network device". To change the MTU value of a virtual function after the pod has started, do not configure the MTU field in the SriovNetworkNodePolicy CR. Instead, configure the Network Manager or use a custom systemd script to set the MTU of the physical function to an appropriate value. For example: # ip link set dev <physical_function> mtu 9000 Additional resources Supported devices Configuring QinQ support for SR-IOV enabled workloads 4.6.5. Logging New in this release No reference design updates in this release Description Use logging to collect logs from the far edge node for remote analysis. The recommended log collector is Vector. Engineering considerations Handling logs beyond the infrastructure and audit logs, for example, from the application workload requires additional CPU and network bandwidth based on additional logging rate. As of OpenShift Container Platform 4.14, Vector is the reference log collector. Use of fluentd in the RAN use models is deprecated. Additional resources Logging 6.0 4.6.6. SRIOV-FEC Operator New in this release No reference design updates in this release Description SRIOV-FEC Operator is an optional 3rd party Certified Operator supporting FEC accelerator hardware. Limits and requirements Starting with FEC Operator v2.7.0: Secure boot is supported vfio drivers for PFs require the usage of a vfio-token that is injected into the pods. Applications in the pod can pass the VF token to DPDK by using EAL parameter --vfio-vf-token . Engineering considerations The SRIOV-FEC Operator uses CPU cores from the isolated CPU set. You can validate FEC readiness as part of the pre-checks for application deployment, for example, by extending the validation policy. Additional resources SRIOV-FEC Operator for Intel(R) vRAN Dedicated Accelerator manager container 4.6.7. Lifecycle Agent New in this release No reference design updates in this release Description The Lifecycle Agent provides local lifecycle management services for single-node OpenShift clusters. Limits and requirements The Lifecycle Agent is not applicable in multi-node clusters or single-node OpenShift clusters with an additional worker. The Lifecycle Agent requires a persistent volume that you create when installing the cluster. For descriptions of partition requirements, see "Configuring a shared container directory between ostree stateroots when using GitOps ZTP". Additional resources Understanding the image-based upgrade for single-node OpenShift clusters Configuring a shared container directory between ostree stateroots when using GitOps ZTP 4.6.8. Local Storage Operator New in this release No reference design updates in this release Description You can create persistent volumes that can be used as PVC resources by applications with the Local Storage Operator. The number and type of PV resources that you create depends on your requirements. Engineering considerations Create backing storage for PV CRs before creating the PV . This can be a partition, a local volume, LVM volume, or full disk. Refer to the device listing in LocalVolume CRs by the hardware path used to access each device to ensure correct allocation of disks and partitions, for example, /dev/disk/by-path/<id> . Logical names (for example, /dev/sda ) are not guaranteed to be consistent across node reboots. 4.6.9. Logical Volume Manager Storage New in this release No reference design updates in this release Description Logical Volume Manager (LVM) Storage is an optional component. It provides dynamic provisioning of both block and file storage by creating logical volumes from local devices that can be consumed as persistent volume claim (PVC) resources by applications. Volume expansion and snapshots are also possible. An example configuration is provided in the RDS with the StorageLVMCluster.yaml file. Limits and requirements In single-node OpenShift clusters, persistent storage must be provided by either LVM Storage or local storage, not both. Volume snapshots are excluded from the reference configuration. Engineering considerations LVM Storage can be used as the local storage implementation for the RAN DU use case. When LVM Storage is used as the storage solution, it replaces the Local Storage Operator, and the CPU required is assigned to the management partition as platform overhead. The reference configuration must include one of these storage solutions but not both. Ensure that sufficient disks or partitions are available for storage requirements. 4.6.10. Workload partitioning New in this release No reference design updates in this release Description Workload partitioning pins OpenShift Container Platform and Day 2 Operator pods that are part of the DU profile to the reserved CPU set and removes the reserved CPU from node accounting. This leaves all unreserved CPU cores available for user workloads. This leaves all non-reserved CPU cores available for user workloads. Workload partitioning is enabled through a capability set in installation parameters: cpuPartitioningMode: AllNodes . The set of management partition cores are set with the reserved CPU set that you configure in the PerformanceProfile CR. Limits and requirements Namespace and Pod CRs must be annotated to allow the pod to be applied to the management partition Pods with CPU limits cannot be allocated to the partition. This is because mutation can change the pod QoS. For more information about the minimum number of CPUs that can be allocated to the management partition, see "Node Tuning Operator". Engineering considerations Workload partitioning pins all management pods to reserved cores. A sufficient number of cores must be allocated to the reserved set to account for operating system, management pods, and expected spikes in CPU use that occur when the workload starts, the node reboots, or other system events happen. Additional resources Workload partitioning 4.6.11. Cluster tuning New in this release No reference design updates in this release Description See "Cluster capabilities" for a full list of components that can be disabled by using the cluster capabilities feature. Limits and requirements Cluster capabilities are not available for installer-provisioned installation methods. Engineering considerations In clusters running OpenShift Container Platform 4.16 and later, the cluster does not automatically revert to cgroup v1 when a PerformanceProfile is applied. If workloads running on the cluster require cgroup v1, the cluster must be configured for cgroup v1. For more information, see "Enabling Linux control group version 1 (cgroup v1)". You should make this configuration as part of the initial cluster deployment. Note Support for cgroup v1 is planned for removal in OpenShift Container Platform 4.19. Clusters running cgroup v1 must transition to cgroup v2. The following table lists the required platform tuning configurations: Table 4.2. Cluster capabilities configurations Feature Description Remove optional cluster capabilities Reduce the OpenShift Container Platform footprint by disabling optional cluster Operators on single-node OpenShift clusters only. Remove all optional Operators except the Node Tuning Operator, Operator Lifecycle Manager, and the Ingress Operator. Configure cluster monitoring Configure the monitoring stack for reduced footprint by doing the following: Disable the local alertmanager and telemeter components. If you use RHACM observability, the CR must be augmented with appropriate additionalAlertManagerConfigs CRs to forward alerts to the hub cluster. Reduce the Prometheus retention period to 24h. Note The RHACM hub cluster aggregates managed cluster metrics. Disable networking diagnostics Disable networking diagnostics for single-node OpenShift because they are not required. Configure a single OperatorHub catalog source Configure the cluster to use a single catalog source that contains only the Operators required for a RAN DU deployment. Each catalog source increases the CPU use on the cluster. Using a single CatalogSource fits within the platform CPU budget. Disable the Console Operator If the cluster was deployed with the console disabled, the Console CR ( ConsoleOperatorDisable.yaml ) is not needed. If the cluster was deployed with the console enabled, you must apply the Console CR. Additional resources Cluster capabilities 4.6.12. Machine configuration New in this release No reference design updates in this release Limits and requirements The CRI-O wipe disable MachineConfig CR assumes that images on disk are static other than during scheduled maintenance in defined maintenance windows. To ensure the images are static, do not set the pod imagePullPolicy field to Always . Table 4.3. Machine configuration options Feature Description Container Runtime Sets the container runtime to crun for all node roles. Kubelet config and container mount namespace hiding Reduces the frequency of kubelet housekeeping and eviction monitoring, which reduces CPU usage SCTP Optional configuration (enabled by default) Kdump Optional configuration (enabled by default) Enables kdump to capture debug information when a kernel panic occurs. The reference CRs that enable kdump have an increased memory reservation based on the set of drivers and kernel modules included in the reference configuration. CRI-O wipe disable Disables automatic wiping of the CRI-O image cache after unclean shutdown SR-IOV-related kernel arguments Include additional SR-IOV-related arguments in the kernel command line Set RCU Normal Systemd service that sets rcu_normal after the system finishes startup One-shot time sync Runs a one-time NTP system time synchronization job for control plane or worker nodes. Additional resources Recommended single-node OpenShift cluster configuration for vDU application workloads . 4.7. Telco RAN DU deployment components The following sections describe the various OpenShift Container Platform components and configurations that you use to configure the hub cluster with RHACM. 4.7.1. Red Hat Advanced Cluster Management New in this release No reference design updates in this release Description RHACM provides Multi Cluster Engine (MCE) installation and ongoing lifecycle management functionality for deployed clusters. You manage cluster configuration and upgrades declaratively by applying Policy custom resources (CRs) to clusters during maintenance windows. RHACM provides the following functionality: Zero touch provisioning (ZTP) of clusters using the MCE component in RHACM. Configuration, upgrades, and cluster status through the RHACM policy controller. During managed cluster installation, RHACM can apply labels to individual nodes as configured through the ClusterInstance CR. Limits and requirements A single hub cluster supports up to 3500 deployed single-node OpenShift clusters with 5 Policy CRs bound to each cluster. Engineering considerations Use RHACM policy hub-side templating to better scale cluster configuration. You can significantly reduce the number of policies by using a single group policy or small number of general group policies where the group and per-cluster values are substituted into templates. Cluster specific configuration: managed clusters typically have some number of configuration values that are specific to the individual cluster. These configurations should be managed using RHACM policy hub-side templating with values pulled from ConfigMap CRs based on the cluster name. To save CPU resources on managed clusters, policies that apply static configurations should be unbound from managed clusters after GitOps ZTP installation of the cluster. Additional resources Using GitOps ZTP to provision clusters at the network far edge Red Hat Advanced Cluster Management for Kubernetes 4.7.2. SiteConfig Operator New in this release No RDS updates in this release Description The SiteConfig Operator is a template-driven solution designed to provision clusters through various installation methods. It introduces the unified ClusterInstance API, which replaces the deprecated SiteConfig API. By leveraging the ClusterInstance API, the SiteConfig Operator improves cluster provisioning by providing the following: Better isolation of definitions from installation methods Unification of Git and non-Git workflows Consistent APIs across installation methods Enhanced scalability Increased flexibility with custom installation templates Valuable insights for troubleshooting deployment issues The SiteConfig Operator provides validated default installation templates to facilitate cluster deployment through both the Assisted Installer and Image-based Installer provisioning methods: Assisted Installer automates the deployment of OpenShift Container Platform clusters by leveraging predefined configurations and validated host setups. It ensures that the target infrastructure meets OpenShift Container Platform requirements. The Assisted Installer streamlines the installation process while minimizing time and complexity compared to manual setup. Image-based Installer expedites the deployment of single-node OpenShift clusters by utilizing preconfigured and validated OpenShift Container Platform seed images. Seed images are preinstalled on target hosts, enabling rapid reconfiguration and deployment. The Image-based Installer is particularly well-suited for remote or disconnected environments, because it simplifies the cluster creation process and significantly reduces deployment time. Limits and requirements A single hub cluster supports up to 3500 deployed single-node OpenShift clusters. 4.7.3. Topology Aware Lifecycle Manager New in this release No reference design updates in this release Description Topology Aware Lifecycle Manager is an Operator that runs only on the hub cluster for managing how changes like cluster upgrades, Operator upgrades, and cluster configuration are rolled out to the network. TALM supports the following features: Progressive rollout of policy updates to fleets of clusters in user configurable batches. Per-cluster actions add ztp-done labels or other user-configurable labels following configuration changes to managed clusters. Precaching of single-node OpenShift clusters images: TALM supports optional pre-caching of OpenShift, OLM Operator, and additional user images to single-node OpenShift clusters before initiating an upgrade. The precaching feature is not applicable when using the recommended image-based upgrade method for upgrading single-node OpenShift clusters. Specifying optional pre-caching configurations with PreCachingConfig CRs. Review the sample reference PreCachingConfig CR for more information. Excluding unused images with configurable filtering. Enabling before and after pre-caching storage space validations with configurable space-required parameters. Limits and requirements Supports concurrent cluster deployment in batches of 400 Pre-caching and backup are limited to single-node OpenShift clusters only Engineering considerations The PreCachingConfig CR is optional and does not need to be created if you only need to precache platform-related OpenShift and OLM Operator images. The PreCachingConfig CR must be applied before referencing it in the ClusterGroupUpgrade CR. Only policies with the ran.openshift.io/ztp-deploy-wave annotation are automatically applied by TALM during cluster installation. Any policy can be remediated by TALM under control of a user created ClusterGroupUpgrade CR. Additional resources Updating managed clusters with the Topology Aware Lifecycle Manager 4.7.4. GitOps Operator and GitOps ZTP New in this release No reference design updates in this release Description GitOps Operator and GitOps ZTP provide a GitOps-based infrastructure for managing cluster deployment and configuration. Cluster definitions and configurations are maintained as a declarative state in Git. You can apply ClusterInstance CRs to the hub cluster where the SiteConfig Operator renders them as installation CRs. In earlier releases, a GitOps ZTP plugin supported the generation of installation CRs from SiteConfig CRs. This plugin is now deprecated. A separate GitOps ZTP plugin is available to enable automatic wrapping of configuration CRs into policies based on the PolicyGenerator or PolicyGenTemplate CR. You can deploy and manage multiple versions of OpenShift Container Platform on managed clusters by using the baseline reference configuration CRs. You can use custom CRs alongside the baseline CRs. To maintain multiple per-version policies simultaneously, use Git to manage the versions of the source and policy CRs by using PolicyGenerator or PolicyGenTemplate CRs. Limits and requirements 300 ClusterInstance CRs per ArgoCD application. Multiple applications can be used to achieve the maximum number of clusters supported by a single hub cluster Content in the source-crs/ directory in Git overrides content provided in the ZTP plugin container, as Git takes precedence in the search path. The source-crs/ directory is specifically expected to be located in the same directory as the kustomization.yaml file, which includes PolicyGenerator or PolicyGenTemplate CRs as a generator. Alternative locations for the source-crs/ directory are not supported in this context. Engineering considerations For multi-node cluster upgrades, you can pause MachineConfigPool ( MCP ) CRs during maintenance windows by setting the paused field to true . You can increase the number of simultaneously updated nodes per MCP CR by configuring the maxUnavailable setting in the MCP CR. The MaxUnavailable field defines the percentage of nodes in the pool that can be simultaneously unavailable during a MachineConfig update. Set maxUnavailable to the maximum tolerable value. This reduces the number of reboots in a cluster during upgrades which results in shorter upgrade times. When you finally unpause the MCP CR, all the changed configurations are applied with a single reboot. During cluster installation, you can pause custom MCP CRs by setting the paused field to true and setting maxUnavailable to 100% to improve installation times. Keep reference CRs and custom CRs under different directories. Doing this allows you to patch and update the reference CRs by simple replacement of all directory contents without touching the custom CRs. When managing multiple versions, the following best practices are recommended: Keep all source CRs and policy creation CRs in Git repositories to ensure consistent generation of policies for each OpenShift Container Platform version based solely on the contents in Git. Keep reference source CRs in a separate directory from custom CRs. This facilitates easy update of reference CRs as required. To avoid confusion or unintentional overwrites when updating content, it is highly recommended to use unique and distinguishable names for custom CRs in the source-crs/ directory and extra manifests in Git. Extra installation manifests are referenced in the ClusterInstance CR through a ConfigMap CR. The ConfigMap CR should be stored alongside the ClusterInstance CR in Git, serving as the single source of truth for the cluster. If needed, you can use a ConfigMap generator to create the ConfigMap CR. Additional resources Preparing the GitOps ZTP site configuration repository for version independence Adding custom content to the GitOps ZTP pipeline 4.7.5. Agent-based Installer New in this release No reference design updates in this release Description The optional Agent-based Installer component provides installation capabilities without centralized infrastructure. The installation program creates an ISO image that you mount to the server. When the server boots it installs OpenShift Container Platform and supplied extra manifests. The Agent-based Installer allows you to install OpenShift Container Platform without a hub cluster. A container image registry is required for cluster installation. Limits and requirements You can supply a limited set of additional manifests at installation time. You must include MachineConfiguration CRs that are required by the RAN DU use case. Engineering considerations The Agent-based Installer provides a baseline OpenShift Container Platform installation. You install Day 2 Operators and the remainder of the RAN DU use case configurations after installation. Additional resources Installing an OpenShift Container Platform cluster with the Agent-based Installer 4.8. Telco RAN DU reference configuration CRs Use the following custom resources (CRs) to configure and deploy OpenShift Container Platform clusters with the telco RAN DU profile. Use the CRs to form the common baseline used in all the specific use models unless otherwise indicated. Note You can extract the complete set of RAN DU CRs from the ztp-site-generate container image. See Preparing the GitOps ZTP site configuration repository for more information. Additional resources Understanding the cluster-compare plugin 4.8.1. Cluster tuning reference CRs Table 4.4. Cluster tuning CRs Component Reference CR Description Optional Cluster capabilities example-sno.yaml Representative SiteConfig CR to install single-node OpenShift with the RAN DU profile No Console disable ConsoleOperatorDisable.yaml Disables the Console Operator. No Disconnected registry 09-openshift-marketplace-ns.yaml Defines a dedicated namespace for managing the OpenShift Operator Marketplace. No Disconnected registry DefaultCatsrc.yaml Configures the catalog source for the disconnected registry. No Disconnected registry DisableOLMPprof.yaml Disables performance profiling for OLM. No Disconnected registry DisconnectedICSP.yaml Configures disconnected registry image content source policy. No Disconnected registry OperatorHub.yaml Optional, for multi-node clusters only. Configures the OperatorHub in OpenShift, disabling all default Operator sources. Not required for single-node OpenShift installs with marketplace capability disabled. No Monitoring configuration ReduceMonitoringFootprint.yaml Reduces the monitoring footprint by disabling Alertmanager and Telemeter, and sets Prometheus retention to 24 hours No Network diagnostics disable DisableSnoNetworkDiag.yaml Configures the cluster network settings to disable built-in network troubleshooting and diagnostic features. No 4.8.2. Day 2 Operators reference CRs Table 4.5. Day 2 Operators CRs Component Reference CR Description Optional Cluster Logging Operator ClusterLogForwarder.yaml Configures log forwarding for the cluster. No Cluster Logging Operator ClusterLogNS.yaml Configures the namespace for cluster logging. No Cluster Logging Operator ClusterLogOperGroup.yaml Configures Operator group for cluster logging. No Cluster Logging Operator ClusterLogServiceAccount.yaml New in 4.18. Configures the cluster logging service account. No Cluster Logging Operator ClusterLogServiceAccountAuditBinding.yaml New in 4.18. Configures the cluster logging service account. No Cluster Logging Operator ClusterLogServiceAccountInfrastructureBinding.yaml New in 4.18. Configures the cluster logging service account. No Cluster Logging Operator ClusterLogSubscription.yaml Manages installation and updates for the Cluster Logging Operator. No Lifecycle Agent ImageBasedUpgrade.yaml Manage the image-based upgrade process in OpenShift. Yes Lifecycle Agent LcaSubscription.yaml Manages installation and updates for the LCA Operator. Yes Lifecycle Agent LcaSubscriptionNS.yaml Configures namespace for LCA subscription. Yes Lifecycle Agent LcaSubscriptionOperGroup.yaml Configures the Operator group for the LCA subscription. Yes Local Storage Operator StorageClass.yaml Defines a storage class with a Delete reclaim policy and no dynamic provisioning in the cluster. No Local Storage Operator StorageLV.yaml Configures local storage devices for the example-storage-class in the openshift-local-storage namespace, specifying device paths and filesystem type. No Local Storage Operator StorageNS.yaml Creates the namespace with annotations for workload management and the deployment wave for the Local Storage Operator. No Local Storage Operator StorageOperGroup.yaml Creates the Operator group for the Local Storage Operator. No Local Storage Operator StorageSubscription.yaml Creates the namespace for the Local Storage Operator with annotations for workload management and deployment wave. No LVM Operator LVMOperatorStatus.yaml Verifies the installation or upgrade of the LVM Storage Operator. Yes LVM Operator StorageLVMCluster.yaml Defines an LVM cluster configuration, with placeholders for storage device classes and volume group settings. Optional substitute for the Local Storage Operator. No LVM Operator StorageLVMSubscription.yaml Manages installation and updates of the LVMS Operator. Optional substitute for the Local Storage Operator. No LVM Operator StorageLVMSubscriptionNS.yaml Creates the namespace for the LVMS Operator with labels and annotations for cluster monitoring and workload management. Optional substitute for the Local Storage Operator. No LVM Operator StorageLVMSubscriptionOperGroup.yaml Defines the target namespace for the LVMS Operator. Optional substitute for the Local Storage Operator. No Node Tuning Operator PerformanceProfile.yaml Configures node performance settings in an OpenShift cluster, optimizing for low latency and real-time workloads. No Node Tuning Operator TunedPerformancePatch.yaml Applies performance tuning settings, including scheduler groups and service configurations for nodes in the specific namespace. No PTP fast event notifications PtpConfigBoundaryForEvent.yaml Configures PTP settings for PTP boundary clocks with additional options for event synchronization. Dependent on cluster role. No PTP fast event notifications PtpConfigForHAForEvent.yaml Configures PTP for highly available boundary clocks with additional PTP fast event settings. Dependent on cluster role. No PTP fast event notifications PtpConfigMasterForEvent.yaml Configures PTP for PTP grandmaster clocks with additional PTP fast event settings. Dependent on cluster role. No PTP fast event notifications PtpConfigSlaveForEvent.yaml Configures PTP for PTP ordinary clocks with additional PTP fast event settings. Dependent on cluster role. No PTP fast event notifications PtpOperatorConfigForEvent.yaml Overrides the default OperatorConfig. Configures the PTP Operator specifying node selection criteria for running PTP daemons in the openshift-ptp namespace. No PTP Operator PtpConfigBoundary.yaml Configures PTP settings for PTP boundary clocks. Dependent on cluster role. No PTP Operator PtpConfigDualCardGmWpc.yaml Configures PTP grandmaster clock settings for hosts that have dual NICs. Dependent on cluster role. No PTP Operator PtpConfigGmWpc.yaml Configures PTP grandmaster clock settings for hosts that have a single NIC. Dependent on cluster role. No PTP Operator PtpConfigSlave.yaml Configures PTP settings for a PTP ordinary clock. Dependent on cluster role. No PTP Operator PtpOperatorConfig.yaml Configures the PTP Operator settings, specifying node selection criteria for running PTP daemons in the openshift-ptp namespace. No PTP Operator PtpSubscription.yaml Manages installation and updates of the PTP Operator in the openshift-ptp namespace. No PTP Operator PtpSubscriptionNS.yaml Configures the namespace for the PTP Operator. No PTP Operator PtpSubscriptionOperGroup.yaml Configures the Operator group for the PTP Operator. No PTP Operator (high availability) PtpConfigBoundary.yaml Configures PTP settings for highly available PTP boundary clocks. No PTP Operator (high availability) PtpConfigForHA.yaml Configures PTP settings for highly available PTP boundary clocks. No SR-IOV FEC Operator AcceleratorsNS.yaml Configures namespace for the VRAN Acceleration Operator. Optional part of application workload. Yes SR-IOV FEC Operator AcceleratorsOperGroup.yaml Configures the Operator group for the VRAN Acceleration Operator. Optional part of application workload. Yes SR-IOV FEC Operator AcceleratorsSubscription.yaml Manages installation and updates for the VRAN Acceleration Operator. Optional part of application workload. Yes SR-IOV FEC Operator SriovFecClusterConfig.yaml Configures SR-IOV FPGA Ethernet Controller (FEC) settings for nodes, specifying drivers, VF amount, and node selection. Yes SR-IOV Operator SriovNetwork.yaml Defines an SR-IOV network configuration, with placeholders for various network settings. No SR-IOV Operator SriovNetworkNodePolicy.yaml Configures SR-IOV network settings for specific nodes, including device type, RDMA support, physical function names, and the number of virtual functions. No SR-IOV Operator SriovOperatorConfig.yaml Configures SR-IOV Network Operator settings, including node selection, injector, and webhook options. No SR-IOV Operator SriovOperatorConfigForSNO.yaml Configures the SR-IOV Network Operator settings for single-node OpenShift, including node selection, injector, webhook options, and disabling node drain, in the openshift-sriov-network-operator namespace. No SR-IOV Operator SriovSubscription.yaml Manages the installation and updates of the SR-IOV Network Operator. No SR-IOV Operator SriovSubscriptionNS.yaml Creates the namespace for the SR-IOV Network Operator with specific annotations for workload management and deployment waves. No SR-IOV Operator SriovSubscriptionOperGroup.yaml Defines the target namespace for the SR-IOV Network Operators, enabling their management and deployment within this namespace. No 4.8.3. Machine configuration reference CRs Table 4.6. Machine configuration CRs Component Reference CR Description Optional Container runtime (crun) enable-crun-master.yaml Configures the container runtime (crun) for control plane nodes. No Container runtime (crun) enable-crun-worker.yaml Configures the container runtime (crun) for worker nodes. No CRI-O wipe disable 99-crio-disable-wipe-master.yaml Disables automatic CRI-O cache wipe following a reboot for on control plane nodes. No CRI-O wipe disable 99-crio-disable-wipe-worker.yaml Disables automatic CRI-O cache wipe following a reboot for on worker nodes. No Kdump enable 06-kdump-master.yaml Configures kdump crash reporting on master nodes. No Kdump enable 06-kdump-worker.yaml Configures kdump crash reporting on worker nodes. No Kubelet configuration and container mount hiding 01-container-mount-ns-and-kubelet-conf-master.yaml Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on control plane nodes. No Kubelet configuration and container mount hiding 01-container-mount-ns-and-kubelet-conf-worker.yaml Configures a mount namespace for sharing container-specific mounts between kubelet and CRI-O on worker nodes. No One-shot time sync 99-sync-time-once-master.yaml Synchronizes time once on master nodes. No One-shot time sync 99-sync-time-once-worker.yaml Synchronizes time once on worker nodes. No SCTP 03-sctp-machine-config-master.yaml Loads the SCTP kernel module on master nodes. Yes SCTP 03-sctp-machine-config-worker.yaml Loads the SCTP kernel module on worker nodes. Yes Set RCU normal 08-set-rcu-normal-master.yaml Disables rcu_expedited by setting rcu_normal after the control plane node has booted. No Set RCU normal 08-set-rcu-normal-worker.yaml Disables rcu_expedited by setting rcu_normal after the worker node has booted. No SRIOV-related kernel arguments 07-sriov-related-kernel-args-master.yaml Enables SR-IOV support on master nodes. No 4.9. Comparing a cluster with the telco RAN DU reference configuration After you deploy a telco RAN DU cluster, you can use the cluster-compare plugin to assess the cluster's compliance with the telco RAN DU reference design specifications (RDS). The cluster-compare plugin is an OpenShift CLI ( oc ) plugin. The plugin uses a telco RAN DU reference configuration to validate the cluster with the telco RAN DU custom resources (CRs). The plugin-specific reference configuration for telco RAN DU is packaged in a container image with the telco RAN DU CRs. For further information about the cluster-compare plugin, see "Understanding the cluster-compare plugin". Prerequisites You have access to the cluster as a user with the cluster-admin role. You have credentials to access the registry.redhat.io container image registry. You installed the cluster-compare plugin. Procedure Login to the container image registry with your credentials by running the following command: USD podman login registry.redhat.io Extract the content from the ztp-site-generate-rhel8 container image by running the following commands:: USD podman pull registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.18 USD mkdir -p ./out USD podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.18 extract /home/ztp --tar | tar x -C ./out Compare the configuration for your cluster to the reference configuration by running the following command: USD oc cluster-compare -r out/reference/metadata.yaml Example output ... ********************************** Cluster CR: config.openshift.io/v1_OperatorHub_cluster 1 Reference File: required/other/operator-hub.yaml 2 Diff Output: diff -u -N /tmp/MERGED-2801470219/config-openshift-io-v1_operatorhub_cluster /tmp/LIVE-2569768241/config-openshift-io-v1_operatorhub_cluster --- /tmp/MERGED-2801470219/config-openshift-io-v1_operatorhub_cluster 2024-12-12 14:13:22.898756462 +0000 +++ /tmp/LIVE-2569768241/config-openshift-io-v1_operatorhub_cluster 2024-12-12 14:13:22.898756462 +0000 @@ -1,6 +1,6 @@ apiVersion: config.openshift.io/v1 kind: OperatorHub metadata: + annotations: 3 + include.release.openshift.io/hypershift: "true" name: cluster -spec: - disableAllDefaultSources: true ********************************** Summary 4 CRs with diffs: 11/12 5 CRs in reference missing from the cluster: 40 6 optional-image-registry: image-registry: Missing CRs: 7 - optional/image-registry/ImageRegistryPV.yaml optional-ptp-config: ptp-config: One of the following is required: - optional/ptp-config/PtpConfigBoundary.yaml - optional/ptp-config/PtpConfigGmWpc.yaml - optional/ptp-config/PtpConfigDualCardGmWpc.yaml - optional/ptp-config/PtpConfigForHA.yaml - optional/ptp-config/PtpConfigMaster.yaml - optional/ptp-config/PtpConfigSlave.yaml - optional/ptp-config/PtpConfigSlaveForEvent.yaml - optional/ptp-config/PtpConfigForHAForEvent.yaml - optional/ptp-config/PtpConfigMasterForEvent.yaml - optional/ptp-config/PtpConfigBoundaryForEvent.yaml ptp-operator-config: One of the following is required: - optional/ptp-config/PtpOperatorConfig.yaml - optional/ptp-config/PtpOperatorConfigForEvent.yaml optional-storage: storage: Missing CRs: - optional/local-storage-operator/StorageLV.yaml ... No CRs are unmatched to reference CRs 8 Metadata Hash: 09650c31212be9a44b99315ec14d2e7715ee194a5d68fb6d24f65fd5ddbe3c3c 9 No patched CRs 10 1 1 The CR under comparison. The plugin displays each CR with a difference from the corresponding template. 2 The template matching with the CR for comparison. 3 The output in Linux diff format shows the difference between the template and the cluster CR. 4 After the plugin reports the line diffs for each CR, the summary of differences are reported. 5 The number of CRs in the comparison with differences from the corresponding templates. 6 The number of CRs represented in the reference configuration, but missing from the live cluster. 7 The list of CRs represented in the reference configuration, but missing from the live cluster. 8 The CRs that did not match to a corresponding template in the reference configuration. 9 The metadata hash identifies the reference configuration. 10 The list of patched CRs. 4.10. Telco RAN DU 4.18 validated software components The Red Hat telco RAN DU 4.18 solution has been validated using the following Red Hat software products for OpenShift Container Platform managed clusters. Table 4.7. Telco RAN DU managed cluster validated software components Component Software version Managed cluster version 4.18 Cluster Logging Operator 6.1 1 Local Storage Operator 4.18 OpenShift API for Data Protection (OADP) 1.4 PTP Operator 4.18 SR-IOV Operator 4.18 SRIOV-FEC Operator 2.10 Lifecycle Agent 4.18 [1] This table will be updated when the aligned Cluster Logging Operator version 6.2 is released. 4.11. Telco RAN DU 4.18 hub cluster validated software components The Red Hat telco RAN 4.18 solution has been validated using the following Red Hat software products for OpenShift Container Platform hub clusters. Table 4.8. Telco hub cluster validated software components Component Software version Hub cluster version 4.18 Red Hat Advanced Cluster Management (RHACM) 2.12 1 Red Hat OpenShift GitOps 1.14 GitOps ZTP site generate plugins 4.18 Topology Aware Lifecycle Manager (TALM) 4.18 [1] This table will be updated when the aligned RHACM version 2.13 is released.

Commands:
query=avg_over_time(pod:container_cpu_usage:sum{namespace="openshift-kube-apiserver"}[30m])
ip link set dev <physical_function> mtu 9000
podman login registry.redhat.io
podman pull registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.18
mkdir -p ./out
podman run --log-driver=none --rm registry.redhat.io/openshift4/ztp-site-generate-rhel8:v4.18 extract /home/ztp --tar | tar x -C ./out
oc cluster-compare -r out/reference/metadata.yaml
********************************** Cluster CR: config.openshift.io/v1_OperatorHub_cluster 1 Reference File: required/other/operator-hub.yaml 2 Diff Output: diff -u -N /tmp/MERGED-2801470219/config-openshift-io-v1_operatorhub_cluster /tmp/LIVE-2569768241/config-openshift-io-v1_operatorhub_cluster --- /tmp/MERGED-2801470219/config-openshift-io-v1_operatorhub_cluster 2024-12-12 14:13:22.898756462 +0000 +++ /tmp/LIVE-2569768241/config-openshift-io-v1_operatorhub_cluster 2024-12-12 14:13:22.898756462 +0000 @@ -1,6 +1,6 @@ apiVersion: config.openshift.io/v1 kind: OperatorHub metadata: + annotations: 3 + include.release.openshift.io/hypershift: "true" name: cluster -spec: - disableAllDefaultSources: true ********************************** Summary 4 CRs with diffs: 11/12 5 CRs in reference missing from the cluster: 40 6 optional-image-registry: image-registry: Missing CRs: 7 - optional/image-registry/ImageRegistryPV.yaml optional-ptp-config: ptp-config: One of the following is required: - optional/ptp-config/PtpConfigBoundary.yaml - optional/ptp-config/PtpConfigGmWpc.yaml - optional/ptp-config/PtpConfigDualCardGmWpc.yaml - optional/ptp-config/PtpConfigForHA.yaml - optional/ptp-config/PtpConfigMaster.yaml - optional/ptp-config/PtpConfigSlave.yaml - optional/ptp-config/PtpConfigSlaveForEvent.yaml - optional/ptp-config/PtpConfigForHAForEvent.yaml - optional/ptp-config/PtpConfigMasterForEvent.yaml - optional/ptp-config/PtpConfigBoundaryForEvent.yaml ptp-operator-config: One of the following is required: - optional/ptp-config/PtpOperatorConfig.yaml - optional/ptp-config/PtpOperatorConfigForEvent.yaml optional-storage: storage: Missing CRs: - optional/local-storage-operator/StorageLV.yaml No CRs are unmatched to reference CRs 8 Metadata Hash: 09650c31212be9a44b99315ec14d2e7715ee194a5d68fb6d24f65fd5ddbe3c3c 9 No patched CRs 10