Chapter 3. Recommended host practices for IBM Z & LinuxONE environments This topic provides recommended host practices for OpenShift Container Platform on IBM Z and LinuxONE. Note The s390x architecture is unique in many aspects. Therefore, some recommendations made here might not apply to other platforms. Note Unless stated otherwise, these practices apply to both z/VM and Red Hat Enterprise Linux (RHEL) KVM installations on IBM Z and LinuxONE. 3.1. Managing CPU overcommitment In a highly virtualized IBM Z environment, you must carefully plan the infrastructure setup and sizing. One of the most important features of virtualization is the capability to do resource overcommitment, allocating more resources to the virtual machines than actually available at the hypervisor level. This is very workload dependent and there is no golden rule that can be applied to all setups. Depending on your setup, consider these best practices regarding CPU overcommitment: At LPAR level (PR/SM hypervisor), avoid assigning all available physical cores (IFLs) to each LPAR. For example, with four physical IFLs available, you should not define three LPARs with four logical IFLs each. Check and understand LPAR shares and weights. An excessive number of virtual CPUs can adversely affect performance. Do not define more virtual processors to a guest than logical processors are defined to the LPAR. Configure the number of virtual processors per guest for peak workload, not more. Start small and monitor the workload. Increase the vCPU number incrementally if necessary. Not all workloads are suitable for high overcommitment ratios. If the workload is CPU intensive, you will probably not be able to achieve high ratios without performance problems. Workloads that are more I/O intensive can keep consistent performance even with high overcommitment ratios. Additional resources z/VM Common Performance Problems and Solutions z/VM overcommitment considerations LPAR CPU management 3.2. How to disable Transparent Huge Pages Transparent Huge Pages (THP) attempt to automate most aspects of creating, managing, and using huge pages. Since THP automatically manages the huge pages, this is not always handled optimally for all types of workloads. THP can lead to performance regressions, since many applications handle huge pages on their own. Therefore, consider disabling THP. The following steps describe how to disable THP using a Node Tuning Operator (NTO) profile. 3.2.1. Disable THP with a Node Tuning Operator (NTO) profile Procedure Copy the following NTO sample profile into a YAML file. For example, thp-s390-tuned.yaml : apiVersion: tuned.openshift.io/v1 kind: Tuned metadata: name: thp-workers-profile namespace: openshift-cluster-node-tuning-operator spec: profile: - data: | [main] summary=Custom tuned profile for OpenShift on IBM Z to turn off THP on worker nodes include=openshift-node [vm] transparent_hugepages=never name: openshift-thp-never-worker recommend: - match: - label: node-role.kubernetes.io/worker priority: 35 profile: openshift-thp-never-worker Create the NTO profile: USD oc create -f thp-s390-tuned.yaml Check the list of active profiles: USD oc get tuned -n openshift-cluster-node-tuning-operator Remove the profile: USD oc delete -f thp-s390-tuned.yaml Verification Log in to one of the nodes and do a regular THP check to verify if the nodes applied the profile successfully: USD cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] 3.3. Boost networking performance with Receive Flow Steering Receive Flow Steering (RFS) extends Receive Packet Steering (RPS) by further reducing network latency. RFS is technically based on RPS, and improves the efficiency of packet processing by increasing the CPU cache hit rate. RFS achieves this, and in addition considers queue length, by determining the most convenient CPU for computation so that cache hits are more likely to occur within the CPU. Thus, the CPU cache is invalidated less and requires fewer cycles to rebuild the cache. This can help reduce packet processing run time. 3.3.1. Use the Machine Config Operator (MCO) to activate RFS Procedure Copy the following MCO sample profile into a YAML file. For example, enable-rfs.yaml : apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 50-enable-rfs spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:text/plain;charset=US-ASCII,%23%20turn%20on%20Receive%20Flow%20Steering%20%28RFS%29%20for%20all%20network%20interfaces%0ASUBSYSTEM%3D%3D%22net%22%2C%20ACTION%3D%3D%22add%22%2C%20RUN%7Bprogram%7D%2B%3D%22/bin/bash%20-c%20%27for%20x%20in%20/sys/%24DEVPATH/queues/rx-%2A%3B%20do%20echo%208192%20%3E%20%24x/rps_flow_cnt%3B%20%20done%27%22%0A filesystem: root mode: 0644 path: /etc/udev/rules.d/70-persistent-net.rules - contents: source: data:text/plain;charset=US-ASCII,%23%20define%20sock%20flow%20enbtried%20for%20%20Receive%20Flow%20Steering%20%28RFS%29%0Anet.core.rps_sock_flow_entries%3D8192%0A filesystem: root mode: 0644 path: /etc/sysctl.d/95-enable-rps.conf Create the MCO profile: USD oc create -f enable-rfs.yaml Verify that an entry named 50-enable-rfs is listed: USD oc get mc To deactivate, enter: USD oc delete mc 50-enable-rfs Additional resources OpenShift Container Platform on IBM Z: Tune your network performance with RFS Configuring Receive Flow Steering (RFS) Scaling in the Linux Networking Stack 3.4. Choose your networking setup The networking stack is one of the most important components for a Kubernetes-based product like OpenShift Container Platform. For IBM Z setups, the networking setup depends on the hypervisor of your choice. Depending on the workload and the application, the best fit usually changes with the use case and the traffic pattern. Depending on your setup, consider these best practices: Consider all options regarding networking devices to optimize your traffic pattern. Explore the advantages of OSA-Express, RoCE Express, HiperSockets, z/VM VSwitch, Linux Bridge (KVM), and others to decide which option leads to the greatest benefit for your setup. Always use the latest available NIC version. For example, OSA Express 7S 10 GbE shows great improvement compared to OSA Express 6S 10 GbE with transactional workload types, although both are 10 GbE adapters. Each virtual switch adds an additional layer of latency. The load balancer plays an important role for network communication outside the cluster. Consider using a production-grade hardware load balancer if this is critical for your application. OpenShift Container Platform SDN introduces flows and rules, which impact the networking performance. Make sure to consider pod affinities and placements, to benefit from the locality of services where communication is critical. Balance the trade-off between performance and functionality. Additional resources OpenShift Container Platform on IBM Z - Performance Experiences, Hints and Tips OpenShift Container Platform on IBM Z Networking Performance Controlling pod placement on nodes using node affinity rules 3.5. Ensure high disk performance with HyperPAV on z/VM DASD and ECKD devices are commonly used disk types in IBM Z environments. In a typical OpenShift Container Platform setup in z/VM environments, DASD disks are commonly used to support the local storage for the nodes. You can set up HyperPAV alias devices to provide more throughput and overall better I/O performance for the DASD disks that support the z/VM guests. Using HyperPAV for the local storage devices leads to a significant performance benefit. However, you must be aware that there is a trade-off between throughput and CPU costs. 3.5.1. Use the Machine Config Operator (MCO) to activate HyperPAV aliases in nodes using z/VM full-pack minidisks For z/VM-based OpenShift Container Platform setups that use full-pack minidisks, you can leverage the advantage of MCO profiles by activating HyperPAV aliases in all of the nodes. You must add YAML configurations for both control plane and compute nodes. Procedure Copy the following MCO sample profile into a YAML file for the control plane node. For example, 05-master-kernelarg-hpav.yaml : USD cat 05-master-kernelarg-hpav.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: master name: 05-master-kernelarg-hpav spec: config: ignition: version: 3.1.0 kernelArguments: - rd.dasd=800-805 Copy the following MCO sample profile into a YAML file for the compute node. For example, 05-worker-kernelarg-hpav.yaml : USD cat 05-worker-kernelarg-hpav.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 05-worker-kernelarg-hpav spec: config: ignition: version: 3.1.0 kernelArguments: - rd.dasd=800-805 Note You must modify the rd.dasd arguments to fit the device IDs. Create the MCO profiles: USD oc create -f 05-master-kernelarg-hpav.yaml USD oc create -f 05-worker-kernelarg-hpav.yaml To deactivate, enter: USD oc delete -f 05-master-kernelarg-hpav.yaml USD oc delete -f 05-worker-kernelarg-hpav.yaml Additional resources Using HyperPAV for ECKD DASD Scaling HyperPAV alias devices on Linux guests on z/VM 3.6. RHEL KVM on IBM Z host recommendations Optimizing a KVM virtual server environment strongly depends on the workloads of the virtual servers and on the available resources. The same action that enhances performance in one environment can have adverse effects in another. Finding the best balance for a particular setting can be a challenge and often involves experimentation. The following section introduces some best practices when using OpenShift Container Platform with RHEL KVM on IBM Z and LinuxONE environments. 3.6.1. Use multiple queues for your VirtIO network interfaces With multiple virtual CPUs, you can transfer packages in parallel if you provide multiple queues for incoming and outgoing packets. Use the queues attribute of the driver element to configure multiple queues. Specify an integer of at least 2 that does not exceed the number of virtual CPUs of the virtual server. The following example specification configures two input and output queues for a network interface: <interface type="direct"> <source network="net01"/> <model type="virtio"/> <driver ... queues="2"/> </interface> Multiple queues are designed to provide enhanced performance for a network interface, but they also use memory and CPU resources. Start with defining two queues for busy interfaces. , try two queues for interfaces with less traffic or more than two queues for busy interfaces. 3.6.2. Use I/O threads for your virtual block devices To make virtual block devices use I/O threads, you must configure one or more I/O threads for the virtual server and each virtual block device to use one of these I/O threads. The following example specifies <iothreads>3</iothreads> to configure three I/O threads, with consecutive decimal thread IDs 1, 2, and 3. The iothread="2" parameter specifies the driver element of the disk device to use the I/O thread with ID 2. Sample I/O thread specification ... <domain> <iothreads>3</iothreads> 1 ... <devices> ... <disk type="block" device="disk"> 2 <driver ... iothread="2"/> </disk> ... </devices> ... </domain> 1 The number of I/O threads. 2 The driver element of the disk device. Threads can increase the performance of I/O operations for disk devices, but they also use memory and CPU resources. You can configure multiple devices to use the same thread. The best mapping of threads to devices depends on the available resources and the workload. Start with a small number of I/O threads. Often, a single I/O thread for all disk devices is sufficient. Do not configure more threads than the number of virtual CPUs, and do not configure idle threads. You can use the virsh iothreadadd command to add I/O threads with specific thread IDs to a running virtual server. 3.6.3. Avoid virtual SCSI devices Configure virtual SCSI devices only if you need to address the device through SCSI-specific interfaces. Configure disk space as virtual block devices rather than virtual SCSI devices, regardless of the backing on the host. However, you might need SCSI-specific interfaces for: A LUN for a SCSI-attached tape drive on the host. A DVD ISO file on the host file system that is mounted on a virtual DVD drive. 3.6.4. Configure guest caching for disk Configure your disk devices to do caching by the guest and not by the host. Ensure that the driver element of the disk device includes the cache="none" and io="native" parameters. <disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none" io="native" iothread="1"/> ... </disk> 3.6.5. Exclude the memory balloon device Unless you need a dynamic memory size, do not define a memory balloon device and ensure that libvirt does not create one for you. Include the memballoon parameter as a child of the devices element in your domain configuration XML file. Check the list of active profiles: <memballoon model="none"/> 3.6.6. Tune the CPU migration algorithm of the host scheduler Important Do not change the scheduler settings unless you are an expert who understands the implications. Do not apply changes to production systems without testing them and confirming that they have the intended effect. The kernel.sched_migration_cost_ns parameter specifies a time interval in nanoseconds. After the last execution of a task, the CPU cache is considered to have useful content until this interval expires. Increasing this interval results in fewer task migrations. The default value is 500000 ns. If the CPU idle time is higher than expected when there are runnable processes, try reducing this interval. If tasks bounce between CPUs or nodes too often, try increasing it. To dynamically set the interval to 60000 ns, enter the following command: # sysctl kernel.sched_migration_cost_ns=60000 To persistently change the value to 60000 ns, add the following entry to /etc/sysctl.conf : kernel.sched_migration_cost_ns=60000 3.6.7. Disable the cpuset cgroup controller Note This setting applies only to KVM hosts with cgroups version 1. To enable CPU hotplug on the host, disable the cgroup controller. Procedure Open /etc/libvirt/qemu.conf with an editor of your choice. Go to the cgroup_controllers line. Duplicate the entire line and remove the leading number sign (#) from the copy. Remove the cpuset entry, as follows: cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuacct" ] For the new setting to take effect, you must restart the libvirtd daemon: Stop all virtual machines. Run the following command: # systemctl restart libvirtd Restart the virtual machines. This setting persists across host reboots. 3.6.8. Tune the polling period for idle virtual CPUs When a virtual CPU becomes idle, KVM polls for wakeup conditions for the virtual CPU before allocating the host resource. You can specify the time interval, during which polling takes place in sysfs at /sys/module/kvm/parameters/halt_poll_ns . During the specified time, polling reduces the wakeup latency for the virtual CPU at the expense of resource usage. Depending on the workload, a longer or shorter time for polling can be beneficial. The time interval is specified in nanoseconds. The default is 50000 ns. To optimize for low CPU consumption, enter a small value or write 0 to disable polling: # echo 0 > /sys/module/kvm/parameters/halt_poll_ns To optimize for low latency, for example for transactional workloads, enter a large value: # echo 80000 > /sys/module/kvm/parameters/halt_poll_ns Additional resources Linux on IBM Z Performance Tuning for KVM Getting started with virtualization on IBM Z

Commands:
apiVersion: tuned.openshift.io/v1 kind: Tuned metadata: name: thp-workers-profile namespace: openshift-cluster-node-tuning-operator spec: profile: - data: | [main] summary=Custom tuned profile for OpenShift on IBM Z to turn off THP on worker nodes include=openshift-node [vm] transparent_hugepages=never name: openshift-thp-never-worker recommend: - match: - label: node-role.kubernetes.io/worker priority: 35 profile: openshift-thp-never-worker
oc create -f thp-s390-tuned.yaml
oc get tuned -n openshift-cluster-node-tuning-operator
oc delete -f thp-s390-tuned.yaml
cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 50-enable-rfs spec: config: ignition: version: 2.2.0 storage: files: - contents: source: data:text/plain;charset=US-ASCII,%23%20turn%20on%20Receive%20Flow%20Steering%20%28RFS%29%20for%20all%20network%20interfaces%0ASUBSYSTEM%3D%3D%22net%22%2C%20ACTION%3D%3D%22add%22%2C%20RUN%7Bprogram%7D%2B%3D%22/bin/bash%20-c%20%27for%20x%20in%20/sys/%24DEVPATH/queues/rx-%2A%3B%20do%20echo%208192%20%3E%20%24x/rps_flow_cnt%3B%20%20done%27%22%0A filesystem: root mode: 0644 path: /etc/udev/rules.d/70-persistent-net.rules - contents: source: data:text/plain;charset=US-ASCII,%23%20define%20sock%20flow%20enbtried%20for%20%20Receive%20Flow%20Steering%20%28RFS%29%0Anet.core.rps_sock_flow_entries%3D8192%0A filesystem: root mode: 0644 path: /etc/sysctl.d/95-enable-rps.conf
oc create -f enable-rfs.yaml
oc get mc
oc delete mc 50-enable-rfs
cat 05-master-kernelarg-hpav.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: master name: 05-master-kernelarg-hpav spec: config: ignition: version: 3.1.0 kernelArguments: - rd.dasd=800-805
cat 05-worker-kernelarg-hpav.yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: 05-worker-kernelarg-hpav spec: config: ignition: version: 3.1.0 kernelArguments: - rd.dasd=800-805
oc create -f 05-master-kernelarg-hpav.yaml
oc create -f 05-worker-kernelarg-hpav.yaml
oc delete -f 05-master-kernelarg-hpav.yaml
oc delete -f 05-worker-kernelarg-hpav.yaml
<interface type="direct"> <source network="net01"/> <model type="virtio"/> <driver ... queues="2"/> </interface>
<domain> <iothreads>3</iothreads> 1 <devices> <disk type="block" device="disk"> 2 <driver ... iothread="2"/> </disk> </devices> </domain>
<disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none" io="native" iothread="1"/> </disk>
<memballoon model="none"/>
sysctl kernel.sched_migration_cost_ns=60000
kernel.sched_migration_cost_ns=60000
cgroup_controllers = [ "cpu", "devices", "memory", "blkio", "cpuacct" ]
systemctl restart libvirtd
echo 0 > /sys/module/kvm/parameters/halt_poll_ns
echo 80000 > /sys/module/kvm/parameters/halt_poll_ns