IBM® Power®

Power10 Performance Quick Start Guides(Power10 QSGs)
November 2021

1

© 2021 IBM Corporation

IBM® Power® Power10 Quick-start Guide – E1080 Memory Performance
Minimum Memory• For each processor socket a minimum of 8 of the 16 DDIMMs are populated• In a node, a minimum of 32 of 64 for the DDIMMs are populated• In a 4-Node system, a minimum of 128 of the 256 DDIMMs are populated

DDIMM Plug Rules• Meet minimum memory allowed (each processor socket a minimum of 8 of the 16
DDIMMs are populated)• All DDIMMs under each processor have to be same capacity• Feature upgrades will be offered in increments of 4 DDIMM’s, all of which have thesame capacity.
• The only valid number of DDIMM’s plugged into sites connected to a given processor
module is 8 or 12 or 16.

Memory Performance• System performance improves as the amount of memory is spread across more DDIMM
slots.
For example, if 1TB is needed in a Node, it is better to have 64 x 32GB DDIMMs thanto have 32 x 64GB DDIMMs.
• Plugging DDIMMs that are all the same size will provide the highest performance• System performance improves as more quads match each other• System performance improves as more processor DDIMMs match each other• System performance improves on a multi drawer system if memory capacity between
drawers are balanced.
2

Memory Bandwidth
DDIMM Capacity

Theoretical Max
Bandwidth

32GB, 64 GB (DDR4 @ 3200 Mbps)

409 GB/s

128GB, 256 GB (DDR4 @ 2933 Mbps)

375 GB/s

Summary• For the best possible performance, it is generally
recommended that memory be installed evenly across all
system node drawers and all processor sockets in thesystem. Balancing memory across the installed system
planar cards enables memory access in a consistent mannerand typically results in better performance for yourconfiguration.
• Though maximum memory bandwidth is achieved by filling all
the memory slots, plans for future memory additions should
be considered when deciding which memory feature size to
use at the time of initial system order.

© 2021 IBM Corporation

IBM® Power®

Power10 Quick Start Guide - P10 MMA Performance Guide

P10 Compute & MMA ArchitectureØ2x Bandwidth matched SIMD*
Ø 8 independent Fixed & Float SIMD engines per Core
Ø4 – 32x Matrix Math Acceleration*
Ø 4 512 bit engine per core = 2048b results / cycles
Ø Matrix math outer products of Single, Double & Reduced precision.
ØMMA Architecture support introduced in POWER ISA v3.1
ØSupports SP, DP, BF16, HP, Int-16, Int-8 & Int-4 precision levels.

P10 MMAApplications & Workload Integration
Ø ML & HPC applications with dense linear algebra computations, matrix
multiplications, convolutions, FFT can be accelerate with MMA
Ø GCC version >= 10 & LLVM version >=12 supports MMA through built-ins.
Ø OpenBLAS, IBM ESSL & Eigen Libraries already optimized with MMA instructions
for P10.
Ø Easy integration of MMA for enterprise applications, ML frameworks and Open
Community packages via above BLAS libraries.

PowerPC Matrix-Multiply Assist Built-in Functions
https://gcc.gnu.org/onlinedocs/gcc/PowerPC-Matrix-Multiply-Assist-Built-in-Functions.html

Matrix-Multiply Assist Best Practices Guide

https://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/redp5612.html?Open3© 2021 IBM Corporation

IBM® Power®

Power10 Quick Start Guide - PowerVM Best Practices

Virtual Processors
•
•
•
•
•

The sum of the entitled cores of all shared partitions cannot exceed the number of cores in the shared pool
Ensure number of configured virtual processors of any shared partitions on a frame is not more than number of cores in the shared pool
Configure the number of virtual processors for a shared partition to sustained peak capacity demand
Configure the number of entitled cores for a shared partition to average utilization of that partition for better performance
To ensure better memory and CPU affinity (avoid unnecessary preemptions of the virtual processor), ensure the sum of the entitled cores of allshared partitions close to the number of the cores in the shared pool

Processor Compatibility Mode

• There are 2 processor compatibility mode available for AIX: POWER9 and POWER9_base. Default is POWER9_base mode.
• There are 2 processor compatibility mode available for Linux: POWER9 and POWER10 mode. Default is POWER10 mode.
• After LPM partitions, need to power cycle when changing processor compatibility mode

Processor Folding Considerations• For share partition running AIX on Power9, the default vpm_throughput_mode = 0, on Power10, the default vpm_throughput_mode = 2. For
workloads have long running jobs, it can potentially help with core usage reduction.
• For dedicated partition running AIX, the default vpm_throughput_mode = 0 on both Power9 and Power10.

LPAR Page Table Size Considerations• Radix page table is supported starting on Power10 running Linux. It can potentially improve workload performance.

Reference:
Hints andTips for Migrating Workload to IBM POWER Systems• https://www.ibm.com/downloads/cas/39XWR7YM
IBM POWERVirtualizationBest PracticesGuide• https://www.ibm.com/downloads/cas/JVGZA8RW
4© 2021 IBM Corporation

IBM® Power®

Power10 Quick Start Guide – AIX Best Practices

Ensure OS level is current• Fix Central provides the latest updates for AIX, IBM i, VIOS, Linux, HMC and F/W. In addition to that, the FLRT tool provides
the recommended levels for each H/W model. Use these tools to maintain your system up to date. If you cannot move up to
the recommended level, then refer to the Known Issue section of the Hints & Tips for Migrating Workload to IBM POWER10
Processor-Based Systems document.
AIX CPU utilization• On POWER10, the AIX OS system is optimized for best raw throughput at higher CPU usage when running with dedicatedprocessors. When running with shared processors, the AIX OS system is optimized to reduce CPU usage (pc). If the customer
requires to further reduce CPU usage (pc), use the schedo tunable vpm_throughput_mode to tune the workload and evaluate
the benefits of raw throughput vs. CPU usage.
NX GZIP• To take advantage of NX GZIP acceleration on POWER10 systems the LPAR must be in POWER9 compatibility mode (notPOWER9_base mode) or POWER10 compatibility mode.

5© 2021 IBM Corporation

IBM® Power®

Power10 Quick-start Guide – IBM i Performance Tips

IBM i

Virtual Processors – shared vs dedicated processors

Ensure the IBM i operating system level is current. Fix Central provides the latest updates
for IBM i, VIOS, HMC and firmware. https://www.ibm.com/support/fixcentral/

Utilize dedicated processors for optimal partition level

FirmwareEnergyScale

Ensure the system firmware level is current. Fix Central provides the latest updates for
IBM i, VIOS, HMC and firmware. https://www.ibm.com/support/fixcentral/

For the best CPU processor speed, ensure that Maximum
Performance is set (default for IBM Power E1080). This setting is
configurable in the ASMI.

Memory DIMMs
Follow proper memory plug-in rules. If possible, fully populate memory DIMM slots andutilize similar sized memory DIMMs.

performance.

Storage and Networking I/O
VIOS provides flexible storage and networking functionality. For thebest possible performance, utilize native IBM i interfaces for I/O.

Processor SMT level
To take full advantage of the performance of Power10 CPUs, we recommend clients
utilize the IBM i default processor multitasking settings, which will maximize the SMTlevel for the LPAR configuration.

More comprehensive information
Refer to link: IBM i on Power - Performance FAQ
https://www.ibm.com/downloads/cas/QWXA9XKN

Partition Placement
Current FW levels ensure optimal placement of the partitions. However, if frequent
DLPAR operations are executed on partitions on the CEC, it is recommended the use DPO
to optimize placement.

6© 2021 IBM Corporation

IBM® Power®

Power10 Quick-start Guide - Enterprise Linux for PowerThe enterprise Linux operating system (OS) is a solid foundation for your hybrid cloud infrastructure and for scale-up enterprise software
solutions. Recent releases are optimized for best-in-class Power10 Enterprise systems

Power10ü SLES15SP3, RHEL8.4 support Power10 native modeü Compat-mode support to allow client to migrate from older generation
Power systems ( P9 and P8 )
ü Default Radix translation support in Power10 mode
ü Significant improvement in encryption performance+

Supported distros:ü Starting with Power9 only RedHat and SUSE are supported in PowerVM
partitions
ü Detailed info on distro support matrix covering older generation HW

ü Support for PowerVM enterprise features : LPM, Shared CPU Pools, DLPARü Innovative solutions : SAP HANA future application growth with 4PB virtual
address spaceü Reduce time to reload the data : Virtual PMEM support for SAP HANAü World-class Support & Service

Best practices :LPM Support:

ü RHEL provide predefined tunings as part of tuned service.
ü Refer to the latest SAP notes for recommended OS settings for SAP
applications. Typically tuned is used in RHEL and saptune or sapconf in SLESü Frequency is managed by the PowerVM. Reference: Energy Management
ü Starting Power8 Huge Dynamic DMA Window helps improve I/O performance.
ü Starting Power9 24x7-Monitoring is integrated with perf tool. Allows
monitoring entire system.
ü Ensure the system firmware level is current.
ü lparnumascore from powerpc-utils shows the LPAR’s current affinity score.
DPO can be used to improve the LPAR affinity score.

Power Specific Packages:

More reads :ü Move Linux logical partitions from older generation Power systems with nearzero application downtime
ü Reference: LPM Guide and related information

ü Powerpc-utils package: Contains utilities for maintenance of IBM powerpc
LPARs. Available as part of the distro.
ü Advance Toolchain for Linux on Power : Contains latest compilers, runtime
libraries.
7

Linux + PowerVM

ü SLES for Power and some compelling features .
ü Get started with Linux on Power systems, Linux on Power Systems servers
ü Enterprise Linux community© 2021 IBM Corporation

IBM® Power®
•
•
•
•
•

Power10 Quick Start Guide – Network IO Considerations

IBM Power systems support various network adapters of different speed and number of ports.
If you are using the same network adapters as your previous system, initially, the same tuning should be used on the new system.
Most Ethernet adapters support multiple receive and transmit queues whose buffer size can be varied to increase max packet count.
The default queue settings are different with different adapters and may not be optimal to achieve maximum message rates in a client-server model.
Using additional queues will increase CPU usage of the system; so optimal queue setting for a specific workload should be used.

Higher speed adapter considerations• Higher speed networks with 25 GigE and 100 GigE network adapters require multiple parallel threads and tuning of driver attributes.
• If it is a Gen4 adapter, make sure the adapted is seated on a Gen4 slot.
• Additional functions such as compression, encryption and duplication can add latency
Changing queue settings in AIX
To change the number of receive/transmit queues in AIX• ifconfig enX detach down
• chdev -l entX -a queues_rx=<value> -a queues_tx=<value>• chdev -l enX -a state=up

Changing queue settings in LinuxTo change the number of queues in Linux• ethtool -L ethX combined <value>

Changing queue size in AIX• ifconfig enX detach down
• chdev -l entX -a rx_max_pkts =<value> -a tx_max_pkts =<value>• chdev -l enX -a state=up

Changing queue size in Linux• ethtool -G ethX rx <value> tx <value>

Virtualization•
Virtualized networking is supported in the form of SRIOV, vNIC, vETH. Virtualization does add latency and can reduce throughput compared to native I/O.
•
Besides the backend hardware, ensure VIOS memory and CPU amounts are enough to provide the required throughput and response times•
IBM PowerVM Best Practices can be very helpful in VIOS sizing

8© 2021 IBM Corporation

IBM® Power®

Power10 Quick Start Guide – Storage IO Considerations• If you are using the same storage adapters as your previous system, initially, the same tuning should be used on the new system. If additional performanceis desired from the existing system, then normal tuning should be performed.
• If the storage subsystems are appreciably different on the newer system than the prior system, the following list of considerations could negatively impact
the perceived speed of applications –o Changing from Direct Attached Storage (DAS or internal) to Storage Area Network (SAN) or Network Attached Storage (NAS) (or external storage) can
increase latency.
o Additional functions such as compression, encryption and deduplication can add latency.
o Reducing the number of Storage LUNs can reduce resources in the server needed to support required throughputs.
o Refer to tuning or setup guides for the new devices to understand these impacts.
o Virtualization does add latency and can reduce throughput compared to native I/O. Besides the backend hardware, ensure VIOS memory and CPU
o Moving to higher speed virtualized adapters in VIOS will require adjusting the VIOS configuration in CPUs and memory. IBM PowerVM Best Practices
can be very helpful in VIOS sizing.
Tuning guidelines - please refer to the IBM Knowledge Center for AIX and Linux guidelines.
PCIe3 12 GB Cache RAID + SAS Adapter Quad-port 6 Gb x8 Adapter
Linux:• https://www.ibm.com/docs/en/power9/9223-42H?topic=availability-ha-asymmetricaccess-optimization• https://www.ibm.com/docs/en/power9/9223-42H?topic=linux-common-sas-raidcontroller-tasks
AIX:• https://www.ibm.com/docs/en/power9/9223-42H?topic=aix-multi-initiator-highavailability• https://www.ibm.com/docs/en/power9/9223-42H?topic=aix-common-controller-diskarray-management-tasks
IBM i:• https://www.ibm.com/docs/en/power9/9223-42H?topic=configurations-dual-storageioa-access-optimization• https://www.ibm.com/docs/en/power9/9223-42H?topic=i-common-controller-diskarray-management-tasks
9

PCIe3 x8 2-port Fibre Channel (32 Gb/s) Adapter• https://www.ibm.com/docs/en/aix/7.2?topic=iompio-device-attributes• https://www.ibm.com/docs/en/power9?topic=channel-npiv-multiple-queue-support
Additional AIX tuning for performance:• SCSI over Fiber Channel (MPIO) : set multipath
algorithm to round_robin for every hdisk• NVMe over Fiber Channel : set nchan attribute to
7 for every NVMe over Fiber Channel Dynamic
controller created during discovery phase
NVMe Adapter AIX tuning for performance• Set nchan attribute to 8 for each NVMe device© 2021 IBM Corporation

IBM® Power®

Power10 Quick-start Guide – IBM Open XL Compilers 17.1.0 for AIX

IBM’s next-generation C/C++/Fortran compilers that combine IBM’s advanced optimizations with open-source LLVM infrastructure

LLVM
ü Greater currency for C/C++ language
ü Faster build speed
ü Community common optimizations
ü Various LLVM-based utilities+

Availability

ü 60-day no-charge trial: download from Open XL product page
ü Obtain IBM world-class Service & Support through flexible licensing options,from dual pipe (AAS and PA)• Perpetual license (per Authorized User or per Concurrent User)• Monthly license (per Virtual Process Core): target cloud use cases,
e.g., on PowerVS instance

Recommended performance tuning options
Optimization Level

Usage recommendations-O2 and -O3

Typical starting point

Link time optimization:-flto (C/C++), -qlto (Fortran)

For workloads with lots of small
function calls

Profile guided optimization:-fprofile-generate, -fprofile-use (C/C++)-qprofile-generate, -qprofile-use (Fortran)

For workloads with lots ofbranching and function calls

More info please visit: https://www.ibm.com/docs/en/openxl-c-and-cpp-aix/17.1.0
https://www.ibm.com/docs/en/openxl-fortran-aix/17.1.0

10

IBM optimizations
ü Full exploitation of Power architectureü Industry-leading advanced optimizations
ü World-class Support & Service

Full Power10 architecture exploitation with Open XL 17.1.0

ü New compiler option ‘–mcpu=pwr10’ to generate code exploiting Power10
instructions and also automatically tune the optimizations for Power10
ü New builtin functions to unlock new Power10 functionalities, e.g., Matrix Multiply
Accelerator (MMA)ü New MASS SIMD and vector libraries added for Power10. All MASS libraryfunctions (SIMD, vector, scalar) tuned for Power10 (also Power9).
Note: Applications compiled with earlier versions of XL Compilers (e.g., XL 16.1.0)
to run on previous Power processors will run compatibly on Power10.

Binary Compatibility on AIX

Note: XL C/C++ for AIX 16.1.0 already introduced a new invocation xlclang++
which leverages the Clang front-end from LLVM projectü C++ objects built with xlC for AIX (based on IBM’s own front-end) are notbinary compatible with C++ objects built with xlclang++ 16.1.0 for AIX
ü C++ objects built with xlclang++ 16.1.0 for AIX will be binary compatible with
new Open XL C/C++ for AIX 17.1.0
ü C compatibility is maintained across all AIX compilers (earlier XL versions for
AIX, Open XL C/C++ for AIX 17.1.0)
ü Fortran compatibility is maintained between earlier XLF version for AIX andOpen XL Fortran for AIX 17.1.0© 2021 IBM Corporation

IBM® Power®

Power10 Quick-start Guide – GNU Compiler Collection (GCC)

Availability• The GCC compilers are available on all Enterprise Linux distributions and on
AIX.
• The installed GCC version is 8.4 on RHEL 8 and 7.4 on SLES 15. RHEL 9 is
expected to ship GCC 11.2.
• There are several ways to obtain a sufficiently recent version of GCC when
the default compilers for the distribution are too old to support Power10.
•••

Red Hat supports the GCC Toolset [1] for this purpose.
SUSE provides the Development Tools Module. [2]IBM provides the latest compilers and libraries via the Advance Toolchain. [3]

IBM Advance Toolchain• The Advance Toolchain provides Power-optimized system libraries alongwith the compilers, debuggers, and other tools.
• Building code with the Advance Toolchain can produce the most highly
optimized code possible on the latest processors.

Languages• C (gcc), C++ (g++), and Fortran (gfortran), along with others such as Go(gccgo), D (gdc), and Ada (gnat).
• Only gcc, g++, and gfortran are usually installed by default.
• The golang compiler [4] is the preferred alternative for building Go programson Power.

Compatibility and New Features on Power10• Applications compiled with earlier versions of GCC to run on POWER8 or
POWER9 processors will run compatibly on Power10 processors.
• GCC 11.2 or later is recommended to exploit all new features available inPower ISA 3.1 and implemented in Power10 processors.
• GCC 11.2 provides access to the Matrix Multiply Assist (MMA) featureprovided by Power10 processors. [5]• MMA programs can be compiled using any of the GCC, LLVM, and Open XL
compilers, provided you use sufficiently recent releases.
11

IBM Recommended and Supported Compiler Flags [6]-O3 or -Ofast-mcpu=powern-flto-funroll-loops

Aggressive optimization. -Ofast is essentially equivalent to -O3-ffast-math, which also relaxes restrictions on IEEE floatingpoint arithmetic.
Compile using instructions supported by the Powern processor.
For example, to use instructions available only on Power10,
select -mcpu=power10.
Optional. Perform “link-time” optimization. This optimizes codeacross function calls where the caller and called functions exist
in different compilation units, and can often provide a significant
performance boost.
Optional. Perform more aggressive duplication of loop bodies
than the compiler normally would. Generally you should omit
this, but on some codes this can provide better performance.

Note:
Although -mcpu=power10 is supported as early as GCC 10.3, GCC 11.2 is preferredbecause earlier compilers don’t support every feature implemented in the Power10
processors. Also, objects created using -mcpu=power10 will not run on POWER9 or
earlier processors! However, there are ways to create code that is optimized for
different processor versions. [7][1] Red Hat: Using GCC Toolset. https://access.redhat.com/documentation/enus/red_hat_enterprise_linux/8/html/developing_c_and_cpp_applications_in_rhel_8/gcc-toolset_toolsets.
[2] SUSE: Understanding the Development Tools Module. https://www.suse.com/c/suse-linux-essentialswhere-are-the-compilers-understanding-the-development-tools-module/.
[3] Advance Toolchain for Linux on IBM Power Systems. https://www.ibm.com/support/pages/advancetoolchain-linux-power.
[4] Go Language. https://golang.org.
[5] Matrix-Multiply Assist Best Practices Guide.
http://www.redbooks.ibm.com/redpapers/pdfs/redp5612.pdf.
[6] Using the GNU Compiler Collection. https://gcc.gnu.org/onlinedocs/gcc.pdf.
[7] Target-Specific Optimization with the GNU Indirect Function Mechanism.
https://developer.ibm.com/tutorials/optimized-libraries-for-linux-on-power/#target-specific-optimizationwith-the-gnu-indirect-function-mechanism.

© 2021 IBM Corporation

IBM® Power®

P10 Quick-start Guide – Java Applications

Java applications can seamlessly take advantage of new P10 ISA features on operating systems running in P10 mode by using theJava runtime versions listed below or newer:Java 8§ IBM SDK 8 SR6 FP36
§ IBM Semeru Runtime Open Edition 8u302: openj9-0.27.1

Java 11§ IBM Semeru Runtime Certified Edition 11.0.12.1: openj9-0.27.1§ IBM Semeru Runtime Open Edition 11.0.12.1: openj9-0.27.1

Java 17 (drivers may not be available yet)

§ IBM Semeru Runtime Certified Edition 17: openj9-0.28§ IBM Semeru Runtime Open Edition 17: openj9-0.28§ OpenJDK 17

Performance tuning references:
IBM WebSphere Application Server Performance Cookbook

12© 2021 IBM Corporation

IBM® Power®

Power10 Quick-start Guide – Oracle Database w/ AIX Best Practices

Page Size

Virtual Processor Folding

The general recommendation for most Oracle databases on AIX is to utilize 64KB page
size and not 16MB page size for the SGA. Typically, 64 KB pages yield nearly the same
performance benefit as 16 MB pages without special management.

This is a critical setting in a RAC environment when using LPARs withprocessor folding enabled. If this setting is not adjusted, there is ahigh risk of RAC node evictions under light database workload
conditions. schedo -p -o vpm_xvcpus=2

TNS Listener

VIOS & RAC Interconnect

Oracle 12.1 database and later releases by default will use 64k pages for text, data andstack. However, for the TNSLISTENER it still uses 4k pages for text, data and stack. To
enable 64k page for the listener use the export command prior to starting the listenerprocess. Note that running in an ASM based environment that the listener runs out of
GRID_HOME and not ORACLE_HOME.

A dedicated 10G (i.e, 10G Ethernet Adapter) connection isrecommended as minimum to provide sufficient bandwidth forcluster timing sensitive traffic. RAC cluster traffic - interconnect
traffic should be dedicated and not shared. Sharing of interconnect
can cause timing delays leading to node hang/eviction issues.

The documentation for the “srvctl setenv” command changed in 12.1 or later releases.
The -t or -T was removed in favor of -env or -envs. In the Oracle Listener environment set
and export:

Network Performance- LDR_CNTRL=DATAPSIZE=64K@TEXTPSIZE=64K@STACKPSIZE=64K <tnslistener user >This is a long-standing network tuning suggestion for Oracle on AIX,
although the default remains at 0. TCP Setting of rfc1323=1- VMM_CNTRL=vmm_fork_policy=COR (add the ‘Copy on Read’ command)

Shared SYMTABThe LDR_CNTRL=SHARED_SYMTAB=Y setting does not need to be specifically set in
11.2.0.4 or later releases. The compiler linker options take care of this setting and nolonger needs to be specifically set. It is not recommended to have
LDR_CNTRL=SHARED_SYMTAB=Y specifically set in 12c or later releases.

13

More comprehensive information
Refer to link: Managing the Stability and Performance of current
Oracle Database versions running AIX on Power Systems including
POWER9https://www.ibm.com/support/pages/node/6355543© 2021 IBM Corporation

IBM® Power®

POWER10 Quick Start Guide - Recommendations for Db2

General§ Use SMT8 mode
§ Use dedicated CPU LPARs
Db2 Warehouse§ Ensure that a high-speed private network exists between all nodes§ Limit MLN configuration to one node per socket
CP4D§ Use PCIe4 for OCP nodes network§ Prior to OCP 4.8, set kernel parameter slub_max_order=0
Db2 Best Practices
https://www.ibm.com/docs/en/db2/11.5?topic=overviews-db2-best-practices

14© 2021 IBM Corporation

IBM® Power®

Power10 Quick-start Guide – OCP Performance

Network••••

For pod network, use private network based on native SRIOV if LPM not required, otherwise use VNICFor application that require high bandwidth or low latency, consider using the SR-IOV Network Operator to assign VF directly to a pod
For services in need of a low timeout, configure the default timeouts for an existing route
Adjust the desired MTU size OCP's cluster network

Operating system• Consider increasing the u-limits within the CoreOS Post-install changes• Refer to the minimum OCP installation requirements for Power platform OCP4.8 installation on Power

Deployment• When deploying applications , note that one vCPU is equivalent to one physical core when simultaneous multithreading (SMT), or hyperthreading, is notenabled. When SMT enabled, a VCPU is equivalent to a hardware thread.
• Refer to minimum sizing guidelines for workers & master nodes Minimum resource requirements• Allocate a separate dedicated storage to the built-in container image registry• Use the following sizing guidelines for OCP’s main directories main directories that OpenShift Container Platform components write data to.

15© 2021 IBM Corporation