## 3.2.6  Apply NUMA Node Affinity Settings
If the physical hardware on a host is shared between several processes, you can use CPU affinity settings to assign a set of logical cores to a specific SAP HANA process. These settings are coarse-grained and apply on the OS and process-level.
## Prerequisites
You can use the affinity configuration parameter to restrict CPU usage of SAP HANA server processes to certain CPUs or ranges of CPUs.
Start by analyzing how the system CPUs are configured using the Linux lscpu command and then, based on the information returned, apply affinity settings in daemon.ini to bind specific processes to logical CPU cores. Processes must be restarted before the changes become effective. This approach applies primarily to the use cases of SAP HANA tenant databases and multiple SAP HANA instances on one server; you can use this approach, for example, to partition the CPU resources of the system by tenant database.
To make the changes described here, you require access to the operating system of the SAP HANA instance to run the lscpu command and you require the privilege INIFILE ADMIN.
## Context
For Xen and VMware, the users in the VM guest system see what is configured in the VM host. So, the quality of the reported information depends on the configuration of the VM guest. Therefore, SAP can't give any performance guarantees in this case.
## Procedure
1. Firstly, to confirm the physical and logical details of your CPU architecture, analyze the system using the lscpu command. This command returns a listing of details of the system architecture. The following table gives a commentary on the most useful values based on an example system with 2 physical chips (sockets), each containing 8 physical cores. These cores are hyperthreaded to give a total of 32 logical cores.
|   # | Feature      | Example Value   |
|-----|--------------|-----------------|
|   1 | Architecture | x86_64          |
|   2 | CPU op-modes | 32-bit, 64-bit  |
|   3 | Byte Order   | LittleEndian    |
|   4 | CPUs              | 32              |
|   5 | On-line CPUs list | 0-31            |
|   6 | Threads per core  | 2               |
|   7 | Cores per socket  | 8               |
|   8 | Sockets           | 2               |
|   9 | NUMAnodes         | 2               |
|  21 | NUMAnode0CPUs     | 0-7,16-23       |
|  22 | NUMAnode1 CPUs    | 8-15,24-31      |
- 4-5: This example server has 32 logical cores numbered 0-31
- 6-8: Logical cores ("threads") are assigned to physical cores. Hyperthreading is where multiple threads are assigned to a single physical core In this example, there are 2 sockets, with each socket containing 8 physical cores (total 16). Two logical cores (#6) are assigned to each physical core, thus, each core exposes two execution contexts for the independent and concurrent execution of two threads.
- 9: In this example there are 2 NUMA nodes, one for each socket. Other systems may have multiple NUMA nodes per socket.
- 21-22: The 32 logical cores are numbered and assigned to one of the two NUMA nodes.
##  Note
Even on a system with 32 logical cores and two sockets the assignment of logical cores to physical CPUs and sockets can be different. It's important to collect the assignment in advance before making changes. A more detailed analysis is possible using the system commands described in the next step. These provide detailed information for each core including how CPU cores are grouped as siblings.
2. In addition to the lscpu command, you can use the set of system commands in the /sys/devices/ system/cpu/ directory tree. For each logical core, there is a numbered subdirectory beneath this node ( /cpu12/ in the following examples). The examples show how to retrieve this information and the table gives details of some of the most useful commands available:
##  Example
cat /sys/devices/system/cpu/present cat /sys/devices/system/cpu/cpu12/topology/thread_siblings_list
| Command   | Example Output   | Commentary                                            |
|-----------|------------------|-------------------------------------------------------|
| present   | 0-15             | The number of logical cores available for scheduling. |
| Command                             | Example Output   | Commentary                                                             |
|-------------------------------------|------------------|------------------------------------------------------------------------|
| cpu12/topology/core_siblings_list   | 4-7, 12-15       | The cores on the same socket.                                          |
| cpu12/topology/thread_siblings_list | 4, 12            | The logical cores assigned to the same physical core (hyperthreading). |
| cpu12/topology/physical_package_id  | 1                | The socket of the current core - in this case cpu12.                   |
Other Linux commands that are relevant here are sched_setaffinity and numactl . sched_setaffinity limits the set of CPU cores available (by applying a CPU affinity mask) for execution of a specific process (this can be used, for example, to isolate tenants) and numactl controls NUMA policy for processes or shared memory.
3. Based on the results returned you can use the affinity setting to restrict CPU usage of SAP HANA server processes to certain CPUs or ranges of CPUs. You can set restrictions for the following servers: nameserver, indexserver, compileserver, preprocessor, and xsengine (each server has a section in the daemon.ini file). The affinity setting is applied by the TrexDaemon when it starts the other HANA processes using the command sched_setaffinity . Changes to the affinity settings take effect only after restarting the HANA process. The following examples and commentary show the syntax for the ALTER SYSTEM CONFIGURATION commands required.
##  Example
To restrict the nameserver to two logical cores of the first CPU of socket 0 (see line 21 in the previous example), use the following affinity setting:
```
ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini', 'SYSTEM') SET ('nameserver', 'affinity') = '0,16'
```
##  Example
To restrict the preprocessor and the compileserver to all remaining cores (that is, all except 0 and 16) on socket 0 (see line 21 in the previous example), use the following affinity settings:
```
ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini', 'SYSTEM') SET ('preprocessor', 'affinity') = '1-7,17-23' ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini', 'SYSTEM') SET ('compileserver', 'affinity') = '1-7,17-23'
```
##  Example
To restrict the indexserver to all cores on socket 1 (see line 22 in the previous example), use the following affinity settings:
```
ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini', 'SYSTEM') SET ('indexserver', 'affinity') = '8-15,24-31'
```
4. You can assign affinities to different tenants of a multi-tenant database on the same host as shown here. Run these SQL statements on the SYSTEMDB.
```
 Example In this scenario tenant NM1 already exists, here we add another tenant NM2: CREATE DATABASE NM2 ADD AT LOCATION 'host:30040' SYSTEM USER PASSWORD Manager1; Set the configuration parameter to bind CPUs to specific NUMA nodes on each tenant. You can use the following notation with a dot to identify the specific tenant: ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini','SYSTEM')  SET (' indexserver.NM1 ', 'affinity') ='0-7,16-23'; ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini','SYSTEM')  SET (' indexserver.NM2 ', 'affinity') ='8-15,24-31';
```
5. To assign affinities to multiple indexservers of the same tenant on the same host, execute the following SQL statements on the SYSTEMDB to apply the instance_affinity[port] configuration parameter:
##  Example
In this scenario, an indexserver is already running on tenant NM1 on port 30003. Here we add another indexserver on a different port:
```
ALTER DATABASE NM1 ADD 'indexserver' AT LOCATION 'host:30040';
```
Set the different instances of the instance_affinity[port] configuration parameter to bind CPUs to specific NUMA nodes on each indexserver. The configuration parameter has a 1-2 digit suffix to identify the final significant digits of the port number, in this example 3000 3 and 300 40 :
```
ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini','SYSTEM')  SET ('indexserver.NM1', 'instance_affinity[3]')='0-7,16-23'; ALTER SYSTEM ALTER CONFIGURATION ('daemon.ini','SYSTEM')  SET ('indexserver.NM1', 'instance_affinity[40]')='8-15,24-31';
```
Restart the indexserver processes to make the affinity settings effective.
6. You can test the settings either in SQL or using hdbcons as shown here:
Run this query on the tenant or SystemDB:
```
select * from M_NUMA_NODES;
```
Using hdbcons the process ID of the indexserver process is required as a parameter:
```
hdbcons -p <PID> "jexec info"
```
## Related Information
Memory and CPU Usage for Tenant Databases
SAP HANA Monitoring Views for CPU Topology Details SQL Statements to Apply NUMA Location Preferences
SAP Note 2470289