Chapter 4. Installation This chapter describes the installation of the additional SAP HANA instance. 4.1. Check the 2-node Base Installation with a failover test Verify that the installation is done based on Automating SAP HANA Scale-Up System Replication using the RHEL HA Add-On . To be able to use SAP HANA Multitarget System Replication , the version of resource-agents-sap-hana must be 0.162.1 or later. This can be checked, as shown below: # rpm -q resource-agents-sap-hana You can run a failover test to ensure that the environment is working. You can move the SAPHana resource, which is also described in Failover the SAPHana Resource using Move . 4.2. Install SAP HANA on third site On the third site, you also need to install SAP HANA using the same version and parameters as for the SAP HANA instances on the two-node Pacemaker cluster as shown below: Parameter Value SID RH2 InstanceNumber 02 <sid>adm user ID rh2adm 999 sapsys group ID sapsys 999 The SAP HANA installation is done using hdblcm . For more details, see SAP HANA Installation using hdbclm . Optionally, the installation can also be done using Ansible. In the examples in this chapter, we are using: hosts:clusternode1 on site DC1, clusternode2 on site DC2 and remotehost3 on site DC3 SID RH2 adminuser rh2adm 4.3. Setup SAP HANA System Replication to the third site In the existing installation, there is already SAP HANA System Replication configured between the primary and secondary SAP HANA instance in a two-node cluster. SAP HANA System Replication is enabled on the up and running primary SAP HANA database instance. This chapter describes how to register the third SAP HANA instance as an additional secondary HANA System Replication site on node remotehost3 at site DC3. This step is similar to the registration of the original secondary HANA instance (DC2) on node clusternode2. More details are described in the following chapters. If you need further information, you can also check General Prerequisites for Configuring SAP HANA System Replication . 4.3.1. Check the primary database You must check that the other databases are running and the system replication is working properly. Please refer to: Check database Check SAP HANA System Replication status Discover primary and secondary SAP HANA database You can discover the primary HANA instance with: 4.3.2. Copy database keys Before you are able to register a new secondary HANA instance, the database keys of the primary HANA instance need to be copied to the new additional HANA replication site. In our example, the hostname of the third site is remotehost3. For example, on the primary node clusternode1 run: clusternode1:rh2adm> scp -rp /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_USD{SAPSYSTEMNAME}.DAT remotehost3:/usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_USD{SAPSYSTEMNAME}.DAT clusternode1:rh2adm> scp -rp /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_USD{SAPSYSTEMNAME}.KEY remotehost3:/usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_USD{SAPSYSTEMNAME}.KEY 4.3.3. Register the additional HANA instance as a secondary HANA replication site You need to know the hostname of the node that is running the primary database . To monitor the registration, you can run the following command in a separate terminal on the primary node: clusternode1:rh2adm> watch python /usr/sap/USD{SAPSYSTEMNAME}/HDBUSD{TINSTANCE}/python_support/systemReplicationStatus.py This will show you the progress and any errors if they occur. To register the HANA instance on the third site (DC3) as an additional secondary SAP HANA instance, run the following command on the third site host remotehost3: remotehost3:rh2adm> hdbnsutil -sr_register --name=DC3 --remoteHost=clusternode1 --remoteInstance=USD{TINSTANCE} --replicationMode=async --operationMode=logreplay --online In this example, DC3 is the name of the third site, clusternode1 is the hostname of the primary node. If the database instance is already running, you don't have to stop it, you can use the option --online , which will register the instance while it is online. The necessary restart (stop and start) of the instance will then be initiated by hdbnsutil itself. Note The option --online works in any case, both when the HANA instance is online and offline (this option is available with SAP HANA 2.0 SPS04 and later). If the HANA instance is offline, you have to start it after the third node is registered. You can find additional information in SAP HANA System Replication . 4.3.4. Add SAP HANA Multitarget System Replication autoregister support We are using a SAP HANA System Replication option called register_secondaries_on_takeover = true . This will automatically re-register a secondary HANA instance with the new primary site in case of a failover between the primary site and the other secondary site. This option must be added to the global.ini file on all potential primary sites. All HANA instances should have this entry in their global.ini : [system_replication] register_secondaries_on_takeover = true The following two chapters describe the global.ini configuration in detail. Caution Despite the parameter, if the additional secondary HANA instance on the third node is down when the failover is initiated, this HANA instance needs to be re-registered manually. 4.3.5. Configure global.ini on the pacemaker nodes The option register_secondaries_on_takeover = true needs to be added to the global.ini of the SAP HANA instances, which are managed by the pacemaker cluster. Please edit the file global.ini always on the respective node, and do not copy the file from another node. Note The global.ini file should only be edited if the HANA instance of a site has stopped processing. Edit the global.ini as the rh2adm user: clusternode1:rh2adm> vim /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/hdb/custom/config/global.ini Example: # global.ini last modified 2023-07-14 16:31:14.120444 by hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=02 --replicationMode=syncmem --operationMode=logreplay --name=DC2 [multidb] mode = multidb database_isolation = low singletenant = yes [ha_dr_provider_SAPHanaSR] provider = SAPHanaSR path = /hana/shared/myHooks execution_order = 1 [persistence] basepath_datavolumes = /hana/data/RH2 basepath_logvolumes = /hana/log/RH2 log_mode = normal enable_auto_log_backup = true [system_replication] register_secondaries_on_takeover = true timetravel_logreplay_mode = auto operation_mode = logreplay mode = primary actual_mode = syncmem site_id = 1 site_name = DC2 [system_replication_site_masters] 2 = clusternode1:30201 [trace] ha_dr_saphanasr = info This option is active as soon as the SAP HANA database instance is started. 4.3.6. Configure global.ini on the third site Edit the global.ini as a <sid>adm user: remotehost3:rh2adm> vim /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/hdb/custom/config/global.ini On remotehost3, the ha_dr_provider_SAPHanaSR section is not used. Example of global.ini on remotehost3: # global.ini last modified 2023-06-22 17:22:54.154508 by hdbnameserver [multidb] mode = multidb database_isolation = low singletenant = yes [persistence] basepath_datavolumes = /hana/data/RH2 basepath_logvolumes = /hana/log/RH2 log_mode = normal enable_auto_log_backup = true [system_replication] operation_mode = logreplay register_secondaries_on_takeover = true reconnect_time_interval = 5 timetravel_logreplay_mode = auto site_id = 3 mode = syncmem actual_mode = syncmem site_name = DC3 [system_replication_site_masters] 2 = clusternode1:30201 4.3.7. Verify installation After the installation, you have to check if all HANA instances are up and running and that HANA System Replication is working between them. The easiest way is to check the systemReplicationStatus as described in more detail in Check the System Replication status . Please also refer to Check Database status , for further information. For HANA System Replication to work correctly, please ensure that the "log_mode" parameter is set to "normal". Please refer to Checking the log_mode of the SAP HANA database , for more information. To verify that the setup is working as expected, please run the test cases as described in the following chapters.

Commands:
rpm -q resource-agents-sap-hana
clusternode1:rh2adm> hdbnsutil -sr_state | egrep -e "primary masters|^mode" mode: primary
clusternode1:rh2adm> scp -rp /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_USD{SAPSYSTEMNAME}.DAT remotehost3:/usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/data/SSFS_USD{SAPSYSTEMNAME}.DAT clusternode1:rh2adm> scp -rp /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_USD{SAPSYSTEMNAME}.KEY remotehost3:/usr/sap/USD{SAPSYSTEMNAME}/SYS/global/security/rsecssfs/key/SSFS_USD{SAPSYSTEMNAME}.KEY
clusternode1:rh2adm> watch python /usr/sap/USD{SAPSYSTEMNAME}/HDBUSD{TINSTANCE}/python_support/systemReplicationStatus.py
remotehost3:rh2adm> hdbnsutil -sr_register --name=DC3 --remoteHost=clusternode1 --remoteInstance=USD{TINSTANCE} --replicationMode=async --operationMode=logreplay --online
[system_replication] register_secondaries_on_takeover = true
clusternode1:rh2adm> vim /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/hdb/custom/config/global.ini
global.ini last modified 2023-07-14 16:31:14.120444 by hdbnsutil -sr_register --remoteHost=remotehost3 --remoteInstance=02 --replicationMode=syncmem --operationMode=logreplay --name=DC2 [multidb] mode = multidb database_isolation = low singletenant = yes [ha_dr_provider_SAPHanaSR] provider = SAPHanaSR path = /hana/shared/myHooks execution_order = 1 [persistence] basepath_datavolumes = /hana/data/RH2 basepath_logvolumes = /hana/log/RH2 log_mode = normal enable_auto_log_backup = true [system_replication] register_secondaries_on_takeover = true timetravel_logreplay_mode = auto operation_mode = logreplay mode = primary actual_mode = syncmem site_id = 1 site_name = DC2 [system_replication_site_masters] 2 = clusternode1:30201 [trace] ha_dr_saphanasr = info
remotehost3:rh2adm> vim /usr/sap/USD{SAPSYSTEMNAME}/SYS/global/hdb/custom/config/global.ini
global.ini last modified 2023-06-22 17:22:54.154508 by hdbnameserver [multidb] mode = multidb database_isolation = low singletenant = yes [persistence] basepath_datavolumes = /hana/data/RH2 basepath_logvolumes = /hana/log/RH2 log_mode = normal enable_auto_log_backup = true [system_replication] operation_mode = logreplay register_secondaries_on_takeover = true reconnect_time_interval = 5 timetravel_logreplay_mode = auto site_id = 3 mode = syncmem actual_mode = syncmem site_name = DC3 [system_replication_site_masters] 2 = clusternode1:30201