Chapter 2. Configuring SAP HANA System Replication Before the HA cluster can be configured, SAP HANA System Replication must be configured and tested according to the guidelines from SAP: SAP HANA System Replication: Configuration . The following example shows how to enable SAP HANA System Replication on the nodes that will later become part of the HA cluster that will manage the SAP HANA System Replication setup. Please refer to RHEL for SAP Subscriptions and Repositories , for more information on how to ensure the correct subscription and repos are enabled on each HA cluster node. SAP HANA configuration used in the example: SID: RH1 Instance Number: 02 node1 FQDN: node1.example.com node2 FQDN: node2.example.com node1 SAP HANA site name: DC1 node2 SAP HANA site name: DC2 SAP HANA 'SYSTEM' user password: <HANA_SYSTEM_PASSWORD> SAP HANA administrative user: rh1adm 2.1. Prerequisites Ensure that both systems can resolve the FQDN of both systems without issues. To ensure that FQDNs can be resolved even without DNS you can place them into /etc/hosts like in the example below: [root]# cat /etc/hosts ... 192.168.0.11 node1.example.com node1 192.168.0.12 node2.example.com node2 Note As documented at hostname | SAP Help Portal SAP HANA only supports hostnames with lowercase characters. For the system replication to work, the SAP HANA log_mode variable must be set to normal, which is also the default value. Please refer to SAP Note 3221437 - System replication is failed due to "Connection refused: Primary has to run in log mode normal for system replication!" , for more information. This can be verified as the SAP HANA administrative user using the command below on both nodes. [rh1adm]USD hdbsql -u system -p <HANA_SYSTEM_PASSWORD> -i 02 "select value from "SYS"."M_INIFILE_CONTENTS" where key='log_mode'" VALUE "normal" 1 row selected A lot of the configuration steps are performed by the SAP HANA administrative user for the SID that was selected during installation. For the example setup described in this document, the user id rh1adm is used for the SAP HANA administrative user, since the SID used is RH1 . To switch from the root user to the SAP HANA administrative user, you can use the following command: [root]# sudo -i -u rh1adm [rh1adm]USD 2.2. Performing an initial SAP HANA database backup SAP HANA System Replication will only work after an initial backup has been performed on the HANA instance that will be the primary instance for the SAP HANA System Replication setup. The following shows an example for creating an initial backup in /tmp/foo directory. Please note that the size of the backup depends on the database size and may take some time to complete. The directory to which the backup will be placed must be writable by the SAP HANA administrative user. On single-tenant SAP HANA setups, the following command can be used to create the initial backup: [rh1adm]USD hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "BACKUP DATA USING FILE ('/tmp/foo')" 0 rows affected (overall time xx.xxx sec; server time xx.xxx sec) On multi-tenant SAP HANA setups, the SYSTEMDB and all tenant databases need to be backed up. The following example shows how to backup the SYSTEMDB : [rh1adm]USD hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> -d SYSTEMDB "BACKUP DATA USING FILE ('/tmp/foo')" 0 rows affected (overall time xx.xxx sec; server time xx.xxx sec) [rh1adm]# hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> -d SYSTEMDB "BACKUP DATA FOR RH1 USING FILE ('/tmp/foo-RH1')" 0 rows affected (overall time xx.xxx sec; server time xx.xxx sec) Please check the SAP HANA documentation on how to backup the tenant databases. 2.3. Configuring the SAP HANA primary replication instance After the initial backup has been successfully completed, initialize SAP HANA System Replication with the following command: [rh1adm]USD hdbnsutil -sr_enable --name=DC1 checking for active nameserver ... nameserver is active, proceeding ... successfully enabled system as system replication source site done. Verify that after the initialization the SAP HANA System Replication status shows the current node as 'primary': [rh1adm]#USD hdbnsutil -sr_state checking for active or inactive nameserver ... System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~ mode: primary site id: 1 site name: DC1 Host Mappings: 2.4. Configuring the SAP HANA secondary replication instance After installing the secondary SAP HANA instance on the other HA cluster node using the same SID and instance number as the SAP HANA primary instance, it needs to be registered to the already running SAP HANA primary instance. The SAP HANA instance that will become the secondary replication instance needs to be stopped first before it can be registered to the primary instance: [rh1adm]USD HDB stop When the secondary SAP HANA instance has been stopped, copy the SAP HANA system PKI SSFS_RH1.KEY and SSFS_RH1.DAT files from the primary SAP HANA instance to the secondary SAP HANA instance: [rh1adm]USD scp root@node1:/usr/sap/RH1/SYS/global/security/rsecssfs/key/SSFS_RH1.KEY /usr/sap/RH1/SYS/global/security/rsecssfs/key/SSFS_RH1.KEY ... [rh1adm]USD scp root@node1:/usr/sap/RH1/SYS/global/security/rsecssfs/data/SSFS_RH1.DAT /usr/sap/RH1/SYS/global/security/rsecssfs/data/SSFS_RH1.DAT ... Please refer to SAP Note 2369981 - Required configuration steps for authentication with HANA System Replication , for more information. Now the SAP HANA secondary replication instance can be registered to the SAP HANA primary replication instance with the following command: [rh1adm]USD hdbnsutil -sr_register --remoteHost=node1 --remoteInstance=USD{TINSTANCE} --replicationMode=syncmem --operationMode=logreplay --name=DC2 adding site ... checking for inactive nameserver ... nameserver node2:30201 not responding. collecting information ... updating local ini files ... done. Please choose the values for replicationMode and operationMode according to your requirements for HANA System Replication. Please refer to Replication Modes for SAP HANA System Replication and Operation Modes for SAP HANA System Replication , for more information. When the registration is successful, the SAP HANA secondary replication instance can be started again: [rh1adm]USD HDB start Verify that the secondary node is running and that 'mode' matches the value used for the replicationMode parameter in the hdbnsutil -sr_register command. If registration was successful, the SAP HANA System Replication status on the SAP HANA secondary replication instance should look similar to the following: [rh1adm]USD hdbnsutil -sr_state checking for active or inactive nameserver ... System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~ mode: syncmem site id: 2 site name: DC2 active primary site: 1 Host Mappings: ~~~~~~~~~~~~~~ node2 -> [DC1] node1 node2 -> [DC2] node2 2.5. Checking SAP HANA System Replication state To check the current state of SAP HANA System Replication, you can use the systemReplicationStatus.py Python script provided by SAP HANA as the SAP HANA administrative user on the current primary SAP HANA node. On single tenant SAP HANA setups, the output should look similar to the following: [rh1adm]USD python /usr/sap/RH1/HDB02/exe/python_support/systemReplicationStatus.py | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication | | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details | | ----- | ----- | ------------ | --------- | ------- | --------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- | | node1 | 30201 | nameserver | 1 | 1 | DC1 | node2 | 30201 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | node1 | 30207 | xsengine | 2 | 1 | DC1 | node2 | 30207 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | node1 | 30203 | indexserver | 3 | 1 | DC1 | node2 | 30203 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | status system replication site "2": ACTIVE overall system replication status: ACTIVE Local System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mode: PRIMARY site id: 1 site name: DC1 On multi-tenant SAP HANA setups, the output should look similar to the following: [rh1adm]USD python /usr/sap/RH1/HDB02/exe/python_support/systemReplicationStatus.py | Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication | | | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details | | -------- | ----- | ----- | ------------ | --------- | ------- | --------- | ----------| --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- | | SYSTEMDB | node1 | 30201 | nameserver | 1 | 1 | DC1 | node2 | 30201 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | RH1 | node1 | 30207 | xsengine | 2 | 1 | DC1 | node2 | 30207 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | RH1 | node1 | 30203 | indexserver | 3 | 1 | DC1 | node2 | 30203 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | status system replication site "2": ACTIVE overall system replication status: ACTIVE Local System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mode: PRIMARY site id: 1 site name: DC1 In both cases, please also check the return code: echo USD? 15 A return code of 15 (Active) is fine. 14 means synchronizing and 13 is initiaizing. 2.6. Testing SAP HANA System Replication The test phase is a very important phase to verify if the KPIs are met and the landscape performs the way it was configured. If the SAP HANA System Replication setup does not work as expected without the HA cluster, it can lead to unexpected behavior when the HA cluster is configured later on to manage the SAP HANA System Replication setup. Therefore, a few test cases are suggested below as guidelines, which should be enhanced by your specific requirements. The tests should be performed with realistic data loads and sizes. Test case Description Full Replication Measure how long the initial synchronization takes, from when Lost Connection Measure how long it takes until primary and secondary are Takeover Measure how long it takes for the secondary system to be fully Data Consistency Create or change data, then perform a takeover and check if the data is still available. Client Reconnect Test client access after a take-over, to check if the DNS/Virtual IP switch worked. Primary becomes secondary Measure how long it takes until both systems are in sync, when the former primary becomes the secondary after a takeover. Please refer to section "9. Testing", in How To Perform System Replication for SAP HANA , for more information.

Commands:
SID: RH1 Instance Number: 02 node1 FQDN: node1.example.com node2 FQDN: node2.example.com node1 SAP HANA site name: DC1 node2 SAP HANA site name: DC2 SAP HANA 'SYSTEM' user password: <HANA_SYSTEM_PASSWORD> SAP HANA administrative user: rh1adm
cat /etc/hosts 192.168.0.11 node1.example.com node1 192.168.0.12 node2.example.com node2
[rh1adm]USD hdbsql -u system -p <HANA_SYSTEM_PASSWORD> -i 02 "select value from "SYS"."M_INIFILE_CONTENTS" where key='log_mode'" VALUE "normal" 1 row selected
sudo -i -u rh1adm [rh1adm]USD
[rh1adm]USD hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> "BACKUP DATA USING FILE ('/tmp/foo')" 0 rows affected (overall time xx.xxx sec; server time xx.xxx sec)
[rh1adm]USD hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> -d SYSTEMDB "BACKUP DATA USING FILE ('/tmp/foo')" 0 rows affected (overall time xx.xxx sec; server time xx.xxx sec) hdbsql -i 02 -u system -p <HANA_SYSTEM_PASSWORD> -d SYSTEMDB "BACKUP DATA FOR RH1 USING FILE ('/tmp/foo-RH1')" 0 rows affected (overall time xx.xxx sec; server time xx.xxx sec)
[rh1adm]USD hdbnsutil -sr_enable --name=DC1 checking for active nameserver nameserver is active, proceeding successfully enabled system as system replication source site done.
USD hdbnsutil -sr_state checking for active or inactive nameserver System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~ mode: primary site id: 1 site name: DC1 Host Mappings:
[rh1adm]USD HDB stop
[rh1adm]USD scp root@node1:/usr/sap/RH1/SYS/global/security/rsecssfs/key/SSFS_RH1.KEY /usr/sap/RH1/SYS/global/security/rsecssfs/key/SSFS_RH1.KEY [rh1adm]USD scp root@node1:/usr/sap/RH1/SYS/global/security/rsecssfs/data/SSFS_RH1.DAT /usr/sap/RH1/SYS/global/security/rsecssfs/data/SSFS_RH1.DAT
[rh1adm]USD hdbnsutil -sr_register --remoteHost=node1 --remoteInstance=USD{TINSTANCE} --replicationMode=syncmem --operationMode=logreplay --name=DC2 adding site checking for inactive nameserver nameserver node2:30201 not responding. collecting information updating local ini files done.
[rh1adm]USD HDB start
[rh1adm]USD hdbnsutil -sr_state checking for active or inactive nameserver System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~ mode: syncmem site id: 2 site name: DC2 active primary site: 1 Host Mappings: ~~~~~~~~~~~~~~ node2 -> [DC1] node1 node2 -> [DC2] node2
[rh1adm]USD python /usr/sap/RH1/HDB02/exe/python_support/systemReplicationStatus.py | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication | | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details | | ----- | ----- | ------------ | --------- | ------- | --------- | --------- | --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- | | node1 | 30201 | nameserver | 1 | 1 | DC1 | node2 | 30201 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | node1 | 30207 | xsengine | 2 | 1 | DC1 | node2 | 30207 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | node1 | 30203 | indexserver | 3 | 1 | DC1 | node2 | 30203 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | status system replication site "2": ACTIVE overall system replication status: ACTIVE Local System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mode: PRIMARY site id: 1 site name: DC1
[rh1adm]USD python /usr/sap/RH1/HDB02/exe/python_support/systemReplicationStatus.py | Database | Host | Port | Service Name | Volume ID | Site ID | Site Name | Secondary | Secondary | Secondary | Secondary | Secondary | Replication | Replication | Replication | | | | | | | | | Host | Port | Site ID | Site Name | Active Status | Mode | Status | Status Details | | -------- | ----- | ----- | ------------ | --------- | ------- | --------- | ----------| --------- | --------- | --------- | ------------- | ----------- | ----------- | -------------- | | SYSTEMDB | node1 | 30201 | nameserver | 1 | 1 | DC1 | node2 | 30201 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | RH1 | node1 | 30207 | xsengine | 2 | 1 | DC1 | node2 | 30207 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | | RH1 | node1 | 30203 | indexserver | 3 | 1 | DC1 | node2 | 30203 | 2 | DC2 | YES | SYNCMEM | ACTIVE | | status system replication site "2": ACTIVE overall system replication status: ACTIVE Local System Replication State ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ mode: PRIMARY site id: 1 site name: DC1
echo USD? 15