## 9.2 Replication Performance Problems
If system replication appears to slow down transaction processing, you can check the network and disk I/O on the secondary site.
A slow-down related to system replication can occur in the following scenarios:
- ASYNC replication mode is configured over long distances;
- multi-tier system replication is configured and a tier 3 system is attached;
- SYNC/SYNCMEM replication mode is configured over short distances.
The following troubleshooting steps can help you determine and resolve the underlying cause.
## Check If Log Can Be Shipped in Time
You can check the system replication KPI values to analyze the problem and verify that it is really related to system replication:
- check if log shipping is significantly slower than local log write (SYNC/SYNCMEM)
- check Async Buffer Full Count (ASYNC)
You can check system replication KPIs in SAP HANA cockpit (see Monitoring SAP HANA System Replication in the SAP HANA Administration Guide ). Y ou can also get an overview of basic system replication KPIs by running the query HANA\_Replication\_SystemReplication\_Overview\_*\_MDC.txt (from SAP KBA 1969700 ). This query is based on the system view M\_SERVICE\_REPLICATION and can be used to compare log shipping time to local log write time. For synchronous replication the following KPIs are shown:
Example output of SQL Statement HANA\_Replication\_SystemReplication\_Overview.txt
| KEY                                      | VALUE               |
|------------------------------------------|---------------------|
| Replication mode                         | SYNC                |
| Secondary connect time                   | 2016/10/18 14:02:36 |
| Days since secondary connect time        | 1.26                |
| Used persistence size (GB)               | 205.76              |
| Log backup size / day (GB)               | 83.58               |
| Local log buffer write size (MB)         | 101130.54           |
| Shipped log buffer size (MB)             | 100258.52           |
| Avg. local log buffer write size (KB)    | 6.14                |
| Avg. shipped log buffer size (KB)        | 6.14                |
| Avg. local log buffer write time (ms)    | 0.13                |
| Avg. log buffer shipping time (ms)       | 0.24                |
| Local log buffer write throughput (MB/s) | 44.68               |
| Log buffer shipping throughput (MB/s)    | 24.99               |
| Initial data shipping size (MB)          | 0.00                |
| Initial data shipping time (s)           | 0.00                |
| Last delta data shipping size (MB)       | 2736.00             |
| KEY                                   | VALUE     |
|---------------------------------------|-----------|
| Last delta data shipping time (s)     | 13.00     |
| Delta data shipping size (MB)         | 758704.00 |
| Delta data shipping time (s)          | 3538.20   |
| Delta data shipping throughput (MB/s) | 214.43    |
| Delta data shipping size / day (MB)   | n/a       |
| Replication delay (s)                 | 0.00      |
The following KPIs are of particular importance, the shipping time should not be significantly higher than the local write time:
- Avg. local log buffer write time (ms)
- Avg. log buffer shipping time (ms)
You can see a graphical comparison of these local and shipped values in the cockpit System Replication Overview (Network tab). The graph displayed compares the local write wait time with the remote write wait time monitored over the last 24 hours:
<!-- image -->
For asynchronous replication scenarios the redo log is written into an Asynchronous Log Buffer, which occasionally can run full in case the logs are not shipped in a timely manner to the secondary instance. This can lead to a performance overhead on the primary site as by default it waits with new COMMITS until there is free space in the buffer. This can be avoided by setting the parameter logshipping\_async\_wait\_on\_buffer\_full in the system\_replication section of the global.ini file to FALSE .
##  Note
In order to maintain a stable connection during the initial data shipment this parameter should be set to true. This is recommended because if the log shipping connection is reset for any reason the data shipment connection is also reset and the initial data shipment has to start again from the beginning. In multi-tier scenarios a restarted full data shipment from primary to secondary site also results in a completely new full data shipment to a tertiary site. For the duration of the initial shipment, therefore, you may also increase the value of the logshipping\_timeout parameter on the primary which has a default value of 30 seconds.
The size of the asynchronous log shipping buffer on the primary site is normally adequate; the default value of the logshipping\_async\_buffer\_size parameter is 256MB for the indexserver (in the indexserver.ini) and 64MB for all other services (maintained in the global.ini). However, if additional free memory is available this value can also be increased for specific services with a high log generation (such as the indexserver). You should make such changes only in the service­specific ini files rather than in the global.ini file.
Once the asynchronous replication connection is established you can see how much is in the async buffer by checking the value of BACKLOG\_SIZE in the system view M\_SERVICE\_REPLICATION. If there is no connection this column shows the number of log entries that have been generated on the primary but which have not yet reached the secondary. You can also see this information (the backlogSize) by running the following command on the primary with admin rights:
```
hdbcons 'replication info' - backlogSize : 2781184 bytes
```
For further details of the most common performance issues caused by system replication and under which circumstances they occur, please refer to SAP KBA 1999880 - FAQ: SAP HANA System Replication .
## Check If Data Load Can Be Handled by Network Link
To estimate the required bandwidth for Data/Log shipping, use HANA\_Replication\_SystemReplication\_Bandwidth.txt (from SAP Note 1069700 ), which is based on the I/O statistics from the primary site. We recommend executing this SQL statement when system replication is disabled. The data returned will help you to estimate the amount of data/log shipped from the primary site and compare this to the available bandwidth.
You can also do a network performance test using, for example, the open source IPERF tool or similar, to measure the real application network performance. The recommended bandwidth is 10 Gbit/s.
If the network bandwidth is not adequate you can activate data and log compression which significantly reduces the shipment size by setting the following parameters in the system\_replication section of global.ini :
- enable\_log\_compression = TRUE
- enable\_data\_compression = TRUE
## Check Network Configuration (Long Distance)
Increasing the TCP window size can result in better network utilization and higher throughput. If the bandwidth can handle load, check if the network is shared and whether other applications may be interfering with performance. Collect network information on bandwidth and latency from the Linux kernel parameters as described here. For these values refer also to SAP Note 2382421 Optimizing the Network Configuration on HANA and OS-Level (Linux Kernel Parameters) :
- Check the network utilization profile for the network link to see if the maximum capacity of the network has been reached.
- If the network is not fully utilized, check the linux kernel TCP configuration with sysctl -a | egrep 'net.core|net.ipv4.tcp' .
- Check that window scaling is set to the default value of 1. net.ipv4.tcp\_window\_scaling = 1 .
- Check whether the max size can be increased for net.ipv4.tcp\_wmem and net.ipv4.tcp\_rmem .
- Calculate the Bandwidth Delay Product (BDP): Bandwidth * Latency (for example, BDP = 50ms * 3 Gbps = 19.2 MB). The BDP tells you what TCP window size is needed to use the network link fully.
## Check Disk I/O on a Secondary Site
Slow disk I/O on the secondary can postpone releasing log buffers on the primary, which results in wait situations on the primary. You can do the following:
- Use a Disk Performance Test Tool Execute fsperf on log volume, for example: $ fsperf /usr/sap/TST/SYS/global/hdb/log/mnt00001/hdb00002
- Check the Monitoring and Administration area If SQL is not available, use command line tools (this has to be done for each individual service), for example: $ hdbcons "statreg print -n M\_VOLUME\_IO\_TOTAL\_STATISTICS -h" A runtime dump also contains I/O statistics, which you can see with: $ hdbcons 'runtimedump dump'
##  Caution
Technical expertise is required to use hdbcons. To avoid incorrect usage, use hdbcons only with the guidance of SAP HANA development support.
- Check I/O relevant tables in the proxy schema of the corresponding secondary site, which provide SQL access on the primary site on statistic views of the secondary. For more information, see Monitoring Secondary Sites in the SAP HANA Administration Guide .
## Related Information
SAP Note 1969700
SAP Note 1999880
SAP Note 2382421
.
Monitoring SAP HANA System Replication with the SAP HANA Cockpit [page 175]