Chapter 37. Performance Tuning for Bulk Provisioning of Entries Adding a large number of entries using the usual workflow, such as Chapter 11, Managing User Accounts for adding users, can be very slow. This chapter describes how to tune the process to ensure the provisioning is completed as quickly as possible. As part of the procedure: Identity Management (IdM) reads entries to be provisioned from an LDIF file and then imports them to the target IdM LDAP instance. The administrator sets custom values for certain attributes, such as cache sizes, and disables the MemberOf and Schema Compatibility plug-ins. The procedure includes running the fixup-memberof.pl plug-in on the provisioned entries to compensate for disabling MemberOf. This procedure has been designed and tested to provision the following entry types: user, user group, host, host group, sudo rules, and host-based access control (HBAC) rules. Recommendations and Prerequisites for Bulk Provisioning Recommendations: When provisioning a large number of entries (10,000 or more), do not allow any LDAP client to access the server on which the entries are provisioned or to rely on the information from the server. For example, you can achieve this by disabling ports 389 and 636 on the server and using LDAPI to work over Unix sockets. Reason: The MemberOf plug-in is disabled on the server, which means that membership information on the server is not valid. Stop applications that are not required to be running during the provisioning. Reason: This helps free as much memory on the machine as possible. The free memory will be used by the file system cache, thus improving the performance of the provisioning. Note that the procedure below already includes steps to stop the IdM services, and restart only the Directory Server (DS) instance. IdM services, especially tomcat , consume a lot of memory, but are not used during the provisioning. Run the procedure on a fresh IdM deployment with only one server. Create replicas only after the provisioning has been completed. Reason: The provisioning throughput is much faster than replication. In a deployment with more that one server, information on the replicas would become significantly outdated. Prerequisites: Generate an LDIF file containing the entries you want to provision. For example, if you are migrating an existing IdM deployment, create the LDIF file by exporting all the entries using the ldapsearch utility. For details on the LDIF format, see About the LDIF File Format in the Red Hat Directory Server 10 Administration Guide . Backing up the Current DS Tuning Parameter Values Retrieve the current values for the DS tuning parameters: the database cache size and database locks: the entry cache size and DN cache size: Make note of the obtained values. You will reset the parameters back to these values after you finish the provisioning. Adjusting the Database, Domain Entry, and DN Cache Size For the database cache size: Determine the required value. The recommended value is typically between 200 MB and 500 MB. The value appropriate for your use case depends on the memory available on your system: More than 8 GB of memory 500 MB 8 GB - 4 GB of memory 200 MB Less than 4 GB of memory 100 MB Set the determined value by using this template: For an example of modifying LDAP attributes using the ldapmodify utility, see Example 37.1, "Using ldapmodify to Change an LDAP Attribute" . Example 37.1. Using ldapmodify to Change an LDAP Attribute Run the ldapmodify command, and then add the statements to modify the attribute value. For example: Press Ctrl + D to confirm and send the changes to the server. If the operation finishes successfully, the following message is displayed: For the domain entry cache size: Determine the required value. The recommended value is between 100 MB and 400 MB. The appropriate value depends on the memory available on your system: More than 4 GB of memory 400 MB 2 GB - 4 GB of memory 200 MB Less than 2 GB of memory 100 MB If you are provisioning a large static group, it is recommended that the entry cache is large enough to fit all entries: groups and members. Set the determined value by using this template: For the domain name (DN) cache size: For the best possible performance, it is recommended that the DN cache fits all the DNs of the provisioned entries. To estimate the value appropriate for your use case: Determine the number of all DN entries in the file. The DN entries are on lines starting with dn: . For example, using # grep , sed , and wc : Determine the size of all DN entry strings in the LDIF file. Get the average DN size: divide the size of all DN entry strings by the number of all the DN entries in the file. For example: 9,802,460 / 92,200 ~ 106 Get the average memory size: multiple the average DN size by 2, and then add 32 to the result. For example: (106 * 2) + 32 = 244 Get the appropriate DN cache size: multiply the average memory size by the total number of DN entries in the LDIF file. For example: 244 * 92,200 = 22,496,800 Set the determined value by using this template: Disabling Unnecessary Services and Adjusting Database Locks Disable the MemberOf and Schema Compatibility plug-ins: Disabling MemberOf significantly speeds up the provisioning. Disabling Schema Compatibility also helps reduce the duration of the operation. For an example of modifying LDAP attributes using the ldapmodify utility, see Example 37.1, "Using ldapmodify to Change an LDAP Attribute" . If no replicas are installed in your topology (as recommended in the section called "Recommendations and Prerequisites for Bulk Provisioning" ), disable the Content Synchronization and Retro Changelog plug-ins: Disabling these additional plug-ins helps improve the performance of the provisioning. Stop the IdM server. This also stops the DS instance. Stopping DS is required to set the number of database locks in the step. You will restart it again later. Adjust the number of database locks. The appropriate value equals half the number of provisioned entries. the minimum value is 10,000 the maximum value is 200,000 Because DS is stopped, you must set the value by modifying the /etc/dirsrv/slapd- EXAMPLE-COM /dse.ldif file: IdM accesses a large number of database pages when computing membership. The more pages it accesses, the more locks are required for the provisioning. Start DS: Importing the Entries To import the new entries from the LDIF file to the IdM LDAP instance. For example, using the ldapadd utility: For details on using ldapadd , see the ldapadd (1) man page. Re-enabling the Disabled Services and Restoring the Original Attribute Values Enable MemberOf: For an example of modifying LDAP attributes using the ldapmodify utility, see Example 37.1, "Using ldapmodify to Change an LDAP Attribute" . Restart DS: Restarting DS at this point is required because you enabled MemberOf in the step. Run the fixup-memberof.pl script with the (objectClass=*) filter to regenerate and update the memberOf attribute on all provisioned entries. For example: Running fixup-memberof.pl is necessary because the MemberOf plug-in was disabled when you imported the entries. To be able to continue with the provisioning, the script must complete successfully. For details on fixup-memberof.pl , see the fixup-memberof.pl (8) man page. Enable the Schema Compatibility plug-in: If you disabled the Content Synchronization and Retro Changelog plug-ins in the section called "Disabling Unnecessary Services and Adjusting Database Locks" , re-enable them: Restore the original values for the database cache, entry cache, and DN cache size that you backed up in the section called "Backing up the Current DS Tuning Parameter Values" : Stop DS: Restore the original value for database locks that you backed up in the section called "Backing up the Current DS Tuning Parameter Values" . Because DS is stopped, you must set the value by modifying the /etc/dirsrv/slapd- EXAMPLE-COM /dse.ldif file: Start the IdM server: This starts all IdM services, including DS.

Commands:
ldapsearch -D "cn=directory manager" -w secret -b "cn=config,cn=ldbm database,cn=plugins,cn=config" nsslapd-dbcachesize nsslapd-db-locks nsslapd-dbcachesize: 10000000 nsslapd-db-locks: 50000
ldapsearch -D "cn=directory manager" -w secret -b "cn=userRoot,cn=ldbm database,cn=plugins,cn=config" nsslapd-cachememsize nsslapd-dncachememsize nsslapd-cachememsize: 10485760 nsslapd-dncachememsize: 10485760
dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dbcachesize nsslapd-dbcachesize: db_cache_size_in_bytes
ldapmodify -D "cn=directory manager" -w secret -x dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dbcachesize nsslapd-dbcachesize: 200000000
modifying entry "cn=config,cn=ldbm database,cn=plugins,cn=config"
dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-cachememsize nsslapd-cachememsize: entry_cache_size_in_bytes
grep '^dn: ' ldif_file | sed 's/^dn: //' | wc -l 92200
grep '^dn: ' ldif_file | sed 's/^dn: //' | wc -c 9802460
dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config changetype: modify Replace: nsslapd-dncachememsize Nsslapd-dncachememsize: dn_cache_size
dn: cn=MemberOf Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
dn: cn=Schema Compatibility,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
dn: cn=Content Synchronization,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
dn: cn=Retro Changelog Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: off
ipactl stop
dn: cn=config,cn=ldbm database,cn=plugins,cn=config nsslapd-db-locks: db_lock_number
systemctl start dirsrv.target
ldapadd -D " binddn " -y password_file -f ldif_file
dn: cn=MemberOf Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
systemctl restart dirsrv.target
fixup-memberof.pl -D "cn=directory manager" -j password_file -Z server_id -b " suffix " -f "(objectClass=*)" -P LDAP
dn: cn=Schema Compatibility,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
dn: cn=Content Synchronization,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
dn: cn=Retro Changelog Plugin,cn=plugins,cn=config changetype: modify replace: nsslapd-pluginEnabled nsslapd-pluginEnabled: on
dn: cn=config,cn=ldbm database,cn=plugins,cn=config changetype: modify replace: nsslapd-dbcachesize nsslapd-dbcachesize: backup_db_cache_size dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config changetype: modify Replace: nsslapd-dncachememsize Nsslapd-dncachememsize: backup_dn_cache_size - replace: nsslapd-cachememsize nsslapd-cachememsize: backup_entry_cache_size
systemctl stop dirsrv.target
dn: cn=config,cn=ldbm database,cn=plugins,cn=config nsslapd-db-locks: backup_db_lock_number
ipactl start