Chapter 29. KIE Server clusters in a runtime environment In a runtime environment, KIE Server runs services that contain rules and processes that support business decisions. The primary benefit of clustering a KIE Server runtime environment is load balancing. If activity on one node of the cluster increases, that activity can be shared among the other nodes of the cluster to improve performance. To create a KIE Server clustered runtime environment, you download and extract Red Hat JBoss EAP 7.4 and KIE Server. Then, you configure Red Hat JBoss EAP 7.4 for a cluster, start the cluster, and install KIE Server on each cluster node. Optionally, you can then cluster the headless Process Automation Manager controller and Smart Router. Note Do not connect KIE Server to Business Central in high availability (HA) on premise environments. Business Central instances are not able to keep in sync with the status of each KIE Server. For example, if a KIE Server is up but not in sync, Business Central will not be able to deploy through that instance. 29.1. Downloading and extracting Red Hat JBoss EAP 7.4 and KIE Server Complete the steps in this section to download and install Red Hat JBoss EAP 7.4 and KIE Server for installation in a clustered environment. Procedure Download Red Hat JBoss EAP 7.4 on each node of the cluster: Navigate to the Software Downloads page in the Red Hat Customer Portal (login required) and select the product and version from the drop-down options: Product: Enterprise Application Platform Version: 7.4 Click Download to Red Hat JBoss Enterprise Application Platform 7.4.14 . ( jboss-eap-7.4.14.zip ). Extract the jboss-eap-7.4.14.zip file. The jboss-eap-7.4/jboss-eap-7.4 directory is referred to as EAP_HOME . Download and apply the latest Red Hat JBoss EAP patch, if available. Download KIE Server: Navigate to the Software Downloads page in the Red Hat Customer Portal and select the product and version from the drop-down options: PRODUCT: Process Automation Manager Version: 7.13.5 Download Red Hat Process Automation Manager 7.13.5 KIE Server for All Supported EE8 Containers ( rhpam-7.13.5-kie-server-ee8.zip ). Extract the rhpam-7.13.5-kie-server-ee8.zip archive to a temporary directory. In the following examples this directory is called TEMP_DIR . Copy the TEMP_DIR /rhpam-7.13.5-kie-server-ee8/rhpam-7.13.5-kie-server-ee8/kie-server.war directory to EAP_HOME /standalone/deployments/ . Warning Ensure the names of the Red Hat Process Automation Manager deployments you copy do not conflict with your existing deployments in the Red Hat JBoss EAP instance. Copy the contents of the TEMP_DIR /rhpam-7.13.5-kie-server-ee8/rhpam-7.13.5-kie-server-ee8/SecurityPolicy/ to EAP_HOME /bin . When prompted to overwrite files, click Replace . In the EAP_HOME /standalone/deployments/ directory, create an empty file named kie-server.war.dodeploy . This file ensures that KIE Server is automatically deployed when the server starts. Download and apply the latest Red Hat Process Automation Manager patch, if available. Navigate to the EAP_HOME /bin directory. Create a user with the kie-server role that you will use to log in to KIE Server. USD ./bin/jboss-cli.sh --commands="embed-server --std-out=echo,/subsystem=elytron/filesystem-realm=ApplicationRealm:add-identity(identity=<USERNAME>),/subsystem=elytron/filesystem-realm=ApplicationRealm:set-password(identity=<USERNAME>, clear={password='<PASSWORD>'}),/subsystem=elytron/filesystem-realm=ApplicationRealm:add-identity-attribute(identity=<USERNAME>, name=role, value=['kie-server'])" Make a note of your user names and passwords. 29.2. Configuring and running a Red Hat JBoss EAP 7.4 cluster for KIE Server Configure the Red Hat JBoss EAP cluster for KIE Server, and then start the cluster. Procedure Install the JDBC driver on all Red Hat JBoss EAP instances that are part of this cluster. For more information, see the "JDBC Drivers" section of the Red Hat JBoss EAP 7.4 Configuration Guide . Open the EAP_HOME /standalone/configuration/standalone-full.xml file in a text editor. Edit the data-stores property and the timer-service thread-pool-name above it: The datasource-jndi-name is the JNDI name of the database specified at the beginning of this procedure. You can enter any name for the value of the partition property. However, a node will only see timers from other nodes that have the same partition name. Grouping nodes into partitions by assigning partition names enables you to break a large cluster up into several smaller clusters. Doing this improves performance. For example, instead of having a cluster of 100 nodes, where all 100 nodes are trying to execute and refresh the same timers, you can create 20 clusters of 5 nodes by giving every group of 5 a different partition name. Replace the default-data-store attribute value with ejb_timer_ds . Set the value of refresh-interval in milliseconds to specify how often the EJB timer connects to the database to synchronize and load tasks to be processed. <timer-service thread-pool-name="default" default-data-store="ejb_timer_ds"> <data-stores> <database-data-store name="ejb_timer_ds" datasource-jndi-name="java:jboss/datasources/ejb_timer" database="postgresql" partition="ejb_timer_part" refresh-interval="30000"/> </data-stores> </timer-service> The following table lists the supported databases and the corresponding database attribute value: Table 29.1. Supported databases Database Attribute value Hyper SQL (for demonstration purposes only, not supported) hsql PostgreSQL postgresql Oracle oracle IBM DB2 db2 Microsoft SQL Server mssql MySQL and MariaDB mysql Add the KIE Server and EJB timer data sources to the standalone-full.xml file. In these examples, <DATABASE> is the name of the database, <SERVER_NAME> is the host name of the JNDI database, and <USER_NAME> and <USER_PWD> are the credentials for that database. Add the data source to allow KIE Server to connect to the database, for example: <xa-datasource jndi-name="java:/jboss/datasources/rhpam" pool-name="rhpam-RHPAM" use-java-context="true" enabled="true"> <xa-datasource-property name="DatabaseName"><DATABASE></xa-datasource-property> <xa-datasource-property name="PortNumber">5432</xa-datasource-property> <xa-datasource-property name="ServerName"><SERVER_NAME></xa-datasource-property> <driver>postgresql</driver> <security> <user-name><USER_NAME></user-name> <password><USER_PWD></password> </security> </xa-datasource> Add the data source to enable the EJB timer, for example: <xa-datasource jndi-name="java:jboss/datasources/ejb_timer" pool-name="ejb_timer" use-java-context="true" enabled="true"> <xa-datasource-property name="DatabaseName"><DATABASE></xa-datasource-property> <xa-datasource-property name="PortNumber">5432</xa-datasource-property> <xa-datasource-property name="ServerName"><SERVER_NAME></xa-datasource-property> <driver>postgresql</driver> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <security> <user-name><USER_NAME></user-name> <password><USER_PWD></password> </security> </xa-datasource> Warning You must use two different databases for KIE Server runtime data and EJB timer data. Add the following properties to the <system-properties> element and replace the following placeholders: <JNDI_NAME> is the JNDI name of your data source. For Red Hat Process Automation Manager, this is java:/jboss/datasources/rhpam . <DIALECT> is the hibernate dialect for your database. The following dialects are supported: DB2: org.hibernate.dialect.DB2Dialect MSSQL: org.hibernate.dialect.SQLServer2012Dialect MySQL: org.hibernate.dialect.MySQL5InnoDBDialect MariaDB: org.hibernate.dialect.MySQL5InnoDBDialect Oracle: org.hibernate.dialect.Oracle10gDialect PostgreSQL: org.hibernate.dialect.PostgreSQL82Dialect PostgreSQL plus: org.hibernate.dialect.PostgresPlusDialect <system-properties> <property name="org.kie.server.persistence.ds" value="<JNDI_NAME>"/> <property name="org.kie.server.persistence.dialect" value="<DIALECT>"/> <property name="org.jbpm.ejb.timer.tx" value="true"/> </system-properties> Save the standalone-full.xml file. To start the cluster, navigate to EAP_HOME /bin and enter one of the following commands: On Linux or UNIX-based systems: USD ./standalone.sh -c standalone-full.xml On Windows: standalone.bat -c standalone-full.xml 29.3. Clustering KIE Servers with the headless Process Automation Manager controller The Process Automation Manager controller is integrated with Business Central. However, if you do not install Business Central, you can install the headless Process Automation Manager controller and use the REST API or the KIE Server Java Client API to interact with it. Prerequisites A backed-up Red Hat JBoss EAP installation version 7.4 or later is available. The base directory of the Red Hat JBoss EAP installation is referred to as EAP_HOME . Sufficient user permissions to complete the installation are granted. An NFS server with a shared folder is available as described in Section 28.3, "Configuring an NFS version 4 server" . Procedure Navigate to the Software Downloads page in the Red Hat Customer Portal (login required), and select the product and version from the drop-down options: PRODUCT: Process Automation Manager Version: 7.13.5 Download Red Hat Process Automation Manager 7.13.5 Add Ons (the rhpam-7.13.5-add-ons.zip file). Extract the rhpam-7.13.5-add-ons.zip file. The rhpam-7.13.5-controller-ee7.zip file is in the extracted directory. Extract the rhpam-7.13.5-controller-ee7.zip archive to a temporary directory. In the following examples this directory is called TEMP_DIR . Copy the TEMP_DIR /rhpam-7.13.5-controller-ee7/controller.war directory to EAP_HOME /standalone/deployments/ . Warning Ensure that the names of the headless Process Automation Manager controller deployments you copy do not conflict with your existing deployments in the Red Hat JBoss EAP instance. Copy the contents of the TEMP_DIR /rhpam-7.13.5-controller-ee7/SecurityPolicy/ directory to EAP_HOME /bin . When prompted to overwrite files, click Yes . In the EAP_HOME /standalone/deployments/ directory, create an empty file named controller.war.dodeploy . This file ensures that the headless Process Automation Manager controller is automatically deployed when the server starts. Open the EAP_HOME /standalone/configuration/standalone.xml file in a text editor. Add the following properties to the <system-properties> element and replace <NFS_STORAGE> with the absolute path to the NFS storage where the template configuration is stored: Template files contain default configurations for specific deployment scenarios. If the value of the org.kie.server.controller.templatefile.watcher.enabled property is set to true, a separate thread is started to watch for modifications of the template file. The default interval for these checks is 30000 milliseconds and can be further controlled by the org.kie.server.controller.templatefile.watcher.interval system property. If the value of this property is set to false, changes to the template file are detected only when the server restarts. To start the headless Process Automation Manager controller, navigate to EAP_HOME /bin and enter the following command: On Linux or UNIX-based systems: USD ./standalone.sh On Windows: standalone.bat 29.4. Configuring failover for jobs in the RUNNING state on failed KIE Server nodes By default, when multiple KIE Server nodes are running in cluster, if one KIE Server node fails, other available KIE Server nodes pick up any jobs in the QUEUED or RETRYING state from the failed KIE Server node. However, these nodes do not pick up jobs that are in the RUNNING state. You can configure your cluster to include jobs in the RUNNING state, to be requeued to another KIE Server node. If there is a timer start node in a process on the failed KIE Server node, the process is started on only one of the nodes. Prerequisites You have configured a Red Hat JBoss EAP cluster for KIE Server. Procedure Prepare the environment: Verify that the following extensions are present in the standalone.xml file and add them if necessary: <extension module="org.jboss.as.clustering.infinispan"/> <extension module="org.jboss.as.clustering.jgroups"/> In the standalone.xml file, locate the <subsystem xmlns="urn:jboss:domain:infinispan:9.0"> subsystem and create a cached container entry named jbpm that includes a cache named nodes and a cache named jobs as shown in the following example: <cache-container name="jbpm"> <transport lock-timeout="60000"/> <replicated-cache name="nodes"> <transaction mode="BATCH"/> </replicated-cache> <replicated-cache name="jobs"> <transaction mode="BATCH"/> </replicated-cache> </cache-container> Add the jgroups subsystem for the replicated and distributed caches: <subsystem xmlns="urn:jboss:domain:jgroups:7.0"> <channels default="ee"> <channel name="ee" stack="udp" cluster="ejb"/> </channels> <stacks> <stack name="udp"> <transport type="UDP" socket-binding="jgroups-udp"/> <protocol type="PING"/> <protocol type="MERGE3"/> <socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="UFC"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> <stack name="tcp"> <transport type="TCP" socket-binding="jgroups-tcp"/> <socket-protocol type="MPING" socket-binding="jgroups-mping"/> <protocol type="MERGE3"/> <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> </stacks> </subsystem> Configure the following interface: <interface name="private"> <inet-address value="USD{jboss.bind.address.private:127.0.0.1}"/> </interface> Configure the following socket: <socket-binding name="jgroups-mping" interface="private" multicast-address="USD{jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/> <socket-binding name="jgroups-tcp" interface="private" port="7600"/> <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/> <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="USD{jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/> <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/> Add the Infinispan extension to KIE Server: Open the kie-server.war/WEB-INF/jboss-deployment-structure.xml file in a text editor and add the following dependencies to the file: <module name="org.infinispan" services="export"/> <module name="org.jgroups" /> Download the JAR file. Navigate to the Software Downloads page in the Red Hat Customer Portal (login required), and select the product and version from the drop-down options: Product: Process Automation Manager Version: 7.13.5 Download Red Hat Process Automation Manager 7.13.5 Maven Repository ( rhpam-7.13.5-maven-repository.zip ) and extract the ZIP file. Navigate to the directory containing the extracted files and copy the maven-repository/org/kie/server/kie-server-services-jbpm-cluster/7.67.0.Final-redhat-00024/kie-server-services-jbpm-cluster-7.67.0.Final-redhat-00024.jar file to the kie-server.war/WEB-INF/lib directory. The cluster is now configured to use failover for jobs in the RUNNING state.

Commands:
./bin/jboss-cli.sh --commands="embed-server --std-out=echo,/subsystem=elytron/filesystem-realm=ApplicationRealm:add-identity(identity=<USERNAME>),/subsystem=elytron/filesystem-realm=ApplicationRealm:set-password(identity=<USERNAME>, clear={password='<PASSWORD>'}),/subsystem=elytron/filesystem-realm=ApplicationRealm:add-identity-attribute(identity=<USERNAME>, name=role, value=['kie-server'])"
<timer-service thread-pool-name="default" default-data-store="ejb_timer_ds"> <data-stores> <database-data-store name="ejb_timer_ds" datasource-jndi-name="java:jboss/datasources/ejb_timer" database="postgresql" partition="ejb_timer_part" refresh-interval="30000"/> </data-stores> </timer-service>
<xa-datasource jndi-name="java:/jboss/datasources/rhpam" pool-name="rhpam-RHPAM" use-java-context="true" enabled="true"> <xa-datasource-property name="DatabaseName"><DATABASE></xa-datasource-property> <xa-datasource-property name="PortNumber">5432</xa-datasource-property> <xa-datasource-property name="ServerName"><SERVER_NAME></xa-datasource-property> <driver>postgresql</driver> <security> <user-name><USER_NAME></user-name> <password><USER_PWD></password> </security> </xa-datasource>
<xa-datasource jndi-name="java:jboss/datasources/ejb_timer" pool-name="ejb_timer" use-java-context="true" enabled="true"> <xa-datasource-property name="DatabaseName"><DATABASE></xa-datasource-property> <xa-datasource-property name="PortNumber">5432</xa-datasource-property> <xa-datasource-property name="ServerName"><SERVER_NAME></xa-datasource-property> <driver>postgresql</driver> <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation> <security> <user-name><USER_NAME></user-name> <password><USER_PWD></password> </security> </xa-datasource>
<system-properties> <property name="org.kie.server.persistence.ds" value="<JNDI_NAME>"/> <property name="org.kie.server.persistence.dialect" value="<DIALECT>"/> <property name="org.jbpm.ejb.timer.tx" value="true"/> </system-properties>
./standalone.sh -c standalone-full.xml
standalone.bat -c standalone-full.xml
<system-properties> <property name="org.kie.server.controller.templatefile.watcher.enabled" value="true"/> <property name="org.kie.server.controller.templatefile" value="<NFS_STORAGE>"/> </system-properties>
./standalone.sh
standalone.bat
<extension module="org.jboss.as.clustering.infinispan"/> <extension module="org.jboss.as.clustering.jgroups"/>
<cache-container name="jbpm"> <transport lock-timeout="60000"/> <replicated-cache name="nodes"> <transaction mode="BATCH"/> </replicated-cache> <replicated-cache name="jobs"> <transaction mode="BATCH"/> </replicated-cache> </cache-container>
<subsystem xmlns="urn:jboss:domain:jgroups:7.0"> <channels default="ee"> <channel name="ee" stack="udp" cluster="ejb"/> </channels> <stacks> <stack name="udp"> <transport type="UDP" socket-binding="jgroups-udp"/> <protocol type="PING"/> <protocol type="MERGE3"/> <socket-protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="UFC"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> <stack name="tcp"> <transport type="TCP" socket-binding="jgroups-tcp"/> <socket-protocol type="MPING" socket-binding="jgroups-mping"/> <protocol type="MERGE3"/> <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/> <protocol type="FD_ALL"/> <protocol type="VERIFY_SUSPECT"/> <protocol type="pbcast.NAKACK2"/> <protocol type="UNICAST3"/> <protocol type="pbcast.STABLE"/> <protocol type="pbcast.GMS"/> <protocol type="MFC"/> <protocol type="FRAG3"/> </stack> </stacks> </subsystem>
<interface name="private"> <inet-address value="USD{jboss.bind.address.private:127.0.0.1}"/> </interface>
<socket-binding name="jgroups-mping" interface="private" multicast-address="USD{jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/> <socket-binding name="jgroups-tcp" interface="private" port="7600"/> <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/> <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="USD{jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/> <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<module name="org.infinispan" services="export"/> <module name="org.jgroups" />