IBM JRules or ODM module fails to access database with StaleConnectionException - United States  TECHNOTE (TROUBLESHOOTING)

PROBLEM(ABSTRACT)
 JRules and ODM modules such as Rule Execution Server (RES) and Rule Team Server (RTS) / Decision Center (DC) may fail to access their data source with the following type of exception present in the log:

com.ibm.websphere.ce.cm.StaleConnectionException: Io exception: Connection timed out 

SYMPTOM
The problem may occur when a JRules or ODM module accesses its data source after a period of database access inactivity. For example: 

 * The first user accessing a RTS / DC server that remained idle overnight experiences a failure upon logging and the server log shows an com.ibm.websphere.ce.cm.StaleConnectionException occurred at that time. 
 * After a RES server has been processing execution requests normally for some time, an attempt to deploy a ruleset update fails and the server log shows an com.ibm.websphere.ce.cm.StaleConnectionException occurred upon an attempt to invalidate a ruleset or to retrieve its definition.


CAUSE
The underlying network connection for a data source is no longer valid although the data source itself remains considered active from the WebSphere Application Server data source connection pool management perspective. An attempt by a JRules module to try and access its data base through one such data source connection provided by the Application Server may then fail. 
The network connection of a data source may become invalid for a number of reasons, one typical example is that a firewall between the Application Server and the Database has a policy to discard network connections that remain idle for some time.


ENVIRONMENT
The exception com.ibm.websphere.ce.cm.StaleConnectionException is specific to WebSphere Application Server environments. But a similar issue can be experienced with Weblogic, see: Rule Execution Server on WebLogic unable to load ruleset after period of inactivity [http://www-01.ibm.com/support/docview.wss?uid=swg21615801]



RESOLVING THE PROBLEM
Configure the WebSphere Application Server data source connection pool [http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/udat_conpoolset.html] so as to eliminate occurrences of StaleConnectionException : 

 * Set the Minimum connections to 0 to avoid situations where a connection is kept in the pool to meet the minimum requirement, even though it may have been idle for longer than Unused timeout 
 * Set the values for Unused timeout and Reap timeout so that the sum does not exceed any network level idle connection timeout


The document Troubleshooting stale connection problems [http://www-01.ibm.com/support/docview.wss?uid=swg21247168#SCETroubleshooting] provides further information. RELATED INFORMATION
#RES unable to load ruleset after period of inactivity [http://www-01.ibm.com/support/docview.wss?uid=swg21615801]
Troubleshooting stale connection problems [http://www-01.ibm.com/support/docview.wss?uid=swg21247168#SCETroubleshooting]





Cross reference information Segment Product Component Platform Version Edition Business Integration IBM Operational Decision Manager Platform Independent 8.5, 8.0.1, 8.0, 7.5 Enterprise

Question: Recurrent RES StaleConnectionException How to prevent rule execution failing because of recurrent StaleConnectionException on ODM 8.8 Rule Execution Server? 

Answer: Configure the WebSphere Application Server data source connection pool so as to eliminate occurrences of StaleConnectionException :

    Set the Minimum connections to 0 to avoid situations where a connection is kept in the pool to meet the minimum requirement, even though it may have been idle for longer than Unused timeout
    Set the values for Unused timeout and Reap timeout so that the sum does not exceed any network level idle connection timeout