IBM JRules or ODM module fails to access database with StaleConnectionException - United States  TECHNOTE (TROUBLESHOOTING)

PROBLEM(ABSTRACT)
 JRules and ODM modules such as Rule Execution Server (RES) and Rule Team Server (RTS) / Decision Center (DC) may fail to access their data source with the following type of exception present in the log:

com.ibm.websphere.ce.cm.StaleConnectionException: Io exception: Connection timed out 

SYMPTOM
The problem may occur when a JRules or ODM module accesses its data source after a period of database access inactivity. For example: 

 * The first user accessing a RTS / DC server that remained idle overnight experiences a failure upon logging and the server log shows an com.ibm.websphere.ce.cm.StaleConnectionException occurred at that time. 
 * After a RES server has been processing execution requests normally for some time, an attempt to deploy a ruleset update fails and the server log shows an com.ibm.websphere.ce.cm.StaleConnectionException occurred upon an attempt to invalidate a ruleset or to retrieve its definition.


CAUSE
The underlying network connection for a data source is no longer valid although the data source itself remains considered active from the WebSphere Application Server data source connection pool management perspective. An attempt by a JRules module to try and access its data base through one such data source connection provided by the Application Server may then fail. 
The network connection of a data source may become invalid for a number of reasons, one typical example is that a firewall between the Application Server and the Database has a policy to discard network connections that remain idle for some time.


ENVIRONMENT
The exception com.ibm.websphere.ce.cm.StaleConnectionException is specific to WebSphere Application Server environments. But a similar issue can be experienced with Weblogic, see: Rule Execution Server on WebLogic unable to load ruleset after period of inactivity [http://www-01.ibm.com/support/docview.wss?uid=swg21615801]



RESOLVING THE PROBLEM
Configure the WebSphere Application Server data source connection pool [http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/udat_conpoolset.html] so as to eliminate occurrences of StaleConnectionException : 

 * Set the Minimum connections to 0 to avoid situations where a connection is kept in the pool to meet the minimum requirement, even though it may have been idle for longer than Unused timeout 
 * Set the values for Unused timeout and Reap timeout so that the sum does not exceed any network level idle connection timeout


The document Troubleshooting stale connection problems [http://www-01.ibm.com/support/docview.wss?uid=swg21247168#SCETroubleshooting] provides further information. RELATED INFORMATION
#RES unable to load ruleset after period of inactivity [http://www-01.ibm.com/support/docview.wss?uid=swg21615801]
Troubleshooting stale connection problems [http://www-01.ibm.com/support/docview.wss?uid=swg21247168#SCETroubleshooting]





Cross reference information Segment Product Component Platform Version Edition Business Integration IBM Operational Decision Manager Platform Independent 8.5, 8.0.1, 8.0, 7.5 Enterprise

Question: Recurrent RES StaleConnectionException How to prevent rule execution failing because of recurrent StaleConnectionException on ODM 8.8 Rule Execution Server? 

Answer: Configure the WebSphere Application Server data source connection pool [http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.multiplatform.doc/info/ae/ae/udat_conpoolset.html] so as to eliminate occurrences of StaleConnectionException : 

 * Set the Minimum connections to 0 to avoid situations where a connection is kept in the pool to meet the minimum requirement, even though it may have been idle for longer than Unused timeout 
 * Set the values for Unused timeout and Reap timeout so that the sum does not exceed any network level idle connection timeout