Project: springxd
Issue XD-1678: XD does not reconnect to jobstore if connection is  lost, leaving Jobs in inconsistent state.
XD Deployment Description	XD Cluster (1 Container) Environment	EC2 Type Of Test	Shell Command Line Test Failed On	hdfs (only test that was run) Build Used	Built May 7, 10:29 PST [Overall issue] XD Admin and container lost connectivity to the the jobstore (MySql on RDS) and did not reconnect. Exception Displayed in log. Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 54,321,927 milliseconds ago. The last packet sent successfully to the server was 54,321,928 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem. sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) java.lang.reflect.Constructor.newInstance(Constructor.java:526) com.mysql.jdbc.Util.handleNewInstance(Util.java:411) com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117) com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3871) com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2484) com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2664) com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2788) com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2738) com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1617) org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:452) org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:402)	... 57 more [Side Effects] [Unable to create new Job with same name] User executed a job list and then removed the jobs in the list. When the user tried to create new jobs using the same names the application reported: "Command failed org.springframework.xd.rest.client.impl.SpringXDException: StatementCallback; SQL [SELECT JOB_NAME FROM JOB_REGISTRY_NAMES]; No operations allowed after connection closed.; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed." The only way to resolve it completely was to: 1) shutdown the admin and the containers. 2) Clear the jobs from the Batch job tables by hand 3) restart XD admin and containers.
Story points: 5