Repository: dmwm/wmcore
Issue: RetryManager/ArchiveDataPoller crashing on cx_Oracle.DatabaseError: ORA-25408

I see a couple of testbed agents with components down, RetryManager and ArchiveDataPoller, and the reason comes from an oracle exception (that I have never seen before).

It seems safe to add this error to the soft database errors and retry again in the next cycle.

Traceback is:
```
2019-06-27 15:31:16,650:140228047771392:INFO:BaseWorkerThread:RetryManagerPoller took 0.014 secs to execute
2019-06-27 15:35:22,161:140228047771392:WARNING:HeartbeatAPI:Heartbeat update failed! Wait for the next time...:
(cx_Oracle.DatabaseError) ORA-25408: can not safely replay call

[SQL: UPDATE wm_workers
                      SET last_updated = :last_updated
               , state = :state WHERE name = :worker_name]
[parameters: {'state': 'Running', 'worker_name': 'RetryManagerPoller', 'last_updated': 1561642516}]
(Background on this error at: http://sqlalche.me/e/4xp6)
2019-06-27 15:35:22,527:140228047771392:INFO:RetryManagerPoller:Found 0 jobs in createcooloff
2019-06-27 15:35:22,592:140228047771392:INFO:RetryManagerPoller:Found 0 jobs in submitcooloff
2019-06-27 15:35:22,594:140228047771392:INFO:RetryManagerPoller:Found 0 jobs in jobcooloff
2019-06-27 15:35:22,791:140228047771392:INFO:RetryManagerPoller:Found 0 jobs in jobpaused
2019-06-27 15:35:22,793:140228047771392:INFO:RetryManagerPoller:Found 0 jobs in createpaused
2019-06-27 15:35:22,795:140228047771392:INFO:RetryManagerPoller:Found 0 jobs in submitpaused
2019-06-27 15:35:23,093:140228047771392:ERROR:RetryManagerPoller:Caught exception in RetryManager
(cx_Oracle.DatabaseError) ORA-25408: can not safely replay call

(Background on this error at: http://sqlalche.me/e/4xp6)Traceback (most recent call last):
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.2.2.patch3/lib/python2.7/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 160, in algorithm
    myThread.transaction.commit()
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.2.2.patch3/lib/python2.7/site-packages/WMCore/Database/Transaction.py", line 55, in commit
    self.transaction.commit()
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1716, in commit
    self._do_commit()
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1747, in _do_commit
    self.connection._commit_impl()
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 763, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 761, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/external/py2-sqlalchemy/1.3.3/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 505, in do_commit
    dbapi_connection.commit()
DatabaseError: (cx_Oracle.DatabaseError) ORA-25408: can not safely replay call
...
(Background on this error at: http://sqlalche.me/e/4xp6)


  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.2.2.patch3/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.2.2.patch3/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 38, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.2.2.patch3/lib/python2.7/site-packages/Utils/Timers.py", line 24, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.2.2.patch3/sw/slc7_amd64_gcc630/cms/wmagent/1.2.2.patch3/lib/python2.7/site-packages/WMComponent/RetryManager/RetryManagerPoller.py", line 177, in algorithm
    raise Exception(msg)

2019-06-27 15:35:23,095:140228047771392:INFO:Harness:>>>Terminating worker threads
2019-06-27 15:35:23,096:140228047771392:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.RetryManager.RetryManagerPoller.RetryManagerPoller object at 0x7f89710a1cd0> Caught exception in RetryManager
```

Changed files:
src/python/WMCore/Database/DBExceptionHandler.py

diff --git a/src/python/WMCore/Database/DBExceptionHandler.py b/src/python/WMCore/Database/DBExceptionHandler.py
index b719d349e..eb6f8cfde 100644
--- a/src/python/WMCore/Database/DBExceptionHandler.py
+++ b/src/python/WMCore/Database/DBExceptionHandler.py
@@ -12,10 +12,11 @@ import threading
 # ORA-00060: deadlock detected while waiting for resource
 # ORA-01033: ORACLE initialization or shutdown in progress
 # (cx_Oracle.InterfaceError) not connected  # same as ORA-03114, in the new SQLAlchemy
+# (cx_Oracle.DatabaseError) ORA-25408: can not safely replay call
 # and those two MySQL exceptions
 DB_CONNECTION_ERROR_STR = ["ORA-03113", "ORA-03114", "ORA-03135", "ORA-12545", "ORA-00060", "ORA-01033",
                            "MySQL server has gone away", "Lock wait timeout exceeded",
-                           "(cx_Oracle.InterfaceError) not connected"]
+                           "(cx_Oracle.InterfaceError) not connected", "ORA-25408"]
 
 
 def db_exception_handler(f):