Repository: dmwm/wmcore
Issue: DBS3Uploader hangs due to ORA-25401

**Impact of the bug**
WMAgent/DBSUpload

**Describe the bug**
While looking on some, at a first glance, trivial component hiccups of `WMAgent/DBS3Uppload` component  I've found it completely hung with the following error: [1]. Reading several different sources I am left with the impression that this `ORA-25401` error is  related to some Database internals, especially to broken sessions due to single servers failures, not the whole databse itself. This is also exactly what `sqlAlchemy` tells about this type of error: 

https://docs.sqlalchemy.org/en/14/errors.html#error-4xp6

I am not sure if we intentionally/(by design) we do not cover this or any of the Oracle errors inside our code (component code, DAO modules etc...) but in this case it seems to me we can simply retry the write operation. A simple restart seems to be fixing the issue, but we may consider operation retries with this particular error. In case we do not plan to take actions  on that  we may just close the issue. 

**How to reproduce it**
Not clear. Those interruptions seem to be Oracle infrastructure dependent. 

**Expected behavior**
To not completely hang the component, but rather retry the write operation.

**Additional context and error message**
[1]
```
2023-01-18 09:15:51,742:140468398376704:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.DBS3Buffer.DBSUploadPoller.DBSUploadPoller object at 0x7fc16637b8e0> <@========== WMException Start ==========@>
Exception Class: DBSUploadException
Message: Unhandled exception while loading uploadable files for DatasetPath.
(cx_Oracle.DatabaseError) ORA-25401: can not continue fetches
(Background on this error at: https://sqlalche.me/e/14/4xp6)
        ClassName : None
        ModuleName : WMComponent.DBS3Buffer.DBSUploadPoller
        MethodName : loadFiles
        ClassInstance : None
        FileName : /data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py
        LineNumber : 436
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 428, in loadFiles
    loadedFiles = self.dbsUtil.findUploadableFilesByDAS(datasetpath=datasetpath)

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBSBufferUtil.py", line 148, in findUploadableFilesByDAS
    results = findFiles.execute(datasetpath=datasetpath, transaction=False)

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMComponent/DBS3Buffer/MySQL/LoadDBSFilesByDAS.py", line 234, in execute
    result   = self.dbi.processData(self.fileInfoSQL,

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMCore/Database/DBCore.py", line 172, in processData
    r = self.executebinds(s, b, connection=connection,

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMCore/Database/DBCore.py", line 70, in executebinds
    result.add(resultProxy)

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMCore/Database/ResultSet.py", line 37, in add
    for r in resultproxy:

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/engine/result.py", line 382, in iterrows
    for row in self._fetchiter_impl():

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/engine/cursor.py", line 1796, in _fetchiter_impl
    row = fetchone(self, self.cursor)

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/engine/cursor.py", line 960, in fetchone
    self.handle_exception(result, dbapi_cursor, e)

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/engine/cursor.py", line 941, in handle_exception
    result.connection._handle_dbapi_exception(

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception

  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/external/py3-sqlalchemy/1.4.40/lib/python3.8/site-packages/sqlalchemy/engine/cursor.py", line 955, in fetchone
    row = dbapi_cursor.fetchone()

<@---------- WMException End ----------@>
Backtrace:
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 209, in __call__
    raise ex
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 183, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMCore/Database/DBExceptionHandler.py", line 40, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/Utils/Timers.py", line 26, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 309, in algorithm
    self.loadFiles()
  File "/data/srv/wmagent/v2.1.4/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.4/lib/python3.8/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 436, in loadFiles
    raise DBSUploadException(msg) from None


```

Changed files:
src/python/WMCore/Database/DBExceptionHandler.py

diff --git a/src/python/WMCore/Database/DBExceptionHandler.py b/src/python/WMCore/Database/DBExceptionHandler.py
index e5ada4aaa..3bdd044fc 100644
--- a/src/python/WMCore/Database/DBExceptionHandler.py
+++ b/src/python/WMCore/Database/DBExceptionHandler.py
@@ -14,10 +14,11 @@ import threading
 # ORA-01033: ORACLE initialization or shutdown in progress
 # (cx_Oracle.InterfaceError) not connected  # same as ORA-03114, in the new SQLAlchemy
 # (cx_Oracle.DatabaseError) ORA-25408: can not safely replay call
+# (cx_Oracle.DatabaseError) ORA-25401: can not continue fetches
 # and those two MySQL exceptions
 DB_CONNECTION_ERROR_STR = ["ORA-03113", "ORA-03114", "ORA-03135", "ORA-12545", "ORA-00060", "ORA-01033",
                            "MySQL server has gone away", "Lock wait timeout exceeded",
-                           "(cx_Oracle.InterfaceError) not connected", "ORA-25408"]
+                           "(cx_Oracle.InterfaceError) not connected", "ORA-25408", "ORA-25401"]
 
 
 def db_exception_handler(f):