Repository: dmwm/wmcore
Issue: Global workqueue throwing an exception when the database is compacting

**Impact of the bug**
Global workqueue

**Describe the bug**
When central workqueue/workqueue_inbox database is compacting, some tasks might be skipped until the compaction is over.
With that, it can cause a `TypeError` exception with unexpected data type returned by a given function, as can be seen in this log [1]

**How to reproduce it**
Run global workqueue while couchdb is compacting the workqueue database.

**Expected behavior**
Properly deal with this scenario, i.e., `workClosed` should be 0 instead of None.

**Additional context and error message**
[1]
```
2022-06-17 20:26:48,997:INFO:WorkQueueReqMgrInterface:Closing open requests
2022-06-17 20:26:49,023:INFO:WorkQueueBackend:CouchDB compacting - try again later.
2022-06-17 20:26:49,023:WARNING:WorkQueue:Backend busy or down: Can not close work at this time
2022-06-17 20:26:49,023:ERROR:WorkQueueReqMgrInterface:Generic error while closing open requests. Details: object of type 'NoneType' has no len()
Traceback (most recent call last):
  File "/data/srv/HG2206e/sw/slc7_amd64_gcc630/cms/workqueue/2.0.4.pre4/lib/python3.8/site-packages/WMCore/WorkQueue/WorkQueueReqMgrInterface.py", line 63, in __call__
    msg += "Work closed: %d, " % len(workClosed)
TypeError: object of type 'NoneType' has no len()
2022-06-17 20:26:49,024:INFO:WorkQueueReqMgrInterface:Adding new elements to open requests
```

Changed files:
test/python/WMCore_t/WorkQueue_t/WorkQueue_t.py, src/python/WMCore/WorkQueue/WorkQueue.py

diff --git a/src/python/WMCore/WorkQueue/WorkQueue.py b/src/python/WMCore/WorkQueue/WorkQueue.py
index 138bd350c..e4ddfe845 100644
--- a/src/python/WMCore/WorkQueue/WorkQueue.py
+++ b/src/python/WMCore/WorkQueue/WorkQueue.py
@@ -869,15 +869,16 @@ class WorkQueue(WorkQueueBase):
 
         :return: list of workqueue_inbox elements that have been closed
         """
+        workflowsToClose = []
         if self.params['LocalQueueFlag']:
-            return  # GlobalQueue-only service
+            # this is a Global WorkQueue only functionality
+            return workflowsToClose
         if not self.backend.isAvailable():
             self.logger.warning('Backend busy or down: Can not close work at this time')
-            return
+            return workflowsToClose
 
         workflowsToCheck = self.backend.getInboxElements(OpenForNewData=True)
         self.logger.info("Retrieved a list of %d open workflows", len(workflowsToCheck))
-        workflowsToClose = []
         currentTime = time.time()
         for element in workflowsToCheck:
             # fetch attributes from the inbox workqueue element
diff --git a/test/python/WMCore_t/WorkQueue_t/WorkQueue_t.py b/test/python/WMCore_t/WorkQueue_t/WorkQueue_t.py
index 0032cf2fa..eb05dd33c 100644
--- a/test/python/WMCore_t/WorkQueue_t/WorkQueue_t.py
+++ b/test/python/WMCore_t/WorkQueue_t/WorkQueue_t.py
@@ -86,9 +86,8 @@ def syncQueues(queue, skipWMBS=False):
     time.sleep(1)
     work = queue.processInboundWork()
     queue.performQueueCleanupActions(skipWMBS=skipWMBS)
-    # queue.backend.forceQueueSync() ## FIXME TODO: AMR, maybe revert it
     # after replication need to wait a while to update result
-    time.sleep(1)
+    time.sleep(3)
     return work
 
 
@@ -814,6 +813,8 @@ class WorkQueueTest(WorkQueueTestCase):
         # self.globalQueue.updateLocationInfo()
         self.assertEqual(self.localQueue.pullWork({'T2_XX_SiteA': 1000}), totalSpec)
         syncQueues(self.localQueue)
+        # give a few extra seconds for work to move between local/global queues
+        time.sleep(2)
         self.assertEqual(len(self.localQueue.status(status='Available')), totalSpec)
         self.assertEqual(len(self.globalQueue.status(status='Acquired')), totalSpec)
         self.localQueue.updateLocationInfo()