Repository: dmwm/wmcore
Issue: TaskArchiver on T0 production agents seems to be stuck.

**Impact of the bug**
Inform which systems get affected by this bug. Which agent(s)? Which central service(s)?
T0-production agent with the TaskArchiver component. This issue affects the ability of T0 to properly archive workflows and following the logic of T0 machinery,  the ability to delete rucio dataset rules that were already replicated to the destination. This is a **potential** issue/bug due that the storage quota where T0 operates could be easily depleted in run3 if the TaskArchiver shows the same behaviour.
**Describe the bug**
TaskArchiver component seems to be stuck and within a loop trying to archive workflows. 
```
2022-01-10 22:43:23,943:140033573566208:INFO:TaskArchiverPoller:Found 218 candidate workflows for completing:
2022-01-10 22:43:23,953:140033573566208:INFO:TaskArchiverPoller:There are 0 requests in ['force-complete', 'aborted'] status in central couch.
2022-01-10 22:43:23,954:140033573566208:INFO:TaskArchiverPoller:Marking subscriptions as Done ...
2022-01-10 22:43:23,997:140033573566208:INFO:TaskArchiverPoller:status updated to completed PromptReco_Run345881_MinimumBias
...
2022-01-10 22:43:33,819:140033573566208:INFO:TaskArchiverPoller:status updated to completed PromptReco_Run346512_ZeroBias4
2022-01-10 22:43:33,821:140033573566208:INFO:BaseWorkerThread:TaskArchiverPoller took 9.990 secs to execute


2022-01-10 22:47:34,573:140033573566208:INFO:TaskArchiverPoller:Found 218 candidate workflows for completing:
2022-01-10 22:47:34,582:140033573566208:INFO:TaskArchiverPoller:There are 0 requests in ['force-complete', 'aborted'] status in central couch.
2022-01-10 22:47:34,582:140033573566208:INFO:TaskArchiverPoller:Marking subscriptions as Done ...
2022-01-10 22:47:34,625:140033573566208:INFO:TaskArchiverPoller:status updated to completed PromptReco_Run345881_MinimumBias
...
2022-01-10 22:47:45,207:140033573566208:INFO:TaskArchiverPoller:status updated to completed PromptReco_Run346512_ZeroBias4
2022-01-10 22:47:45,212:140033573566208:INFO:BaseWorkerThread:TaskArchiverPoller took 10.806 secs to execute
```
This issue is affecting only the production agent running on vocms0314. Restarting the component each 30 min seems to help with the other workflows but in the end the component is stuck . Also it would be nice to understand what's happening with this component that seems to be stuck in this loop.
One peculiarity of this issue is that the Componentlog shows the following error for the following workflows `Express_Run345846_StreamExpress` and `PromptReco_Run345922_MinimumBias`
```
2022-01-10 22:04:19,384:140033565173504:DEBUG:CleanCouchPoller:{'status': 'warning', 'message': 'no PromptReco_Run345922_MinimumBias exist'} docs deleted from JobDump
2022-01-10 22:04:19,393:140033565173504:DEBUG:CleanCouchPoller:{'status': 'warning', 'message': 'no PromptReco_Run345922_MinimumBias exist'} docs deleted from FWJRDump
2022-01-10 22:04:19,400:140033565173504:ERROR:CleanCouchPoller:Error creating workload and/or cleaning up couch: '_rev'
Traceback (most recent call last):
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/TaskArchiver/CleanCouchPoller.py", line 177, in algorithm
    self.cleanCouchDBAndChangeToArchiveStatus()
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/TaskArchiver/CleanCouchPoller.py", line 244, in cleanCouc
hDBAndChangeToArchiveStatus
    numUpdated = self.archiveWorkflows(deletableWorkflows, "normal-archived")
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/TaskArchiver/CleanCouchPoller.py", line 196, in archiveWo
rkflows
    if self.cleanAllLocalCouchDB(workflowName):
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/TaskArchiver/CleanCouchPoller.py", line 330, in cleanAllLocalCouchDB
    summaryReport = self.deleteWorkflowFromJobCouch(workflowName, "SummaryStats")
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMComponent/TaskArchiver/CleanCouchPoller.py", line 287, in deleteWorkflowFromJobCouch
    committed = couchDB.delete_doc(workflowName)
  File "/data/tier0/srv/wmagent/3.0.1/sw/slc7_amd64_gcc630/cms/t0/3.0.1/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 439, in delete_doc
    rev = doc["_rev"]
KeyError: '_rev'
2022-01-10 22:04:19,400:140033565173504:INFO:CleanCouchPoller:Cleaning up wmbs and disk
```
Checking T0-wmstasts, those workflows didn't run any kind of jobs.  Only one job for `PromptReco_Run345922_MinimumBias ` run but were killed by the agent with the following message:
`The job was killed by the WMAgent for using too much wallclock time (WMAgent) Job status was Running.`
**How to reproduce it**
Hard to say but with the proper instructions from WMCore if is it possible to publish a workflow with empty jobs probably it can be reproduced by a replay.

**Expected behavior**
If the bug is that TaskArchiver are not able to archive workflows that didn't run jobs it should be able to do it after a long period of time since the workflows were published.

**Additional context and error message**
Full component log is in `/afs/cern.ch/user/c/cmst0/public/ErrorLogs/WMCore/TaskArchiver_vm0314`


Changed files:
src/python/WMCore/Database/CMSCouch.py

diff --git a/src/python/WMCore/Database/CMSCouch.py b/src/python/WMCore/Database/CMSCouch.py
index d7caac78a..87b45f4f9 100644
--- a/src/python/WMCore/Database/CMSCouch.py
+++ b/src/python/WMCore/Database/CMSCouch.py
@@ -438,11 +438,22 @@ class Database(CouchDBRequests):
     def delete_doc(self, id, rev=None):
         """
         Immediately delete a document identified by id and rev.
+        If revision is not provided, we need to first fetch this
+        document to read the current revision number.
+
+        :param id: string with the document name
+        :param rev: string with the revision number
+        :return: an empty dictionary if it fails to fetch the document,
+            or a dictionary with the deletion outcome, e.g.:
+            {'ok': True, 'id': 'doc_name', 'rev': '3-f68156d'}
         """
         uri = '/%s/%s' % (self.name, urllib.parse.quote_plus(id))
         if not rev:
             # then we need to fetch the latest revision number
             doc = self.getDoc(id)
+            if "_rev" not in doc:
+                logging.warning("Failed to retrieve doc id: %s for deletion.", id)
+                return doc
             rev = doc["_rev"]
         uri += '?' + urllib.parse.urlencode({'rev': rev})
         return self.delete(uri)