Repository: dmwm/wmcore
Issue: Large backlog of workflows in announced state (microservices restarting)

**Impact of the bug**
MSOutput and MSRuleCleaner

**Describe the bug**
Given that it's the second time that it happens within ~10 days, I am creating this issue to have this and the actions taken tracked here.

We have been piling up workflows in the `announced` state and it has reached 11.7k workflows at this very moment. Looking at the "Kube eagle k8s metrics" dashboard [1], we can see that it directly impacts the resources required by MSOutput and MSRuleCleaner services, which eventually run out of hardware resources and start either crashing or getting restarted by one of the k8s mechanisms put in place (k8s resource management, liveness probe, etc).

When I checked the kubernetes pods a few minutes ago, only the `ms-output` pod was having apparent restarts [2]. But if one inspect their log, it's possible to see that the service is getting restarted multiple times every minute [3]. The same applies to MSRuleCleaner [4]

**How to reproduce it**
Exhaust the resources defined in the service yaml file, e.g.: https://github.com/dmwm/CMSKubernetes/blob/master/kubernetes/cmsweb/services/reqmgr2ms-output.yaml#L98-L104

**Expected behavior**
This ticket is meant to keep track of actions to recover both MSOutput and MSRuleCleaner to a stable state and to get over the backlog of `announced` workflows.

A possible long-term and permanent solution - perhaps to be addressed in another ticket - to this would be to either stream data from CouchDB, or implement a way to slice data to be retrieved from the database backend, such that memory footprint can have a more predictable and controlled pattern.

**Additional context and error message**
[1]
https://monit-grafana.cern.ch/d/M2U6T6uGk/kube-eagle-k8s-metrics?orgId=11&from=1668442898738&to=1668486098738&refresh=30m

[2]
```
amaltaro@lxplus8s14:~ $ kubepods dmwm
NAME                                       READY   STATUS      RESTARTS       AGE
ms-output-67c6755866-r6xtx                 2/2     Running     22 (42m ago)   6d15h
ms-output-mongo-686dc4b9f-vp8bj            1/1     Running     2 (27d ago)    27d
ms-rulecleaner-8684b896b9-8kxrd            2/2     Running     0              6d15h
```

[3]
```
amaltaro@vocms0750:/cephfs/product/dmwm-logs $ grep WATCHDOG ms-output-20221115-ms-output-67c6755866-r6xtx.log | tail -n5
[15/Nov/2022:05:01:31]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:02:17]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:02:55]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:03:27]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:03:58]  WATCHDOG: server exited with exit code signal 9... restarting
```

[4]
```
amaltaro@vocms0750:/cephfs/product/dmwm-logs $ grep WATCHDOG ms-rulecleaner-20221115-ms-rulecleaner-8684b896b9-8kxrd.log | tail -n5
[15/Nov/2022:05:03:00]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:03:22]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:03:43]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:04:09]  WATCHDOG: server exited with exit code signal 9... restarting
[15/Nov/2022:05:04:38]  WATCHDOG: server exited with exit code signal 9... restarting
```

Changed files:
src/python/WMCore/MicroService/MSRuleCleaner/MSRuleCleaner.py

diff --git a/src/python/WMCore/MicroService/MSRuleCleaner/MSRuleCleaner.py b/src/python/WMCore/MicroService/MSRuleCleaner/MSRuleCleaner.py
index 7c66941c9..7c292fa9f 100644
--- a/src/python/WMCore/MicroService/MSRuleCleaner/MSRuleCleaner.py
+++ b/src/python/WMCore/MicroService/MSRuleCleaner/MSRuleCleaner.py
@@ -593,15 +593,18 @@ class MSRuleCleaner(MSCore):
         if transferInfo is None:
             msg = f"Workflow {wflow['RequestName']} is still missing the output transfer document."
             self.logger.warning(msg)
+        elif transferInfo['TransferStatus'] == 'pending':
+            msg = f"Workflow {wflow['RequestName']} is still pending the output data placement."
+            self.logger.warning(msg)
         elif transferInfo['TransferStatus'] == 'done':
             # Set Transfer status - information fetched from MSOutput only
             wflow['TransferDone'] = True
-        elif transferInfo['OutputMap']:
+
             # Set Tape rules status - information fetched from Rucio (tape rule ids from MSOutput)
-            tapeRulesStatusList = []
             # For setting 'TransferTape' = True we require either no tape rules for the
             # workflow have been created or all existing tape rules to be in status 'OK',
             # so every empty TapeRuleID we consider as completed.
+            tapeRulesStatusList = []
             for mapRecord in transferInfo['OutputMap']:
                 if not mapRecord['TapeRuleID']:
                     continue