Repository: dmwm/wmcore
Issue: MSMonitor breaks when finding an active transfer but the transfer's campaign is missing at ReqMgr  

**Impact of the bug**
MSMonitor

**Describe the bug**
During the validation of HG2212, We had some intermediate operational activities, some of which including old campaigns cleanup. This left us with some transfer records present in the system, but the relevant campaigns they were belonging to were missing at ReqMgr. Here [[1]](https://cmsweb-testbed.cern.ch/reqmgr2/data/campaignconfig/ALL_DOCS) and here [[2]](https://cmsweb-testbed.cern.ch/reqmgr2/data/transferinfo/ALL_DOCS) are the links  to the REST APIs where the lists of campaigns and transfer records are fetched from at the beginning of the service's  polling cycle.  The high level logic of `MSMonitor` is as follows:
* Fetch all requests in `staging`
* Fetch all transfer records and Campaigns from `ReqMgr` : [[3]](https://github.com/dmwm/WMCore/blob/f313eacd4020b0ec85c7baf4821c8bf32a1dee16/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py#L92)
* Filter all transfer docs leaving only the ones that are having an associated workflow in status `staging` and has not been recently updated (by a configurable amount of time)
* Iterate through all the transfers for a given workflow in a transfer record and check if they may be considered complete based on the `PartialCopy` parameter coming from the campaign configuration. [[4]](https://github.com/dmwm/WMCore/blob/f313eacd4020b0ec85c7baf4821c8bf32a1dee16/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py#L233-L238)

Because the whole `execution` of the above logic is enclosed in two separate `try/except` blocks, the service does not break, but the polling cycle is still interrupted with the following error [6] if a single workflow with a missing campaign configuration is found. And this is because the whole list of filtered transfer records is passed to the function taking care of the last bullet from above at this line: [[5]](https://github.com/dmwm/WMCore/blob/f313eacd4020b0ec85c7baf4821c8bf32a1dee16/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py#L112). Hence the rest of the workflows in staging are skipped. Here is an example of such transfer record [7] and the missing campaign is `RunIIAutumn18NanoAODv5`.  


**How to reproduce it**
Delete a campaign from Reqmgr which is still having active workflows in the system.

**Expected behavior**
The service to continue  working and not to break due to a misconfiguration or an error in a single workflow/transfer. The possible actions would be to:
* Either alarm for such a workflow/transfer 
* Or set a default value for the missing parameter 

But in both cases the current execution must continue and the the current polling cycle not to be interrupted.

**Additional context and error message**
[1]
https://cmsweb-testbed.cern.ch/reqmgr2/data/campaignconfig/ALL_DOCS 

[2]
https://cmsweb-testbed.cern.ch/reqmgr2/data/transferinfo/ALL_DOCS

[3]
https://github.com/dmwm/WMCore/blob/f313eacd4020b0ec85c7baf4821c8bf32a1dee16/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py#L92

[4]
https://github.com/dmwm/WMCore/blob/f313eacd4020b0ec85c7baf4821c8bf32a1dee16/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py#L233-L238

[5]
https://github.com/dmwm/WMCore/blob/f313eacd4020b0ec85c7baf4821c8bf32a1dee16/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py#L112

[6]
```
2022-12-07 08:33:38,952:ERROR:MSMonitor: Unknown exception processing the transfer records. Error: 'RunIIAutumn18NanoAODv5'
Traceback (most recent call last):
  File "/data/srv/HG2212c/sw/slc7_amd64_gcc630/cms/reqmgr2ms/1.1.5rc4/lib/python3.8/site-packages/WMCore/MicroService/MSMonitor/MSMonitor.py", line 112, in execute
    requestsToStage = self.getCompletedWorkflows(transferRecords, campaigns)
  File "/data/srv/HG2212c/sw/slc7_amd64_gcc630/cms/reqmgr2ms/1.1.5rc4/lib/python3.8/site-packages/WMCore/MicroService/MSMonitor/MSMonitor.py", line 233, in getCompletedWorkflows
    cdict = campaigns[transfer['campaignName']]
KeyError: 'RunIIAutumn18NanoAODv5'
```

[7]
```
{
  "_id": "TRANSFER_amaltaro_TC_Nano_Agent212_Val_220902_230557_2309",
  "ConfigType": "TRANSFER",
  "workflowName": "amaltaro_TC_Nano_Agent212_Val_220902_230557_2309",
  "lastUpdate": 1662395871,
  "transfers": [
    {
      "dataset": "/WplusToJJZToLNuJJ_mjj100_pTj10_QCD_LO_TuneCP5_13TeV-madgraph-pythia8/RunIISummer20UL17MiniAODv2-106X_mc2017_realistic_v9-v2/MINIAODSIM",
      "dataType": "primary",
      "transferIDs": [
        "326fbf6cad094da6aba27e360a708f74"
      ],
      "campaignName": "RunIIAutumn18NanoAODv5",
      "completion": [
        0.0,
        61.765,
        68.627,
        68.627,
        68.627,
        68.627,
        69.608,
        69.608,
        72.549,
        75.49,
        75.49,
        76.471,
        80.392
      ]
    }
  ]
```


Changed files:
src/python/WMCore/MicroService/MSMonitor/MSMonitor.py

diff --git a/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py b/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py
index e3268ad95..c9818930a 100644
--- a/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py
+++ b/src/python/WMCore/MicroService/MSMonitor/MSMonitor.py
@@ -230,9 +230,18 @@ class MSMonitor(MSCore):
             # check completion of all transfers
             statuses = []
             for transfer in record['transfers']:
-                cdict = campaigns[transfer['campaignName']]
+                try:
+                    cdict = campaigns[transfer['campaignName']]
+                    partialCopy = cdict['PartialCopy']
+                except Exception as ex:
+                    # Just log the error and stop checking any further transfers for the current request.
+                    status = 0
+                    statuses.append(status)
+                    msg = "Missing or broken campaign configuration at ReqMgr for request: %s. Error: %s"
+                    self.logger.exception(msg, reqName, str(ex))
+                    break
                 # compare against the last completion number, which is from the last cycle execution
-                if transfer['completion'][-1] >= cdict['PartialCopy'] * 100:
+                if transfer['completion'][-1] >= partialCopy * 100:
                     status = 1
                 else:
                     status = 0