Repository: dmwm/wmcore
Issue: MSOutput - unhandled exception in _getDatasetSize()

**Impact of the bug**
MSOutput

**Describe the bug**
While trying to determine dataset size if the service for any reason fails to do so (e.g. fails to get DID info from Rucios etc.) we do not handle the exception [1] but just let it reach the upper level pipeline exception handler, while proper actions  should be taken at place. In result we pollute the logs with exceptions, which in many cases could be just predictable/expected behavior.   


[1]
https://github.com/dmwm/WMCore/blob/e45a4425c5437d194d0d02814b7011c752c33548/src/python/WMCore/MicroService/MSOutput/MSOutput.py#L696

[2]
```
2021-03-23 01:14:48,521:ERROR:MSOutput: MSOutputProducer Pipeline Possibly broken read from ReqMgr2 API or other. Err: 'bytes'. Continue to the next document.
Traceback (most recent call last):
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 545, in msOutputProducer
    msPipeline.run(request)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 140, in run
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 140, in <lambda>
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 72, in __call__
    return self.run(obj)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/Utils/Pipeline.py", line 75, in run
    return self.func(obj, *self.args, **self.kwargs)
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 622, in docInfoUpdate
    bytesSize = self._getDatasetSize(dataItem['Dataset'])
  File "/data/srv/HG2103g/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.6.pre9/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 695, in _getDatasetSize
    return didInfo["bytes"]
KeyError: 'bytes'
```

**How to reproduce it**
Just look at the MSOutput  logs

**Expected behavior**
In case the situation is expected, we should handle the exception at the place where it happens (even if we need to create a new class of exceptions), and propagate only the unexpected ones upstream.
 
**Additional context and error message**
N/A


Changed files:
src/python/WMCore/MicroService/MSOutput/MSOutput.py

diff --git a/src/python/WMCore/MicroService/MSOutput/MSOutput.py b/src/python/WMCore/MicroService/MSOutput/MSOutput.py
index 1c0cab49c..26816bddd 100644
--- a/src/python/WMCore/MicroService/MSOutput/MSOutput.py
+++ b/src/python/WMCore/MicroService/MSOutput/MSOutput.py
@@ -613,6 +613,7 @@ class MSOutput(MSCore):
         """
         self.logger.info("Producing MongoDB record for workflow: %s", msOutDoc["RequestName"])
         updatedOutputMap = []
+        notFoundDIDs = []
         for dataItem in msOutDoc['OutputMap']:
             if msOutDoc['RequestType'] == "Resubmission":
                 # make sure not to subscribe the same datasets multiple times, even
@@ -621,7 +622,17 @@ class MSOutput(MSCore):
                 updatedOutputMap.append(dataItem)
                 continue
             ### Fetch the dataset size, even if it does not go to Disk (it might go to Tape)
-            bytesSize = self._getDatasetSize(dataItem['Dataset'])
+            try:
+                bytesSize = self._getDatasetSize(dataItem['Dataset'])
+            except KeyError as exc:
+                # then this container is unknown to Rucio, bypass and make an alert
+                # Error is already reported in the Rucio module, do not spam here!
+                dataItem['DatasetSize'] = 0
+                dataItem['Copies'] = 0
+                updatedOutputMap.append(dataItem)
+                notFoundDIDs.append(dataItem['Dataset'])
+                continue
+
             dataItem['DatasetSize'] = bytesSize
 
             if not self.canDatasetGoToDisk(dataItem, msOutDoc['IsRelVal']):
@@ -661,14 +672,6 @@ class MSOutput(MSCore):
                     updatedOutputMap.append(dataItem)
                     continue
             else:
-                # FIXME:
-                #    Here we need to use the already created campaignMap for
-                #    building the destinationOutputMap for nonRelVal workflows.
-                #    For the time being it is a fallback to all T1_* and all T2_*.
-                #    Once we migrate to Rucio we should change those defaults to
-                #    whatever is the format in Rucio (eg. referring a subscription
-                #    rule like: "store it at a good site" or "Store in the USA" etc.)
-
                 # NOTE: This default rseExpression should target all T1_*_Disk and T2_*
                 # sites, where the first part is a Union of those Tiers and the second
                 # part is a general constraint for those to be real entries (not `Test`
@@ -676,6 +679,19 @@ class MSOutput(MSCore):
                 dataItem['DiskDestination'] = '(tier=2|tier=1)&cms_type=real&rse_type=DISK'
             updatedOutputMap.append(dataItem)
 
+        # if there were containers not found in Rucio, create an email alert
+        if notFoundDIDs:
+            # send alert via AlertManager API
+            alertName = "ms-output: output containers not found for workflow: {}".format(msOutDoc["RequestName"])
+            alertSeverity = "high"
+            alertSummary = "[MSOutput] Workflow '{}' has output datasets unknown to Rucio".format(msOutDoc["RequestName"])
+            alertDescription = "Dataset(s): {} cannot be found in Rucio. ".format(notFoundDIDs)
+            alertDescription += "Thus, we are skipping these datasets from the final output "
+            alertDescription += "data placement, such that this workflow can get archived."
+            self.logger.warning(alertDescription)
+            if self.msConfig["sendNotification"]:
+                self.alertManagerAPI.sendAlert(alertName, alertSeverity, alertSummary, alertDescription, self.alertServiceName)
+
         try:
             msOutDoc.updateDoc({"OutputMap": updatedOutputMap}, throw=True)
         except Exception as ex: