Repository: dmwm/wmcore
Issue: MSTransferor exception when calculating the total size of the primary blocks

**Impact of the bug**
MSTransferor

**Describe the bug**
It looks like one of the blocks either fails to return its size from the DM system, or something else is happening and we try to sum up integer with None, thus raising an exception. Traceback will be provided below.

**How to reproduce it**
Maybe by running that same workflow in a private and contained environment.

**Expected behavior**
If block is not found in the DM system, then it should be removed from the `getPrimaryBlocks()` method response, such that MSTransferor doesn't even try to transfer it. Issue might be somewhere else though, so it needs further debugging.

**Additional context and error message**
From MSTransferor logs in vocms0742:
```
2020-12-18 13:03:24,844:INFO:MSTransferor: Handling data subscriptions for request: pdmvserv_task_TOP-RunIISummer19UL17MiniAODv2-00028__v1_T_201217_183214_1460
2020-12-18 13:03:24,844:INFO:MSTransferor:   final list of PSNs to be use: [u'T1_DE_KIT', u'T1_ES_PIC', u'T1_FR_CCIN2P3', u'T1_IT_CNAF', u'T1_RU_JINR', u'T1_UK_RAL', u'T1_US_FNAL', u'T2_BE_IIHE', u'T2_BE_UCL', u'T2_BR_SPRACE', u'T2_BR_UERJ', u'T2_CH_CERN', u'T2_CH_CERN_HLT', u'T2_CH_CSCS', u'T2_CN_Beijing', u'T2_DE_DESY', u'T2_DE_RWTH', u'T2_EE_Estonia', u'T2_ES_CIEMAT', u'T2_ES_IFCA', u'T2_FI_HIP', u'T2_FR_CCIN2P3', u'T2_FR_GRIF_IRFU', u'T2_FR_GRIF_LLR', u'T2_HU_Budapest', u'T2_IT_Bari', u'T2_IT_Legnaro', u'T2_IT_Rome', u'T2_KR_KISTI', u'T2_PT_NCG_Lisbon', u'T2_RU_IHEP', u'T2_RU_INR', u'T2_RU_JINR', u'T2_TR_METU', u'T2_TW_NCHC', u'T2_UA_KIPT', u'T2_UK_London_Brunel', u'T2_UK_London_IC', u'T2_UK_SGrid_Bristol', u'T2_UK_SGrid_RALPP', u'T2_US_Caltech', u'T2_US_Florida', u'T2_US_MIT', u'T2_US_Nebraska', u'T2_US_Purdue', u'T2_US_UCSD', u'T2_US_Vanderbilt', u'T2_US_Wisconsin']
2020-12-18 13:03:24,845:INFO:MSTransferor: List of out-of-space RSEs dropped for 'pdmvserv_task_TOP-RunIISummer19UL17MiniAODv2-00028__v1_T_201217_183214_1460' is: set([u'T2_BR_UERJ', u'T2_DE_DESY', u'T2_FR_GRIF_IRFU', u'T1_FR_CCIN2P3_Disk', u'T1_RU_JINR_Disk', u'T2_IT_Bari', u'T2_BR_SPRACE'])
2020-12-18 13:03:24,845:ERROR:MSTransferor: Unknown exception while making Transfer Request for pdmvserv_task_TOP-RunIISummer19UL17MiniAODv2-00028__v1_T_201217_183214_1460     Error: unsupported operand type(s) for +: 'int' and 'NoneType'
Traceback (most recent call last):
  File "/data/srv/beHG2012f/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.3.pre6/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 207, in execute
    success, transfers = self.makeTransferRequest(wflow)
  File "/data/srv/beHG2012f/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.3.pre6/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 613, in makeTransferRequest
    for blocks, dataSize, idx in self._decideDataDestination(wflow, dataIn, len(nodes)):
  File "/data/srv/beHG2012f/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.3.pre6/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSTransferor.py", line 801, in _decideDataDestination
    listBlockSets, listSetsSize = wflow.getChunkBlocks(numNodes)
  File "/data/srv/beHG2012f/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.4.3.pre6/lib/python2.7/site-packages/WMCore/MicroService/DataStructs/Workflow.py", line 348, in getChunkBlocks
    thisChunkSize = sum([blockInfo['blockSize'] for blockInfo in self.getPrimaryBlocks().values()])
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
2020-12-18 13:03:24,845:INFO:MSTransferor: Checking data location for request: pdmvserv_Run2018D_SingleMuon_UL2018_MiniAODv2_201218_123105_4141, TrustSitelists: False, request white/black list PNNs: set([u'T1_ES_PIC_Disk', u'T2_BR_UERJ', u'T2_TR_METU', u'T2_CH_CSCS', u'T2_UA_KIPT', u'T1_FR_CCIN2P3_Disk', u'T2_US_Purdue', u'T2_TW_NCHC', u'T2_FI_HIP', u'T2_UK_SGrid_RALPP', u'T2_FR_GRIF_LLR', u'T2_BE_UCL', u'T2_ES_IFCA', u'T2_BE_IIHE', u'T2_UK_SGrid_Bristol', u'T2_FR_IPHC', u'T2_US_Caltech', u'T2_DE_DESY', u'T2_IT_Legnaro', u'T2_IT_Rome', u'T1_DE_KIT_Disk', u'T2_UK_London_Brunel', u'T2_RU_JINR', u'T1_UK_RAL_Disk', u'T1_US_FNAL_Disk', u'T2_EE_Estonia', u'T2_CN_Beijing', u'T2_US_Florida', u'T2_FR_GRIF_IRFU', u'T1_IT_CNAF_Disk', u'T1_RU_JINR_Disk', u'T2_UK_London_IC', u'T2_IT_Bari', u'T2_US_Nebraska', u'T2_FR_CCIN2P3', u'T2_US_UCSD', u'T2_ES_CIEMAT', u'T2_RU_IHEP', u'T2_US_Wisconsin', u'T2_HU_Budapest', u'T2_US_MIT', u'T2_DE_RWTH', u'T2_KR_KISTI', u'T2_RU_INR', u'T2_CH_CERN', u'T2_US_Vanderbilt', u'T2_BR_SPRACE'])
2020-12-18 13:03:24,845:INFO:MSTransferor: Request pdmvserv_Run2018D_SingleMuon_UL2018_MiniAODv2_201218_123105_4141 has 579 initial blocks from getPrimaryBlocks
```

Changed files:
src/python/WMCore/MicroService/Unified/RequestInfo.py

diff --git a/src/python/WMCore/MicroService/Unified/RequestInfo.py b/src/python/WMCore/MicroService/Unified/RequestInfo.py
index 639fea9f5..6d9c3609d 100644
--- a/src/python/WMCore/MicroService/Unified/RequestInfo.py
+++ b/src/python/WMCore/MicroService/Unified/RequestInfo.py
@@ -386,15 +386,15 @@ class RequestInfo(MSCore):
                     datasets.add(dataIn['name'])
 
         # fetch all block names and their sizes from Rucio
-        self.logger.info("Fetching parent/primary block info for %d containers against Rucio: %s",
+        self.logger.info("Fetching parent/primary block sizes for %d containers against Rucio: %s",
                          len(datasets), self.msConfig['rucioUrl'])
         blocksByDset = getBlocksAndSizeRucio(datasets, self.msConfig['rucioUrl'], self.rucioToken)
 
         # now check if any of our calls failed; if so, workflow needs to be skipped from this cycle
         # FIXME: isn't there a better way to do this?!?
-        for dset, value in blocksByDset.items():
-            if value is None:
-                retryDatasets.append(dset)
+        for dsetName in blocksByDset:
+            if blocksByDset[dsetName] is None:
+                retryDatasets.append(dsetName)
         if retryDatasets:
             for wflow in workflows:
                 if wflow.getInputDataset() in retryDatasets or wflow.getParentDataset() in retryDatasets:
@@ -423,7 +423,6 @@ class RequestInfo(MSCore):
             except Exception:
                 self.logger.error("Workflow: %s will be retried in the next cycle", wflow.getName())
                 retryWorkflows.append(wflow)
-
         # remove workflows that failed one or more of the bulk queries to the data-service
         self._workflowRemoval(workflows, retryWorkflows)
 
@@ -482,8 +481,7 @@ class RequestInfo(MSCore):
                     if block in blocksDict:
                         finalBlocks[block] = deepcopy(blocksDict[block])
                     else:
-                        # FIXME: I guess it won't ever happen in Rucio...
-                        self.logger.info("FIXME: Dropping block with no replicas in PhEDEx: %s", block)
+                        self.logger.warning("Dropping block existent in DBS but not in Rucio: %s", block)
         elif runBlack:
             # only run blacklist set
             self.logger.info("Fetching runs in blocks for RunBlacklist for %s", wflow.getName())
@@ -530,6 +528,22 @@ class RequestInfo(MSCore):
 
         if not finalBlocks:
             finalBlocks = blocksDict
+        return self._removeZeroSizeBlocks(finalBlocks)
+
+    def _removeZeroSizeBlocks(self, blocksDict):
+        """
+        Given a dictionary of blocks and their block size and location information,
+        return only blocks with >0 bytes of block size (Rucio blocks with no replicas/
+        files result in blocks with None size).
+        :return: dictionary of block names and block size
+        """
+        finalBlocks = {}
+        for blockName in blocksDict:
+            if blocksDict[blockName]['blockSize']:
+                finalBlocks[blockName] = blocksDict[blockName]
+            else:
+                self.logger.info("Dropping block: %s with no files and size: %s",
+                                 blockName, blocksDict[blockName]['blockSize'])
         return finalBlocks
 
     def getParentChildBlocks(self, workflows):