Repository: dmwm/wmcore
Issue: Inconsistencies between TMDB/DBS while filling the pileup json config

I saw WorkQueueManager failing to process the pileup dataset (which creates the json file) and the reason is that we use first DBS in order to fetch dataset/block/file information, then we use PhEDEx to find block subscriptions:
```
2019-02-04 17:20:44,573:140045524363008:ERROR:WMBSHelper:Failed to create subscription. Error: u'/Neutrino_E-10_gun/RunIISummer17PrePremix-PUAutumn18_102X_upgrade2018_realistic_v15-v1/GEN-SIM-DIGI-RAW#b94a8445-24e6-4525-9016-21ffca4c3f8a'
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 408, in createSubscriptionAndAddFiles
    sub = self.createSubscription(self.topLevelTask, self.topLevelFileset)
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 257, in createSubscription
    sub = self._createSubscriptionsInWMBS(task, fileset, alternativeFilesetClose)
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 272, in _createSubscriptionsInWMBS
    self.createSandbox()
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WorkQueue/WMBSHelper.py", line 208, in createSandbox
    sandboxCreator.makeSandbox(self.cachepath, self.wmSpec)
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WMRuntime/SandboxCreator.py", line 131, in makeSandbox
    fetcher(task)
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py", line 200, in __call__
    self._createPileupConfigFile(helper)
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py", line 178, in _createPileupConfigFile
    configDict = self._queryDbsAndGetPileupConfig(helper, dbsReader)
  File "/data/srv/wmagent/v1.1.20.pre4/sw/slc7_amd64_gcc630/cms/wmagent/1.1.20.pre4/lib/python2.7/site-packages/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py", line 76, in _queryDbsAndGetPileupConfig
    blockDict[block]['PhEDExNodeNames'] = list(nodes)
KeyError: u'/Neutrino_E-10_gun/RunIISummer17PrePremix-PUAutumn18_102X_upgrade2018_realistic_v15-v1/GEN-SIM-DIGI-RAW#b94a8445-24e6-4525-9016-21ffca4c3f8a'
```

Changed files:
src/python/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py

diff --git a/src/python/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py b/src/python/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py
index a42fd2b29..3451db5cb 100644
--- a/src/python/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py
+++ b/src/python/WMCore/WMSpec/Steps/Fetchers/PileupFetcher.py
@@ -9,6 +9,7 @@ import datetime
 import os
 import shutil
 import time
+import logging
 from json import JSONEncoder
 
 import WMCore.WMSpec.WMStep as WMStep
@@ -73,8 +74,11 @@ class PileupFetcher(FetcherInterface):
                 blockReplicasInfo = phedex.getReplicaPhEDExNodesForBlocks(dataset=dataset, complete='y')
                 for block in blockReplicasInfo:
                     nodes = set(blockReplicasInfo[block]) - node_filter
-                    blockDict[block]['PhEDExNodeNames'] = list(nodes)
-                    blockDict[block]['FileList'] = sorted(blockDict[block]['FileList'])
+                    try:
+                        blockDict[block]['PhEDExNodeNames'] = list(nodes)
+                        blockDict[block]['FileList'] = sorted(blockDict[block]['FileList'])
+                    except KeyError:
+                        logging.warning("Block '%s' does not have any complete PhEDEx replica", block)
 
             resultDict[pileupType] = blockDict
         return resultDict