Repository: dmwm/wmcore
Issue: Break closure of phedex blocks into smaller chunks

**Impact of the bug**
WMAgent (PhEDExInjector)

**Describe the bug**
Apparently we put too much data into the same PhEDEx call to close blocks here:
https://github.com/dmwm/WMCore/blob/master/src/python/WMComponent/PhEDExInjector/PhEDExInjectorPoller.py#L375

it's one call per location, and that location might have many many blocks from multiple datasets.
The problem is, if there is a problem with any of those blocks (e.g., the current state of vocms0253 which has a **dataset** closed in PhEDEx, see [1]), the whole request would fail and no blocks can be closed...

**How to reproduce it**
none

**Expected behavior**
My suggestion is to break the http requests per location and dataset, something like
for location in locations:
  for dataset in datasets:
    make a phedex request

Yes, it will increase the amount of PhEDEx calls, but still it's going to be 1 call per block that we close, which should be just fine for the phedex data-service.

**Additional context and error message**
[1]
```
2019-11-27 22:23:08,945:140168429426432:WARNING:Service:The cachefile /data/srv/wmagent/v1.2.6.patch1/install/wmagent/PhEDExInjector/.wmcore_cache/.wmcore_cache_31961/requests/cmsweb.cern.ch:8443/-7685143179613178687_POST_inject does n
ot exist and the service at https://cmsweb.cern.ch:8443/phedex/datasvc/json/prod/inject is unavailable - it returned 400 because Bad Request with result: injectData error: dataset /RadionToWW_narrow_M-6500_TuneCUETP8M1_13TeV-madgraph-p
ythia8/RunIISummer16MiniAODv3-PUMoriond17_94X_mcRun2_asymptotic_v3-v1/MINIAODSIM is closed\\n

2019-11-27 22:23:08,946:140168429426432:ERROR:PhEDExInjectorPoller:PhEDEx block close failed with HTTPException: 400 injectData error: dataset /RadionToWW_narrow_M-6500_TuneCUETP8M1_13TeV-madgraph-pythia8/RunIISummer16MiniAODv3-PUMorio
nd17_94X_mcRun2_asymptotic_v3-v1/MINIAODSIM is closed\\n
```

Changed files:
src/python/WMComponent/PhEDExInjector/PhEDExInjectorPoller.py

diff --git a/src/python/WMComponent/PhEDExInjector/PhEDExInjectorPoller.py b/src/python/WMComponent/PhEDExInjector/PhEDExInjectorPoller.py
index a5c3b08f8..6be19e6cc 100644
--- a/src/python/WMComponent/PhEDExInjector/PhEDExInjectorPoller.py
+++ b/src/python/WMComponent/PhEDExInjector/PhEDExInjectorPoller.py
@@ -351,7 +351,7 @@ class PhEDExInjectorPoller(BaseWorkerThread):
 
         migratedBlocks = self.getMigrated.execute()
 
-        for siteName in migratedBlocks.keys():
+        for siteName in migratedBlocks:
             # SE names can be stored in DBSBuffer as that is what is returned in
             # the framework job report.  We'll try to map the SE name to a
             # PhEDEx node name here.
@@ -372,25 +372,25 @@ class PhEDExInjectorPoller(BaseWorkerThread):
                 logging.error(msg)
                 continue
 
-            xmlData = self.createInjectionSpec(migratedBlocks[siteName])
-            logging.debug("closeBlocks XMLData: %s", xmlData)
+            for dset, blocks in migratedBlocks[siteName].items():
+                xmlData = self.createInjectionSpec({dset: blocks})
+                logging.debug("closeBlocks XMLData: %s", xmlData)
 
-            try:
-                injectRes = self.phedex.injectBlocks(location, xmlData)
-            except HTTPException as ex:
-                logging.error("PhEDEx block close failed with HTTPException: %s %s", ex.status, ex.result)
-            except Exception as ex:
-                msg = "PhEDEx block close failed with Exception: %s" % str(ex)
-                logging.exception(msg)
-            else:
-                logging.debug("Block closing result: %s", injectRes)
-
-                if "error" in injectRes:
-                    msg = "Error closing blocks with data %s: %s" % (migratedBlocks[siteName], injectRes["error"])
-                    logging.error(msg)
+                try:
+                    injectRes = self.phedex.injectBlocks(location, xmlData)
+                except HTTPException as ex:
+                    logging.error("PhEDEx block close failed with HTTPException: %s %s", ex.status, ex.result)
+                except Exception as ex:
+                    msg = "PhEDEx block close failed with Exception: %s" % str(ex)
+                    logging.exception(msg)
                 else:
-                    for datasetName in migratedBlocks[siteName]:
-                        for blockName in migratedBlocks[siteName][datasetName]:
+                    logging.debug("Block closing result: %s", injectRes)
+
+                    if "error" in injectRes:
+                        logging.error("Failed to close blocks due to: %s, for data: %s",
+                                      injectRes["error"], migratedBlocks[siteName][dset])
+                    else:
+                        for blockName in blocks:
                             logging.info("Block closed in PhEDEx: %s", blockName)
                             self.setBlockClosed.execute(blockName)