Repository: dmwm/wmcore
Issue: Use of DBS filelumis RESTful API needs to be made compatible

**Impact of the new feature**
MSTransferor, but there could be other parts of the code (including DBS3Reader)

**Is your feature request related to a problem? Please describe.**
While debugging why our LumiList-based templates don't manage to pass the `assigned` request status, I noticed that MSTransferor is actually failing to run the lumi list comparison. The reason for that is that Go and the Python implementation of the `filelumis` RESTful API yield data in different format.

I believe this is only a problem because we make a direct call to the REST endpoint, without using the dbs3-client. Nonetheless, it needs to be verified as well.

In MSTransferor, this line is actually raising an exception:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/MicroService/MSTransferor/RequestInfo.py#L409

as one can interactive check with:
```
>>> fileMask = LumiList(runsAndLumis={'123': 333})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/srv/wmagent/v1.5.7/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.5.7/lib/python3.8/site-packages/WMCore/DataStructs/LumiList.py", line 96, in __init__
    for lumi in sorted(int(l) for l in lumiList):
TypeError: 'int' object is not iterable
```

**Describe the solution you'd like**
The MSTransferor code needs to be updated such that it can deal with both response formats, allowing for a transparent migration of the DBS server. Here is the location calling `filelumis`:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/MicroService/Tools/Common.py#L247

we still need to investigate whether this very same DBS API is called in different places, and we need to ensure that our code will behave properly, regardless of the DBSServer flavor.

In addition to that, we should also investigate whether any changes are required for the `DBS3Reader` module.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
The python-based server does an aggregation of the lumis grouped by run+lfn, example:
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filelumis?validFileOnly=1&block_name=/DoubleMuon/Run2018D-v1/RAW%23590119e4-3060-4c9b-9b38-2b16f6844bab

while the Go implementation yields a row for each lfn+run+lumi information:
https://cmsweb-testbed.cern.ch/dbs/int/global/DBSReader/filelumis?validFileOnly=1&block_name=/DoubleMuon/Run2018D-v1/RAW%23590119e4-3060-4c9b-9b38-2b16f6844bab

Changed files:
src/python/WMCore/MicroService/Tools/Common.py, test/python/WMCore_t/MicroService_t/Tools_t/Common_t.py

diff --git a/src/python/WMCore/MicroService/Tools/Common.py b/src/python/WMCore/MicroService/Tools/Common.py
index 953e76423..a16239758 100644
--- a/src/python/WMCore/MicroService/Tools/Common.py
+++ b/src/python/WMCore/MicroService/Tools/Common.py
@@ -25,12 +25,31 @@ from Utils.CertTools import getKeyCertFromEnv
 from WMCore.Services.pycurl_manager import RequestHandler
 from WMCore.Services.pycurl_manager import getdata as multi_getdata
 
+# DBS agregators
+from dbs.apis.dbsClient import aggRuns, aggFileLumis
+
 
 # static variables
 STEP_PAT = re.compile(r'Step[0-9]')
 TASK_PAT = re.compile(r'Task[0-9]')
 
 
+def isEmptyResults(row):
+    """
+    _isEmptyResults_
+
+    Evaluates whether row data contains empty result set
+    :return: bool
+    """
+    if 'data' not in row:
+        raise Exception("provided result dict does not contain 'data' key")
+    # if code is not present in row it means it was success (HTTP status code 200)
+    code = int(row.get('code', 200))
+    data = row['data']
+    if (code >= 200 and code < 400) and data in (None, []):
+        return True
+    return False
+
 def getMSLogger(verbose, logger=None):
     """
     _getMSLogger_
@@ -63,7 +82,7 @@ def dbsInfo(datasets, dbsUrl):
 
     for row in data:
         dataset = row['url'].split('=')[-1]
-        if row['data'] is None:
+        if isEmptyResults(row):
             print("FAILURE: dbsInfo for %s. Error: %s %s" % (dataset, row.get('code'), row.get('error')))
             continue
         rows = json.loads(row['data'])
@@ -232,7 +251,7 @@ def getBlocksByDsetAndRun(datasetName, runList, dbsUrl):
 
     for row in data:
         dataset = row['url'].rsplit('=')[-1]
-        if row['data'] is None:
+        if isEmptyResults(row):
             msg = "Failure in getBlocksByDsetAndRun for %s. Error: %s %s" % (dataset,
                                                                              row.get('code'),
                                                                              row.get('error'))
@@ -261,12 +280,13 @@ def getFileLumisInBlock(blocks, dbsUrl, validFileOnly=1):
 
     for row in data:
         blockName = unquote(row['url'].rsplit('=')[-1])
-        if row['data'] is None:
+        if isEmptyResults(row):
             msg = "Failure in getFileLumisInBlock for block %s. Error: %s %s" % (blockName,
                                                                                  row.get('code'),
                                                                                  row.get('error'))
             raise RuntimeError(msg)
         rows = json.loads(row['data'])
+        rows = aggFileLumis(rows) # adjust to DBS Go server output
         runLumisByBlock.setdefault(blockName, [])
         for item in rows:
             runLumisByBlock[blockName].append(item)
@@ -288,7 +308,7 @@ def findBlockParents(blocks, dbsUrl):
     for row in data:
         blockName = unquote(row['url'].rsplit('=')[-1])
         dataset = blockName.split("#")[0]
-        if row['data'] is None:
+        if isEmptyResults(row):
             print("Failure in findBlockParents for block %s. Error: %s %s" % (blockName,
                                                                               row.get('code'),
                                                                               row.get('error')))
@@ -322,12 +342,13 @@ def getRunsInBlock(blocks, dbsUrl):
     data = multi_getdata(urls, ckey(), cert())
     for row in data:
         blockName = unquote(row['url'].rsplit('=')[-1])
-        if row['data'] is None:
+        if isEmptyResults(row):
             msg = "Failure in getRunsInBlock for block %s. Error: %s %s" % (blockName,
                                                                             row.get('code'),
                                                                             row.get('error'))
             raise RuntimeError(msg)
         rows = json.loads(row['data'])
+        rows = aggRuns(rows) # adjust to DBS Go server output
         runsByBlock[blockName] = rows[0]['run_num']
     return runsByBlock
 
@@ -366,7 +387,7 @@ def eventsLumisInfo(inputs, dbsUrl, validFileOnly=0, sumOverLumi=0):
     data = multi_getdata(urls, ckey(), cert())
     for row in data:
         data = unquote(row['url'].split('=')[-1])
-        if row['data'] is None:
+        if isEmptyResults(row):
             print("FAILURE: eventsLumisInfo for %s. Error: %s %s" % (data,
                                                                     row.get('code'),
                                                                     row.get('error')))
@@ -585,7 +606,7 @@ def findParent(datasets, dbsUrl):
 
     for row in data:
         dataset = row['url'].split('=')[-1]
-        if row['data'] is None:
+        if isEmptyResults(row):
             print("Failure in findParent for dataset %s. Error: %s %s" % (dataset,
                                                                           row.get('code'),
                                                                           row.get('error')))
diff --git a/test/python/WMCore_t/MicroService_t/Tools_t/Common_t.py b/test/python/WMCore_t/MicroService_t/Tools_t/Common_t.py
index d152bbc41..b80f133aa 100644
--- a/test/python/WMCore_t/MicroService_t/Tools_t/Common_t.py
+++ b/test/python/WMCore_t/MicroService_t/Tools_t/Common_t.py
@@ -7,7 +7,8 @@ from __future__ import division, print_function
 
 import unittest
 
-from WMCore.MicroService.Tools.Common import dbsInfo, getEventsLumis, findParent
+from WMCore.MicroService.Tools.Common import \\
+        dbsInfo, getEventsLumis, findParent, isEmptyResults
 
 
 class CommonTest(unittest.TestCase):
@@ -47,6 +48,19 @@ class CommonTest(unittest.TestCase):
         parents = findParent(self.child, self.dbsUrl)
         self.assertEqual(parents[self.child[0]], '/SingleElectron/Run2016B-v2/RAW')
 
+    def test_isEmptyResults(self):
+        row = {'code': 200, 'data': None}
+        result = isEmptyResults(row)
+        self.assertEqual(result, True)
+        row = {'code': 200, 'data': []}
+        result = isEmptyResults(row)
+        self.assertEqual(result, True)
+        row = {'code': 400, 'data': None}
+        result = isEmptyResults(row)
+        self.assertEqual(result, False)
+        row = {'code': 400, 'data': []}
+        result = isEmptyResults(row)
+        self.assertEqual(result, False)
 
 if __name__ == '__main__':
     unittest.main()