Repository: dmwm/wmcore
Issue: MSUnmerged: Protection for missing protectedLFN list not working

**Impact of the bug**
MSUnmerged

**Describe the bug**
While going through the logs of MSUnmerged I fond the following error: [4]. And the reason for that is [5]. Long story short the protection I have put into the code [1] to stop the service cycle if `WMStatsServer` is unreachable and the `protectedLFN` list is empty, simply does not work. 

Going into details: The query to `WMStatsServer` (and all inintial data setup for the service run) is enclosed  in a try-except block so that we  do not break the thread. The default pollicy for handling those exceptions is to always continue running no matter what. While the check used for protection is against an empty value for the `protectedLFN` list. Both strategies are correct by their own means. It is simply that I have overlooked a situation as  the monthly upgrade when everything is down, including the WMStatServer. Then the query to `WMStatServer` does return an empty value or exit normally but instead fires an exception [4] at line [2]. Which is basically the line just before the check for and empty `protectedLFN` list. So instead of normally exiting the current run with a summary report (as expected), we dive into the cycle and start iterating through the lists of RSEs to clean. It is only by chance that I have left the default  value of the `protectedLFN` list to be a list instead of a set, and then later in the code the set operation at [3] fails and it fails for every RSE, so we do not delete any data, but instead mark the RSE as gone through and we wait until its data at `rucioConmon` is refreshed. So currently, due to the so convoluted situation and an overlap of two bugs (an exception fired before a service protection check && variable type mismatch), we are only skipping the first iteration through the RSEs list, and no data is deleted. On the next run the list of protectedLfns is refreshed and everything goes back to normal. 

**How to reproduce it**
Restart WMStatsServer at the same time with the MSUnmerged micro service.

**Expected behavior**
Respect the check for an empty `portectedLFN` list regardless if it is due to an empty value returned by the API or due to an exception fired because of the `WMStatsServer` unavailiability. 

**Additional context and error message**
[1]
https://github.com/dmwm/WMCore/blob/88b70a21208c1b4b849a122336c600210a71c487/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L156

[2]
https://github.com/dmwm/WMCore/blob/88b70a21208c1b4b849a122336c600210a71c487/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L153

[3]
https://github.com/dmwm/WMCore/blob/88b70a21208c1b4b849a122336c600210a71c487/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L491

[4]
```
2021-08-14 17:19:45,722:ERROR:MSUnmerged: plineUnmerged: General error from pipeline. RSE: T2_FR_GRIF_LLR. Error: unsupported operand type(s) for -: 'set' and 'list' Will retry again in the 
Traceback (most recent call last):
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/MicroService/MSUnmerged/MSUnmerged.py", line 231, in _execute
    pline.run(MSUnmergedRSE(rseName))
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/Utils/Pipeline.py", line 140, in run
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/Utils/Pipeline.py", line 140, in <lambda>
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/Utils/Pipeline.py", line 72, in __call__
    return self.run(obj)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/Utils/Pipeline.py", line 75, in run
    return self.func(obj, *self.args, **self.kwargs)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/MicroService/MSUnmerged/MSUnmerged.py", line 492, in filterUnmergedFiles
    rse['dirs']['toDelete'] = rse['dirs']['allUnmerged'] - self.protectedLFNs
TypeError: unsupported operand type(s) for -: 'set' and 'list'
```

[5]
```
2021-08-14 17:19:45,494:ERROR:MSUnmerged: Unknown exception while running MSUnmerged thread Error: url=https://cmsweb.cern.ch:8443/wmstatsserver/data/protectedlfns, code=503, reason=Service Unavailable, headers={'Date': 'Sat, 14 Aug 2021 15:19:45 GMT', 'Server': 'nginx/1.17.10', 'Content-Type': 'text/html;charset=utf-8', 'X-Rest-Status': '208', 'X-Error-Http': '503', 'X-Error-Id': '6f3069efc807c8820fd3442e854cb7ce', 'X-Error-Detail': 'DataCache is Empty', 'X-Rest-Time': '1252.413 us', 'CMS-Server-Time': 'D=13383 t=1628954385479248', 'Connection': 'close', 'Transfer-Encoding': 'chunked'}, result=b'<!DOCTYPE html PUBLIC\\n"-//W3C//DTD XHTML 1.0 Transitional//EN"\\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\\n<html>\\n<head>\\n    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>\\n    <title>503 Service Unavailable</title>\\n    <style type="text/css">\\n    #powered_by {\\n        margin-top: 20px;\\n        border-top: 2px solid black;\\n        font-style: italic;\\n    }\\n\\n    #traceback {\\n        color: red;\\n    }\\n    </style>\\n</head>\\n    <body>\\n        <h2>503 Service Unavailable</h2>\\n        <p>DataCache is Empty</p>\\n        <pre id="traceback"></pre>\\n    <div id="powered_by">\\n      <span>\\n        Powered by <a href="http://www.cherrypy.org">CherryPy 17.4.0</a>\\n      </span>\\n    </div>\\n    </body>\\n</html>\\n'
Traceback (most recent call last):
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/MicroService/MSUnmerged/MSUnmerged.py", line 154, in execute
    self.protectedLFNs = set(self.wmstatsSvc.getProtectedLFNs())
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/WMStatsServer/WMStatsServer.py", line 134, in getProtectedLFNs
    return self._getResult(callname, verb="GET")
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/WMStatsServer/WMStatsServer.py", line 45, in _getResult
    f = self.refreshCache(cfile, callname, args, encoder=encoder,
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/Service.py", line 206, in refreshCache
    self.getData(cachefile, url, inputdata, incoming_headers, encoder, decoder, verb, contentType)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/Service.py", line 321, in getData
    raise he
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/Service.py", line 279, in getData
    data, dummyStatus, dummyReason, from_cache = self["requests"].makeRequest(uri=url,
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 159, in makeRequest
    result, response = self.makeRequest_pycurl(uri, data, verb, headers)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 176, in makeRequest_pycurl
    response, result = self.reqmgr.request(uri, data, headers, verb=verb,
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/Utils/PortForward.py", line 67, in portMangle
    return callFunc(callObj, newUrl, *args, **kwargs)
  File "/data/srv/HG2108h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.5.1.pre6/lib/python3.8/site-packages/WMCore/Services/pycurl_manager.py", line 306, in request
    raise exc
http.client.HTTPException: url=https://cmsweb.cern.ch:8443/wmstatsserver/data/protectedlfns, code=503, reason=Service Unavailable, headers={'Date': 'Sat, 14 Aug 2021 15:19:45 GMT', 'Server': 'nginx/1.17.10', 'Content-Type': 'text/html;charset=utf-8', 'X-Rest-Status': '208', 'X-Error-Http': '503', 'X-Error-Id': '6f3069efc807c8820fd3442e854cb7ce', 'X-Error-Detail': 'DataCache is Empty', 'X-Rest-Time': '1252.413 us', 'CMS-Server-Time': 'D=13383 t=1628954385479248', 'Connection': 'close', 'Transfer-Encoding': 'chunked'}, result=b'<!DOCTYPE html PUBLIC\\n"-//W3C//DTD XHTML 1.0 Transitional//EN"\\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\\n<html>\\n<head>\\n    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>\\n    <title>503 Service Unavailable</title>\\n    <style type="text/css">\\n    #powered_by {\\n        margin-top: 20px;\\n        border-top: 2px solid black;\\n        font-style: italic;\\n    }\\n\\n    #traceback {\\n        color: red;\\n    }\\n    </style>\\n</head>\\n    <body>\\n        <h2>503 Service Unavailable</h2>\\n        <p>DataCache is Empty</p>\\n        <pre id="traceback"></pre>\\n    <div id="powered_by">\\n      <span>\\n        Powered by <a href="http://www.cherrypy.org">CherryPy 17.4.0</a>\\n      </span>\\n    </div>\\n    </body>\\n</html>\\n'
```
```

Changed files:
src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py

diff --git a/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py b/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
index 38d84a151..e39c5a231 100644
--- a/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
+++ b/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py
@@ -145,35 +145,37 @@ class MSUnmerged(MSCore):
         # start threads in MSManager which should call this method
         summary = dict(UNMERGED_REPORT)
 
-        # refresh statistics on every poling cycle
-
+        # fetch the protectedLFNs list
         try:
-            self.rseConsStats = self.rucioConMon.getRSEStats()
-            # self.logger.debug("RucioConMon Stats: %s", pformat(self.rseConsStats))
             self.protectedLFNs = set(self.wmstatsSvc.getProtectedLFNs())
             # self.logger.debug("protectedLFNs: %s", pformat(self.protectedLFNs))
 
             if not self.protectedLFNs:
-                msg = "Could not fetch list with protectedLFNs from WMStatServer. "
+                msg = "Could not fetch the protectedLFNs list from WMStatServer. "
                 msg += "Skipping the current run."
-                msg += "\\nTotal number of RSEs processed: %s."
-                msg += "\\nTotal number of files fetched from RucioConMon: %s."
-                msg += "\\nNumber of RSEs cleaned: %s."
-                msg += "\\nNumber of files deleted: %s."
-                self.logger.info(msg,
-                                 0,
-                                 0,
-                                 0,
-                                 0)
-                self.updateReportDict(summary, "total_num_rses", 0)
-                self.updateReportDict(summary, "total_num_files", 0)
-                self.updateReportDict(summary, "num_rses_cleaned", 0)
-                self.updateReportDict(summary, "num_files_deleted", 0)
+                self.logger.error(msg)
                 return summary
         except Exception as ex:
-            msg = "Unknown exception while running MSUnmerged thread Error: {}".format(str(ex))
+            msg = "Unknown exception while trying to fetch the protectedLFNs list from WMStatServer. Error: {}".format(str(ex))
+            self.logger.exception(msg)
+            self.updateReportDict(summary, "error", msg)
+            return summary
+
+        # refresh statistics on every poling cycle
+        try:
+            self.rseConsStats = self.rucioConMon.getRSEStats()
+            # self.logger.debug("protectedLFNs: %s", pformat(self.protectedLFNs))
+
+            if not self.rseConsStats:
+                msg = "Could not fetch statistics from Rucio Consistency Monitor. "
+                msg += "Skipping the current run."
+                self.logger.error(msg)
+                return summary
+        except Exception as ex:
+            msg = "Unknown exception while trying to fetch statistics from Rucio Consistency Monitor. Error: {}".format(str(ex))
             self.logger.exception(msg)
             self.updateReportDict(summary, "error", msg)
+            return summary
 
         try:
             rseList = self.getRSEList()
@@ -182,9 +184,10 @@ class MSUnmerged(MSCore):
             self.logger.info(msg, len(rseList), pformat(rseList), self.msConfig["limitRSEsPerInstance"])
             random.shuffle(rseList)
         except Exception as err:  # general error
-            msg = "Unknown exception while trying to estimate the final RSEs to work on. Error: %s", str(err)
+            msg = "Unknown exception while trying to estimate the final list of RSEs to work on. Error: {}".format(str(err))
             self.logger.exception(msg)
             self.updateReportDict(summary, "error", msg)
+            return summary
 
         try:
             totalNumRses, totalNumFiles, numRsesCleaned, numFilesDeleted = self._execute(rseList)