Repository: dmwm/wmcore
Issue: Broken fix in ResourceControlUpdater for the SSB deprecation

**Impact of the bug**
WMAgents (including those in production right now)

**Describe the bug**
I have just scanned the AgentStatusWatcher logs and I see it's hitting an exception when parsing the site state in MonIT. It's either coming from a site with missing information in MonIT, or I might have missed something in my fix made here:
https://github.com/dmwm/WMCore/pull/9555

**How to reproduce it**
Can be seen in any agent that gets deployed and has the grafana token.

**Expected behavior**
Properly deal with site status from MonIT, local database and switch their statuses when required.
Which also means, fix that bug/exception.

**Additional context and error message**
Traceback from the agent:
```
2020-03-26 10:02:25,105:140105039865600:INFO:AgentStatusPoller:592 out of 592 documents successfully sent to AMQ
2020-03-26 10:02:25,106:140105039865600:INFO:BaseWorkerThread:AgentStatusPoller took 15.845 secs to execute
2020-03-26 10:05:19,439:140105031472896:INFO:Grafana:Calling API: https://monit-grafana.cern.ch/api/datasources/proxy/9475/_msearch with query: '{"search_type":"query_then_fetch","ignore_unavailable":true,"index":["monit_prod_cmssst_*"]}\\n{"size":500,"query":{"bool":{"filter":[{"range":{"metadata.timestamp":{"gte":"now-1d","lte":"now","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"metadata.type: ssbmetric AND metadata.type_prefix:raw AND metadata.path: sts15min"}}]}},"sort":{"metadata.timestamp":{"order":"desc","unmapped_type":"boolean"}},"script_fields":{},"docvalue_fields":["metadata.timestamp"]}\\n'
2020-03-26 10:05:19,531:140105031472896:ERROR:ResourceControlUpdater:Unknown status 'unknown' for site T2_IT_Pisa, please check SSB
2020-03-26 10:05:19,531:140105031472896:ERROR:ResourceControlUpdater:Error occurred, will retry later:
2020-03-26 10:05:19,531:140105031472896:ERROR:ResourceControlUpdater:'state'
2020-03-26 10:05:19,531:140105031472896:ERROR:ResourceControlUpdater:Trace back: 
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.3.0/sw/slc7_amd64_gcc630/cms/wmagent/1.3.0/lib/python2.7/site-packages/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py", line 92, in algorithm
    self.checkStatusChanges(sitesRC, ssbSiteStatus)
  File "/data/srv/wmagent/v1.3.0/sw/slc7_amd64_gcc630/cms/wmagent/1.3.0/lib/python2.7/site-packages/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py", line 177, in checkStatusChanges
    if infoRC[site]['state'] != infoSSB[site]['state']:
KeyError: 'state'

2020-03-26 10:05:19,532:140105031472896:INFO:ResourceControlUpdater:Resource control cycle finished updating site state and thresholds.
```


Changed files:
src/python/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py

diff --git a/src/python/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py b/src/python/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py
index cb5d43b87..6e3a6eb0a 100644
--- a/src/python/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py
+++ b/src/python/WMComponent/AgentStatusWatcher/ResourceControlUpdater.py
@@ -26,6 +26,11 @@ class ResourceControlUpdater(BaseWorkerThread):
         BaseWorkerThread.__init__(self)
         self.config = config
 
+        self.ssb2AgentStatus = {'enabled': 'Normal',
+                                'drain': 'Draining',
+                                'disabled': 'Down',
+                                'test': 'Draining',
+                                'unknown': None}
         self.tasksCPU = ['Processing', 'Production']
         self.tasksIO = ['Merge', 'Cleanup', 'Harvesting', 'LogCollect', 'Skim']
         self.minCPUSlots = 50
@@ -260,7 +265,8 @@ class ResourceControlUpdater(BaseWorkerThread):
             ssbStatus = ssbState[site]['prod_status']
             wmcoreStatus = self.getState(str(ssbStatus))
             if not wmcoreStatus:
-                logging.error("Unknown status '%s' for site %s, please check SSB", ssbStatus, site)
+                logging.warning("Site %s has an unknown SSB status '%s'. Skipping it!", site, ssbStatus)
+                ssbStatus.pop(site, None)
             else:
                 ssbState[site] = {'state': wmcoreStatus}
 
@@ -272,14 +278,10 @@ class ResourceControlUpdater(BaseWorkerThread):
 
         Translates SSB states into resource control state
         """
-        ssb2agent = {'enabled': 'Normal',
-                     'drain': 'Draining',
-                     'disabled': 'Down',
-                     'test': 'Draining'}
-        # 'test' state behaviour varies between production and tier0 agents
-        ssb2agent['test'] = 'Normal' if self.tier0Mode else "Draining"
-
-        return ssb2agent.get(stateSSB)
+        if self.tier0Mode and stateSSB == "test":
+            # a test site for T0 has a different meaning than for production
+            return "Normal"
+        return self.ssb2AgentStatus.get(stateSSB)
 
     def updateSiteState(self, siteName, state):
         """