Repository: dmwm/wmcore
Issue: ErrorHandler crash when it fails to pull the agent configuration from reqmgr_aux

**Impact of the bug**
WMAgent

**Describe the bug**
ErrorHandler terminates itself when making an Int vs None comparison for the job retry. Apparently, it's only caused because it failed to fetch the agent configuration from central CouchDB (reqmgr_aux db). Traceback in [1]

**How to reproduce it**
Should be reproducible from the description above.

**Expected behavior**
If the component fails to update its cache of agent configuration, and the in-memory cache is empty, it should simply skip that cycle.
Of course, we could make it better and plug an alert in the case where that cycle gets skipped many times in a row, but I would say it's not required to be delivered with this ticket.

**Additional context and error message**
[1]
```
2021-08-27 17:36:10,608:139922251294464:WARNING:Service:The cachefile /storage/local/data1/cmsdataops/srv/wmagent/v1.5.2/install/wmagentpy3/ErrorHandler/.wmcore_cache/.wmcore_cache_42935/requests/cmsweb.cern.ch:8443/956307719776393200_GET
_wmagentconfig_cmsgwms-submit5.fnal.gov does not exist and the service at https://cmsweb.cern.ch:8443/reqmgr2/data/wmagentconfig/cmsgwms-submit5.fnal.gov is unavailable - it returned 500 because Internal Server Error with result: b'<!DOCT
YPE html PUBLIC\\n"-//W3C//DTD XHTML 1.0 Transitional//EN"\\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\\n<html>\\n<head>\\n    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>\\n    <title>500 Internal
 Server Error</title>\\n    <style type="text/css">\\n    #powered_by {\\n        margin-top: 20px;\\n        border-top: 2px solid black;\\n        font-style: italic;\\n    }\\n\\n    #traceback {\\n        color: red;\\n    }\\n    </style>\\n</he
ad>\\n    <body>\\n        <h2>500 Internal Server Error</h2>\\n        <p>Server error</p>\\n        <pre id="traceback"></pre>\\n    <div id="powered_by">\\n      <span>\\n        Powered by <a href="http://www.cherrypy.org">CherryPy 17.4.0</a
>\\n      </span>\\n    </div>\\n    </body>\\n</html>\\n'

2021-08-27 17:36:10,609:139922251294464:WARNING:GenericDataCache:Passive failure while looking data up in the memory cache. Error: url=https://cmsweb.cern.ch:8443/reqmgr2/data/wmagentconfig/cmsgwms-submit5.fnal.gov, code=500, reason=Inter
nal Server Error, headers={'Date': 'Fri, 27 Aug 2021 22:36:10 GMT', 'Server': 'nginx/1.17.10', 'Content-Type': 'text/html;charset=utf-8', 'Content-Length': '742', 'X-Rest-Status': '400', 'X-Error-Http': '500', 'X-Error-Id': '2010520ab159e
615fc66f2ca111d2766', 'X-Error-Detail': 'Server error', 'X-Rest-Time': '69001.913 us', 'Vary': 'Accept-Encoding', 'CMS-Server-Time': 'D=82152 t=1630103770474212', 'Connection': 'close'}, result=b'<!DOCTYPE html PUBLIC\\n"-//W3C//DTD XHTML 
1.0 Transitional//EN"\\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\\n<html>\\n<head>\\n    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>\\n    <title>500 Internal Server Error</title>\\n    <style ty
pe="text/css">\\n    #powered_by {\\n        margin-top: 20px;\\n        border-top: 2px solid black;\\n        font-style: italic;\\n    }\\n\\n    #traceback {\\n        color: red;\\n    }\\n    </style>\\n</head>\\n    <body>\\n        <h2>500 Int
ernal Server Error</h2>\\n        <p>Server error</p>\\n        <pre id="traceback"></pre>\\n    <div id="powered_by">\\n      <span>\\n        Powered by <a href="http://www.cherrypy.org">CherryPy 17.4.0</a>\\n      </span>\\n    </div>\\n    </
body>\\n</html>\\n'
2021-08-27 17:36:10,609:139922251294464:WARNING:ReqMgrAux:agent config is not correct: {}
2021-08-27 17:36:10,611:139922251294464:INFO:ErrorHandlerPoller:Found 0 failed jobs in state createfailed
2021-08-27 17:36:10,612:139922251294464:INFO:ErrorHandlerPoller:Found 0 failed jobs in state submitfailed
2021-08-27 17:36:10,613:139922251294464:INFO:ErrorHandlerPoller:Found 110 failed jobs in state jobfailed
2021-08-27 17:36:10,676:139922251294464:INFO:ErrorHandlerPoller:About to process 110 failures
2021-08-27 17:36:10,677:139922251294464:INFO:ErrorHandlerPoller:Processing retries for 110 failed jobs of type jobfailed
2021-08-27 17:36:10,677:139922251294464:ERROR:ErrorHandlerPoller:Caught unexpected exception in ErrorHandler:
'<' not supported between instances of 'int' and 'NoneType'
Traceback (most recent call last):
  File "/data/srv/wmagent/v1.5.2/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.5.2/lib/python3.8/site-packages/WMComponent/ErrorHandler/ErrorHandlerPoller.py", line 402, in algorithm
    self.handleErrors()
  File "/data/srv/wmagent/v1.5.2/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.5.2/lib/python3.8/site-packages/WMComponent/ErrorHandler/ErrorHandlerPoller.py", line 329, in handleErrors
    self.handleFailedJobs(jobList, state)
  File "/data/srv/wmagent/v1.5.2/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.5.2/lib/python3.8/site-packages/WMComponent/ErrorHandler/ErrorHandlerPoller.py", line 312, in handleFailedJobs
    self.processRetries(jobList, state)
  File "/data/srv/wmagent/v1.5.2/sw/slc7_amd64_gcc630/cms/wmagentpy3/1.5.2/lib/python3.8/site-packages/WMComponent/ErrorHandler/ErrorHandlerPoller.py", line 173, in processRetries
    if job['retry_count'] < allowedRetries and state != 'create':
TypeError: '<' not supported between instances of 'int' and 'NoneType'
...
```


Changed files:
src/python/WMComponent/ErrorHandler/ErrorHandlerPoller.py

diff --git a/src/python/WMComponent/ErrorHandler/ErrorHandlerPoller.py b/src/python/WMComponent/ErrorHandler/ErrorHandlerPoller.py
index bbd9b4a0e..1da590d00 100644
--- a/src/python/WMComponent/ErrorHandler/ErrorHandlerPoller.py
+++ b/src/python/WMComponent/ErrorHandler/ErrorHandlerPoller.py
@@ -96,11 +96,14 @@ class ErrorHandlerPoller(BaseWorkerThread):
         """
         Initialize (and update every cycle) some of the component's
         parameters, according to the agent type (T0/Production) and agent config.
+        :return: True if the setup was completed, False if any non-critical problem happened
         """
         if self.reqAuxDB:
             agentConfig = self.reqAuxDB.getWMAgentConfig(self.config.Agent.hostName)
-            self.exitCodesNoRetry = agentConfig.get("NoRetryExitCodes", [])
+            if not agentConfig:
+                return False
 
+            self.exitCodesNoRetry = agentConfig.get("NoRetryExitCodes", [])
             if agentConfig.get("UserDrainMode") and agentConfig.get("SpeedDrainMode") \\
                 and agentConfig.get("SpeedDrainConfig")['NoJobRetries']['Enabled']:
                 logging.info("Agent is in speed drain mode. Not retrying any jobs.")
@@ -112,6 +115,7 @@ class ErrorHandlerPoller(BaseWorkerThread):
             self.maxRetries = {'default': self.maxRetries}
         if 'default' not in self.maxRetries:
             raise ErrorHandlerException('Max retries for the default job type must be specified')
+        return True
 
     def setup(self, parameters=None):
         """
@@ -395,8 +399,7 @@ class ErrorHandlerPoller(BaseWorkerThread):
         And deal with it as desired.
         """
         logging.debug("Running error handling algorithm")
-        self.setupComponentParam()
-        if not self.maxRetries:
+        if not self.setupComponentParam():
             msg = "Component failed to retrieve agent configuration from central ReqMgr Aux DB."
             msg += " Skipping this cycle."
             logging.error(msg)