Repository: dmwm/wmcore
Issue: DBS3Upload crash if DNS fails to resolve cmsweb

**Impact of the bug**
WMAgent (DBS3Upload, but can likely affect other components)

**Describe the bug**
If for whatever reason - like a CMSWEB deployment - the components cannot resolve central services (cmsweb / cmsweb-testbed), it could be that the error is not yet handled, causing the component to terminate.

**How to reproduce it**
none

**Expected behavior**
Instead of crashing the component, we should catch that exception/error message, rollback any active transactions and skip that cycle of the component (and of course make an ERROR log record in the logs), waiting for the standard execution of the next cycle.
 
**Additional context and error message**
`DBS3Upload` component went down in every single testbed agent this morning. Here is the traceback from vocms0193:
```
2020-01-14 11:18:15,440:140300540999424:INFO:DBSUploadPoller:Starting the DBSUpload Polling Cycle
2020-01-14 11:18:15,470:140300540999424:WARNING:DBSUploadPoller:Failed to fetch parentage map from WMStats, skipping this cycle. Exception: error. Reason: . Error: (7, 'Failed to connect to cmsweb-testbed.cern.ch port 8443: Connection r
efused')
2020-01-14 11:18:15,477:140300540999424:ERROR:LogDB:LogDBBackend post API failed, error=(7, 'Failed to connect to cmsweb-testbed.cern.ch port 443: Connection refused')
2020-01-14 11:18:15,478:140300540999424:INFO:BaseWorkerThread:DBSUploadPoller took 0.037 secs to execute
2020-01-14 11:18:15,481:140300540999424:ERROR:LogDB:LogDBBackend delete API failed, error=(7, 'Failed to connect to cmsweb-testbed.cern.ch port 443: Connection refused')
2020-01-14 11:19:55,603:140300540999424:ERROR:LogDB:LogDBBackend delete API failed, error=(6, 'Could not resolve: cmsweb-testbed.cern.ch (DNS server returned answer with no data)')
2020-01-14 11:19:55,604:140300540999424:INFO:DBSUploadPoller:Starting the DBSUpload Polling Cycle
2020-01-14 11:19:57,404:140300540999424:ERROR:BaseWorkerThread:Error in worker algorithm (1):
Backtrace:
  <WMComponent.DBS3Buffer.DBSUploadPoller.DBSUploadPoller object at 0x7f9a4b40b250> <@========== WMException Start ==========@>
Exception Class: DBSUploadException
Message: Unknown failure while fetching parentage map from WMStats. Error: (6, 'Could not resolve: cmsweb-testbed.cern.ch (DNS server returned answer with no data)')
        ModuleName : WMComponent.DBS3Buffer.DBSUploadPoller
        MethodName : updateDatasetParentageCache
        ClassInstance : None
        FileName : /data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py
        ClassName : None
        LineNumber : 356
        ErrorNr : 0

Traceback: 
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 342, in updateDatasetParentageCache
    self.datasetParentageCache = self.wmstatsServerSvc.getChildParentDatasetMap()

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/WMStatsServer/WMStatsServer.py", line 105, in getChildParentDatasetMap
    results = self.getFilteredActiveData(filter, mask)

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/WMStatsServer/WMStatsServer.py", line 92, in getFilteredActiveData
    return self._getResult(callname, verb="GET")

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/WMStatsServer/WMStatsServer.py", line 43, in _getResult
    verb=verb, contentType=contentType)

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/Service.py", line 205, in refreshCache
    self.getData(cachefile, url, inputdata, incoming_headers, encoder, decoder, verb, contentType)

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/Service.py", line 284, in getData
    contentType=contentType)

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/Requests.py", line 165, in makeRequest
    result, response = self.makeRequest_pycurl(uri, data, verb, headers)

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/Requests.py", line 183, in makeRequest_pycurl
    ckey=ckey, cert=cert, capath=capath)

  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Services/pycurl_manager.py", line 257, in request
    curl.perform()

<@---------- WMException End ----------@>  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/WorkerThreads/BaseWorkerThread.py", line 182, in __call__
    tSpent, results, _ = algorithmWithDBExceptionHandler(parameters)
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMCore/Database/DBExceptionHandler.py", line 39, in wrapper
    return f(*args, **kwargs)
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/Utils/Timers.py", line 24, in wrapper
    res = func(*arg, **kw)
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 311, in algorithm
    if self.updateDatasetParentageCache() is False:
  File "/data/srv/wmagent/v1.2.8/sw/slc7_amd64_gcc630/cms/wmagent/1.2.8/lib/python2.7/site-packages/WMComponent/DBS3Buffer/DBSUploadPoller.py", line 356, in updateDatasetParentageCache
    raise DBSUploadException(msg)

2020-01-14 11:19:57,404:140300540999424:INFO:Harness:>>>Terminating worker threads
2020-01-14 11:19:57,405:140300540999424:ERROR:BaseWorkerThread:Error in event loop (2): <WMComponent.DBS3Buffer.DBSUploadPoller.DBSUploadPoller object at 0x7f9a4b40b250> <@========== WMException Start ==========@>
```

Changed files:
src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py

diff --git a/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py b/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
index 57d62f6a4..09771a1cb 100644
--- a/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
+++ b/src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py
@@ -347,9 +347,7 @@ class DBSUploadPoller(BaseWorkerThread):
             if 'Service Unavailable' in reason or 'Proxy Error' in reason or \\
                             'Error reading from remote server' in reason:
                 pass
-            elif 'Connection refused' in str(ex):
-                msg += 'Error: %s' % str(ex)
-            elif 'timed out' in str(ex):
+            elif 'Connection refused' in str(ex) or 'timed out' in str(ex) or 'Could not resolve' in str(ex):
                 msg += 'Error: %s' % str(ex)
             else:
                 msg = "Unknown failure while fetching parentage map from WMStats. Error: %s" % str(ex)