Repository: dmwm/wmcore
Issue: ReqMon DataCacheUpdate thread timing out

**Impact of the bug**
ReqMon (aka WMStats)

**Describe the bug**
Likely related to the very high load in the system, as explained here: https://github.com/dmwm/WMCore/issues/9668

Over the last days, WMStats became very sluggish and poking around the logs, I see many timeouts in the `DataCacheUpdate` reqmon cherrypy thread. It seems to have become much worse since yesterday.

From the logs, there are cases that it has to transfer ~200MB worth of data. Given that this is a general purpose, one can multiply that by 5 (1GB of data every 10 to 15min...). It also explains why wmstats sometimes loads and others it remains spinning forever.

**How to reproduce it**
Have 40k workflows in the system with many job failures.

**Expected behavior**
The obvious ones. Cache should always be up-to-date, thus avoiding these requests time out.
These caches were supposed to be "small" as well (<50MB), having such large caches continuously updated in many backends isn't health for the cluster.
We could also calculate the amount of data in the cache and log it (pushing such information to MonIT would be a plus as well).
Anything else that can still be discussed to make this service more robust and resilient to such a high load.
 
**Additional context and error message**
Logs from vocms0766 for today:
```
2020-05-07 02:54:41,440:INFO:DataCacheUpdate:DataCache is updated: 42096
2020-05-07 03:12:55,214:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270001 milliseconds with 115605922 bytes received')
2020-05-07 03:29:13,663:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270001 milliseconds with 53544228 bytes received')
2020-05-07 03:45:05,224:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 68165469 bytes received')
2020-05-07 04:01:29,324:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 225706992 bytes received')
2020-05-07 04:18:01,356:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270008 milliseconds with 121591632 bytes received')
2020-05-07 04:34:01,814:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 23328350 bytes received')
2020-05-07 04:51:40,050:ERROR:DataCacheUpdate:(28, 'Operation timed out after 271232 milliseconds with 22150520 bytes received')
2020-05-07 05:08:50,854:ERROR:DataCacheUpdate:(28, 'Operation timed out after 280071 milliseconds with 20950855 bytes received')
2020-05-07 05:25:37,535:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 54860085 bytes received')
2020-05-07 05:42:15,440:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 62670518 bytes received')
2020-05-07 05:58:33,363:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 122496096 bytes received')
2020-05-07 06:15:27,999:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 63965610 bytes received')
2020-05-07 06:31:27,804:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 213083083 bytes received')
2020-05-07 06:48:05,471:ERROR:DataCacheUpdate:(28, 'Operation timed out after 276537 milliseconds with 215823928 bytes received')
2020-05-07 07:04:44,183:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 25620037 bytes received')
2020-05-07 07:20:36,991:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270000 milliseconds with 99517239 bytes received')
2020-05-07 07:37:25,040:ERROR:DataCacheUpdate:(28, 'Operation timed out after 270001 milliseconds with 113712109 bytes received')
2020-05-07 07:53:46,004:INFO:DataCacheUpdate:DataCache is updated: 42098
```


Changed files:
src/python/WMCore/Services/WMStats/WMStatsReader.py, src/python/WMCore/Services/WMStats/WMStatsWriter.py

diff --git a/src/python/WMCore/Services/WMStats/WMStatsReader.py b/src/python/WMCore/Services/WMStats/WMStatsReader.py
index 2f5821017..cf94265e8 100644
--- a/src/python/WMCore/Services/WMStats/WMStatsReader.py
+++ b/src/python/WMCore/Services/WMStats/WMStatsReader.py
@@ -318,7 +318,7 @@ class WMStatsReader(object):
 
         options = {"group_level": 1, "reduce": True}
 
-        results = self.couchDB.loadView(self.couchapp, "allWorkflows", options=options)['rows']
+        results = self._getCouchView("allWorkflows", options)['rows']
         requestNames = [x['key'] for x in results]
 
         workflowDict = self.reqDB.getStatusAndTypeByRequest(requestNames)
@@ -345,7 +345,7 @@ class WMStatsReader(object):
 
         options = {'reduce': True, 'group_level': 5, 'startkey': [requestName],
                    'endkey': [requestName, {}]}
-        results = self.couchDB.loadView(self.couchapp, "jobsByStatusWorkflow", options=options)
+        results = self._getCouchView("jobsByStatusWorkflow", options)
         jobDetails = {}
         for row in results['rows']:
             # row["key"] = ['workflow', 'task', 'jobstatus', 'exitCode', 'site']
@@ -367,7 +367,7 @@ class WMStatsReader(object):
         options = {'include_docs': True, 'reduce': False,
                    'startkey': startKey, 'endkey': endKey,
                    'limit': limit}
-        result = self.couchDB.loadView(self.couchapp, "jobsByStatusWorkflow", options=options)
+        result = self._getCouchView("jobsByStatusWorkflow", options)
         jobInfoDoc = {}
         for row in result['rows']:
             keys = row['key']
@@ -391,7 +391,7 @@ class WMStatsReader(object):
 
     def getAllAgentRequestRevByID(self, agentURL):
         options = {"reduce": False}
-        results = self.couchDB.loadView(self.couchapp, "byAgentURL", options=options, keys=[agentURL])
+        results = self._getCouchView("byAgentURL", options, keys=[agentURL])
         idRevMap = {}
         for row in results['rows']:
             idRevMap[row['id']] = row['value']['rev']
diff --git a/src/python/WMCore/Services/WMStats/WMStatsWriter.py b/src/python/WMCore/Services/WMStats/WMStatsWriter.py
index e33cc1cb4..7a280a1f8 100644
--- a/src/python/WMCore/Services/WMStats/WMStatsWriter.py
+++ b/src/python/WMCore/Services/WMStats/WMStatsWriter.py
@@ -188,7 +188,7 @@ class WMStatsWriter(WMStatsReader):
         threshold = int(time.time()) - sec
         options = {"startkey": threshold, "descending": True,
                    "stale": "update_after"}
-        result = self.couchDB.loadView(self.couchapp, "time", options)
+        result = self._getCouchView("time", options)
 
         for row in result['rows']:
             doc = {}
@@ -261,9 +261,8 @@ class WMStatsWriter(WMStatsReader):
         """
         delete all wmstats docs for a given requestName
         """
-        view = "allWorkflows"
-        options = {"key": requestName, "reduce": False}
-        docs = self.couchDB.loadView(self.couchapp, view, options=options)['rows']
+        options = {"reduce": False, "key": requestName}
+        docs = self._getCouchView("allWorkflows", options)['rows']
 
         for j in docs:
             doc = {}