Repository: dmwm/wmcore
Issue: ReqMgr2 datatier validation needs to be cached

**Impact of the bug**
ReqMgr2

**Describe the bug**
Sharad reported issues noticed in the Unified logs while assigning workflows, which fails to get a workflow into assigned state because it does not pass the datatier validation check, here:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/ReqMgr/Utils/Validation.py#L253
It happens every once in a while (maybe when we hit a misbehaving DBS backend(?)).

**How to reproduce it**
I didn't manage to reproduce it, but one possible way would be to source the agent/central services init.sh and run:
```
from WMCore.Services.DBS.DBS3Reader import DBS3Reader as DBSReader
dbsUrl = "https://cmsweb.cern.ch/dbs/prod/global/DBSReader/"
dbsTiers = DBSReader.listDatatiers(dbsUrl)
```

**Expected behavior**
We should reuse the same DBSReader object, and use the Services cache for at least 1h of duration.

**Additional context and error message**
Error message/traceback from reqmgr2 logs:
```
[09/Aug/2019:10:46:19]  Assign request pdmvserv_task_EXO-RunIIAutumn18NanoAODv5-01182__v1_T_190613_115143_3503, input args: {u'SoftTimeout': 159600, u'BlockCloseMaxEvents': 200000000, u'MaxMergeEvents': 20000000, u'CustodialSites': [],
 u'ProcessingString': {u'EXO-RunIIAutumn18NanoAODv5-01182_0': u'Nano1June2019_102X_upgrade2018_realistic_v19'}, u'TrustSitelists': False, u'UnmergedLFNBase': '/store/unmerged', u'CustodialSubType': 'Replica', u'TrustPUSitelists': False
, u'GracePeriod': 300, u'MergedLFNBase': '/store/mc', 'HardTimeout': 159900, u'SiteWhitelist': [u'T2_BR_SPRACE'], u'AcquisitionEra': {u'EXO-RunIIAutumn18NanoAODv5-01182_0': u'RunIIAutumn18NanoAODv5'}, u'NonCustodialSubType': 'Replica',
 u'NonCustodialSites': [u'T2_BR_SPRACE'], u'MinMergeSize': 2147483648, u'BlockCloseMaxWaitTime': 172800, u'MaxMergeSize': 4294967296, u'NonCustodialGroup': 'DataOps', u'ProcessingVersion': 1, u'Team': 'production', u'AutoApproveSubscri
ptionSites': [u'T2_BR_SPRACE'], u'SiteBlacklist': [], u'RequestStatus': 'assigned', u'Dashboard': 'production'} ...
[09/Aug/2019:10:46:19]  SERVER REST ERROR WMCore.ReqMgr.DataStructs.RequestError.InvalidSpecParameterValue dca79f8d419df7cf16ef0a63be0b169e (Invalid spec parameter value: Bad datatier(s): ['NANOAODSIM'] not available in DBS.)
[09/Aug/2019:10:46:19]    Traceback (most recent call last):
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/REST/Server.py", line 723, in default
[09/Aug/2019:10:46:19]        return self._call(RESTArgs(list(args), kwargs))
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/REST/Server.py", line 804, in _call
[09/Aug/2019:10:46:19]        obj = apiobj['call'](*safe.args, **safe.kwargs)
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 493, in put
[09/Aug/2019:10:46:19]        result = self._updateRequest(workload, request_args)
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 476, in _updateRequest
[09/Aug/2019:10:46:19]        report = self._handleAssignmentStateTransition(workload, request_args, dn)
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 428, in _handleAssignmentStateTransition
[09/Aug/2019:10:46:19]        validateOutputDatasets(request_args['OutputDatasets'], workload.getDbsUrl())
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/ReqMgr/Utils/Validation.py", line 250, in validateOutputDatasets
[09/Aug/2019:10:46:19]        _validateDatatier(datatier, dbsUrl)
[09/Aug/2019:10:46:19]      File "/data/srv/beHG1908e/sw/slc7_amd64_gcc630/cms/reqmgr2/1.2.5.pre7/lib/python2.7/site-packages/WMCore/ReqMgr/Utils/Validation.py", line 263, in _validateDatatier
[09/Aug/2019:10:46:19]        raise InvalidSpecParameterValue("Bad datatier(s): %s not available in DBS." % badTiers)
[09/Aug/2019:10:46:19]    InvalidSpecParameterValue: InvalidSpecParameterValue dca79f8d419df7cf16ef0a63be0b169e [HTTP 400, APP 1102, MSG "Invalid spec parameter value: Bad datatier(s): ['NANOAODSIM'] not available in DBS.", INFO None, ERR None]
[09/Aug/2019:10:46:19] vocms0136.cern.ch 2001:1458:d00:18::a8 "PUT /reqmgr2/data/request/pdmvserv_task_EXO-RunIIAutumn18NanoAODv5-01182__v1_T_190613_115143_3503 HTTP/1.1" 400 Bad Request [data: 7394 in 793 out 134528 us ] [auth: OK "/DNblah blah" "" ] [ref: "" "" ]
```

Changed files:
src/python/WMCore/ReqMgr/Utils/Validation.py, test/python/WMCore_t/Cache_t/GenericDataCache_t.py, src/python/WMCore/Cache/GenericDataCache.py

diff --git a/src/python/WMCore/Cache/GenericDataCache.py b/src/python/WMCore/Cache/GenericDataCache.py
index 1e6d303d6..723b0cb27 100644
--- a/src/python/WMCore/Cache/GenericDataCache.py
+++ b/src/python/WMCore/Cache/GenericDataCache.py
@@ -84,3 +84,12 @@ class GenericDataCache(object):
             raise CacheWithWrongStructException(cacheName)
         else:
             GenericDataCache._dataCache[cacheName] = memoryCache
+
+    @staticmethod
+    def cacheExists(cacheName):
+        """
+        Return True if provided cache name is already cached, else False.
+        :param cacheName: cache name string
+        :return: boolean
+        """
+        return cacheName in GenericDataCache._dataCache
diff --git a/src/python/WMCore/ReqMgr/Utils/Validation.py b/src/python/WMCore/ReqMgr/Utils/Validation.py
index d581e907d..776d6dce8 100644
--- a/src/python/WMCore/ReqMgr/Utils/Validation.py
+++ b/src/python/WMCore/ReqMgr/Utils/Validation.py
@@ -3,6 +3,7 @@ ReqMgr request handling.
 
 """
 from __future__ import print_function
+from retry import retry
 
 from WMCore.Lexicon import procdataset
 from WMCore.REST.Auth import authz_match
@@ -15,7 +16,7 @@ from WMCore.Services.DBS.DBS3Reader import DBS3Reader as DBSReader
 from WMCore.WMFactory import WMFactory
 from WMCore.WMSpec.WMWorkload import WMWorkloadHelper
 from WMCore.WMSpec.WMWorkloadTools import loadSpecClassByType, setArgumentsWithDefault
-
+from WMCore.Cache.GenericDataCache import GenericDataCache, MemoryCacheStruct
 
 def workqueue_stat_validation(request_args):
     stat_keys = ['total_jobs', 'input_lumis', 'input_events', 'input_num_files']
@@ -250,14 +251,29 @@ def validateOutputDatasets(outDsets, dbsUrl):
     _validateDatatier(datatier, dbsUrl)
 
 
-def _validateDatatier(datatier, dbsUrl):
+@retry(tries=5, delay=1)
+def _listDatatiers(dbsUrl):
+    """
+    Query DBS for list of data tiers.
+    If exception is raised, retry up to 5 times with a 1 second delay.
+    """
+    dbsTiers = DBSReader.listDatatiers(dbsUrl)
+    return dbsTiers
+
+def _validateDatatier(datatier, dbsUrl, expiration=3600):
     """
     _validateDatatier_
 
     Provided a list of datatiers extracted from the outputDatasets, checks
     whether they all exist in DBS.
     """
-    dbsTiers = DBSReader.listDatatiers(dbsUrl)
+    cacheName = "dataTierList"
+    if not GenericDataCache.cacheExists(cacheName):
+        mc = MemoryCacheStruct(expiration, _listDatatiers, kwargs={'dbsUrl': dbsUrl})
+        GenericDataCache.registerCache(cacheName, mc)
+
+    cacheData = GenericDataCache.getCacheData('dataTierList')
+    dbsTiers = cacheData.getData()
     badTiers = list(set(datatier) - set(dbsTiers))
     if badTiers:
         raise InvalidSpecParameterValue("Bad datatier(s): %s not available in DBS." % badTiers)
diff --git a/test/python/WMCore_t/Cache_t/GenericDataCache_t.py b/test/python/WMCore_t/Cache_t/GenericDataCache_t.py
index 5158f1acb..f79192db1 100644
--- a/test/python/WMCore_t/Cache_t/GenericDataCache_t.py
+++ b/test/python/WMCore_t/Cache_t/GenericDataCache_t.py
@@ -62,6 +62,16 @@ class GenericDataCacheTest(unittest.TestCase):
 
         return
 
+    def testExists(self):
+        """
+        Tests whether a given cache already exists
+        """
+        mc = MemoryCacheStruct(1, lambda x: int(time.time()), kwargs={'x':1})
+        self.assertFalse(GenericDataCache.cacheExists("tCache"))
+
+        GenericDataCache.registerCache("tCache", mc)
+        self.assertTrue(GenericDataCache.cacheExists("tCache"))
+        self.assertFalse(GenericDataCache.cacheExists("tCache2"))
 
 if __name__ == "__main__":
     unittest.main()