Repository: dmwm/wmcore
Issue: Fix StepChain parentage for workflows in closed-out as well

**Impact of the new feature**
WMAgent & Unified

**Is your feature request related to a problem? Please describe.**
Given the asynchronous nature of merge execution in StepChain workflows, it could be that its output datasets get announced before all the parentage information has been updated.
The Parentage dependency is resolved through a ReqMgr2 cherrypy thread, which queries DBS, asking for every block in a dataset with no Parent block : [1]. Those are extremely heavy queries to the DBS database and sometimes due to external pressure to the system can lead to delays. When those delays exceed the 5 min timeout (configured in the cmsweb Frontend) the current query is dropped and the Parentage resolution of the current block is retried during the next cycle. This by its own means  can delay the process even further, while in the meantime Unified can see that  the needed statistics for a workflow is reached and announce the data. 
There is already a flag in the request description `ParentageResolved` which is used to sign StepChains with complete parentage information.
We need to fix parentage before it gets announced (closed-out status in ReqMgr2). Set that flag to `True`, and then propagated and take it into account by Unified. 

[1]
1. Given a dataset, find its parent datasets
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/DBS/DBS3Reader.py#L870

2. Find all the blocks in the dataset that do not have a parent block:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/DBS/DBS3Reader.py#L968

3. Given a block name, list files parents by lumi:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Services/DBS/DBS3Reader.py#L953

[2]
https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=amaltaro_SC_LumiMask_PhEDEx_HG2004_Val_200330_093128_1213

**Describe the solution you'd like**
We should enable the `closed-out` status in the `StepChainParentageFixTask` cherrypy thread, such that Unified can check the `ParentageResolved` value before announcing StepChain workflows.
 
**Describe alternatives you've considered**
Other alternatives  are to  have WMAgent to only publish full information to DBS - this requiters long term development plans and redesign of portions of the WMAgent

**Additional context**
There is a related issue in Unified:
https://github.com/CMSCompOps/WmAgentScripts/issues/545


Changed files:
src/python/WMCore/REST/CherryPyPeriodicTask.py, src/python/WMCore/ReqMgr/CherryPyThreads/StepChainParentageFixTask.py, src/python/WMCore/Services/DBS/DBS3Reader.py

diff --git a/src/python/WMCore/REST/CherryPyPeriodicTask.py b/src/python/WMCore/REST/CherryPyPeriodicTask.py
index 9a3496953..9c31e0ff3 100644
--- a/src/python/WMCore/REST/CherryPyPeriodicTask.py
+++ b/src/python/WMCore/REST/CherryPyPeriodicTask.py
@@ -25,6 +25,7 @@ class CherryPyPeriodicTask(object):
         TODO: add validation for config.duration
         """
         self.logger = getTimeRotatingLogger(config._internal_name, config.log_file)
+        self.logger.info("Setting CherryPy periodic task with the following config:\\n%s", config)
         self.setConcurrentTasks(config)
         self.setUpLogDB(config)
 
diff --git a/src/python/WMCore/ReqMgr/CherryPyThreads/StepChainParentageFixTask.py b/src/python/WMCore/ReqMgr/CherryPyThreads/StepChainParentageFixTask.py
index 6dd64699e..93dabae6e 100644
--- a/src/python/WMCore/ReqMgr/CherryPyThreads/StepChainParentageFixTask.py
+++ b/src/python/WMCore/ReqMgr/CherryPyThreads/StepChainParentageFixTask.py
@@ -35,7 +35,7 @@ class StepChainParentageFixTask(CherryPyPeriodicTask):
         super(StepChainParentageFixTask, self).__init__(config)
         self.reqmgrDB = RequestDBWriter(config.reqmgrdb_url)
         self.dbsSvc = DBS3Reader(config.dbs_url, logger=self.logger)
-        self.statusToCheck = ["announced", "normal-archived"]
+        self.statusToCheck = ["closed-out", "announced", "normal-archived"]
 
     def setConcurrentTasks(self, config):
         """
@@ -48,12 +48,14 @@ class StepChainParentageFixTask(CherryPyPeriodicTask):
         Look through the stepchain workflows with ParentageResolved flag is False.
         Fix the StepChain parentage and update the ParentageResolved flag to True
         """
-        self.logger.info("Updating parentage for StepChain workflows for %s", self.statusToCheck)
+        self.logger.info("Running fixStepChainParentage thread for statuses: %s", self.statusToCheck)
         childDatasets = set()
         requests = set()
         requestsByChildDataset = {}
         for status in self.statusToCheck:
             reqByChildDS= getChildDatasetsForStepChainMissingParent(self.reqmgrDB, status)
+            self.logger.info("Retrieved %d datasets to fix parentage, in status: %s",
+                             len(reqByChildDS), status)
             childDatasets = childDatasets.union(set(reqByChildDS.keys()))
             # We need to just get one of the StepChain workflow if multiple workflow contains the same datasets. (i.e. ACDC)
             requestsByChildDataset.update(reqByChildDS)
@@ -65,6 +67,7 @@ class StepChainParentageFixTask(CherryPyPeriodicTask):
         totalChildDS = len(childDatasets)
         fixCount = 0
         for childDS in childDatasets:
+            self.logger.info("Resolving parentage for dataset: %s", childDS)
             start = int(time.time())
             failedBlocks = self.dbsSvc.fixMissingParentageDatasets(childDS, insertFlag=True)
             end = int(time.time())
@@ -84,4 +87,4 @@ class StepChainParentageFixTask(CherryPyPeriodicTask):
             self.reqmgrDB.updateRequestProperty(request, {"ParentageResolved": True})
 
         self.logger.info("Total %s requests' ParentageResolved flag is set to True", len(requestsToUpdate))
-        self.logger.info("Total %s requests will be retried next cycle: %s", len(failedRequests), failedRequests)
\\ No newline at end of file
+        self.logger.info("Total %s requests will be retried next cycle: %s", len(failedRequests), failedRequests)
diff --git a/src/python/WMCore/Services/DBS/DBS3Reader.py b/src/python/WMCore/Services/DBS/DBS3Reader.py
index 8be8c06cd..7f395776f 100644
--- a/src/python/WMCore/Services/DBS/DBS3Reader.py
+++ b/src/python/WMCore/Services/DBS/DBS3Reader.py
@@ -1008,6 +1008,7 @@ class DBS3Reader(object):
         :return: blocks which failed to insert parentage. for retry
         """
         pDatasets = self.listDatasetParents(childDataset)
+        self.logger.info("Parent datasets for %s are: %s", childDataset, pDatasets)
         # print("parent datasets %s\\n" % pDatasets)
         # pDatasets format is
         # [{'this_dataset': '/SingleMuon/Run2016D-03Feb2017-v1/MINIAOD', 'parent_dataset_id': 13265209, 'parent_dataset': '/SingleMuon/Run2016D-23Sep2016-v1/AOD'}]
@@ -1017,6 +1018,7 @@ class DBS3Reader(object):
 
         blocks = self.listBlocksWithNoParents(childDataset)
         failedBlocks = []
+        self.logger.info("Found %d blocks without parentage information", len(blocks))
         for blockName in blocks:
             try:
                 numFiles = self.findAndInsertMissingParentage(blockName, insertFlag=insertFlag)