Repository: dmwm/wmcore
Issue: MSTransferor: rule ids for INJECT/REPLICATING rules must be passed to MSMonitor

**Impact of the bug**
MSTransferor

**Describe the bug**
Hasan reported a workflow that was bypassed straight to `staging` in MSTransferor, and the reason for that is that its pileup dataset already had an input data rule in status REPLICATING. In such cases, we simply consider data to be on the way and advance the workflow to the next status (staging), without keeping record of the rule_ids still unsatisfied. Which causes the same workflow(s) to be bypassed by MSMonitor as well. 

If the workflow is marked as `TrustPUSitelists=true`, then it would be bypassed by global workqueue as well, which could result in many jobs failing to read secondary data in case not all files are available at any given Disk endpoint.

**How to reproduce it**
Assign a workflow with pileup - with secondary AAA enabled - where the pileup only has rules in INJECT or REPLICATING status. 

**Expected behavior**
I think the best way to address it would be:
* if input rule is in status INJECT or REPLICATING, we need to keep that/those rule_id and persist it in the transfer document
* in terms of MSTransferor, we do not require any new input rules for such dataset and allow it to move to `staging`, in case all of the other requirements are met.

**Additional context and error message**
Some context can be found at: https://github.com/dmwm/WMCore/pull/10041

Here is an example: https://cmsweb.cern.ch/ms-transferor/data/info?request=cmsunified_task_TSG-Run3Summer21wmLHEGS-00035__v1_T_220309_215550_5271


Changed files:
src/python/WMCore/MicroService/Tools/PycurlRucio.py

diff --git a/src/python/WMCore/MicroService/Tools/PycurlRucio.py b/src/python/WMCore/MicroService/Tools/PycurlRucio.py
index 7756ad4a1..362215a8b 100644
--- a/src/python/WMCore/MicroService/Tools/PycurlRucio.py
+++ b/src/python/WMCore/MicroService/Tools/PycurlRucio.py
@@ -164,6 +164,8 @@ def listReplicationRules(containers, rucioAccount, grouping,
     :return: a flat dictionary key'ed by the container name, with a list of RSE
       expressions that still need to be resolved
     NOTE: Value `None` is returned in case the data-service failed to serve a given request.
+    NOTE-2: Available rule states can be found at:
+    https://github.com/rucio/rucio/blob/16f39dffa1608caa0a1af8bbc0fcff2965dccc50/lib/rucio/db/sqla/constants.py#L180
     """
     locationByContainer = {}
     if not containers:
@@ -189,8 +191,9 @@ def listReplicationRules(containers, rucioAccount, grouping,
         try:
             locationByContainer.setdefault(container, [])
             for item in parseNewLineJson(row['data']):
-                if item['state'] in ["U", "SUSPENDED"]:
-                    logging.warning("Container %s has a SUSPENDED rule. Skipping rule: %s", container, item)
+                if item['state'] in ["U", "SUSPENDED", "R", "REPLICATING", "I", "INJECT"]:
+                    msg = "Container %s has a rule ID %s in state %s. Will try to create a new rule."
+                    logging.warning(msg, container, item['id'], item['state'])
                     continue
                 elif item['state'] in ["S", "STUCK"]:
                     if item['error'] == 'NO_SOURCES:NO_SOURCES':
@@ -316,10 +319,6 @@ def getBlocksAndSizeRucio(containers, rucioUrl, rucioToken, scope="cms"):
     headers = {"X-Rucio-Auth-Token": rucioToken}
     urls = []
     for cont in containers:
-        ### FIXME: the long attribute value type has recently changed integer to boolean
-        ### see PR: https://github.com/rucio/rucio/pull/3949 , which went in in 1.23.5 series
-        ### we need to make sure CMS production Rucio will be running that version once MicroServices
-        ### get deployed to CMSWEB
         urls.append('{}/dids/{}/dids/search?type=dataset&long=True&name={}'.format(rucioUrl, scope, quote(cont + "#*")))
     logging.info("Executing %d requests against Rucio DIDs search API for containers", len(urls))
     data = multi_getdata(urls, ckey(), cert(), headers=headers)