Repository: dmwm/wmcore
Issue: Only create rules for blocks that have already been inserted into Rucio

**Impact of the bug**
WMAgent (RucioInjectorPoller)

**Describe the bug**
While scanning the RucioInjectorPoller component logs, I noticed many errors popping from replication rules creation.
From a quick look at the logs, I have the impression the component first loads data from the database, then it tries to create a block rule, and only then it creates that block/files in Rucio.

Likely an unnoticed side effect of: https://github.com/dmwm/WMCore/pull/10067

**How to reproduce it**
Likely reproducible with usual business

**Expected behavior**
RucioInjectorPoller should create block rules only if the block has already been injected into Rucio server.
Same applies to container level rules, if we can hit the same issue with those...

**Additional context and error message**
A snippet of the RucioInjectorPoller log from vocms0254 (check the timestamp):
```
2020-12-11 07:25:12,404:140038718158592:ERROR:Rucio:Exception creating rule replica for data: ['/DY3JetsToLL_M-50_TuneCP5_13TeV-madgraphMLM-pythia8/RunIISummer19UL17NanoAODv2-106X_mc2017_realistic_v8-v1/NANOAODSIM#27f0b17c-be0a-44c0-a996-15ec820b6175']. Error: Data identifier not found.
Details: Data identifier cms:/DY3JetsToLL_M-50_TuneCP5_13TeV-madgraphMLM-pythia8/RunIISummer19UL17NanoAODv2-106X_mc2017_realistic_v8-v1/NANOAODSIM#27f0b17c-be0a-44c0-a996-15ec820b6175 is not valid.
2020-12-11 07:25:12,404:140038718158592:ERROR:RucioInjectorPoller:Failed to create rule for block: /DY3JetsToLL_M-50_TuneCP5_13TeV-madgraphMLM-pythia8/RunIISummer19UL17NanoAODv2-106X_mc2017_realistic_v8-v1/NANOAODSIM#27f0b17c-be0a-44c0-a996-15ec820b6175 at T2_UK_London_IC
2020-12-11 07:25:12,404:140038718158592:INFO:RucioInjectorPoller:Checking if there are block rules to be deleted...
2020-12-11 07:25:12,407:140038718158592:INFO:RucioInjectorPoller:No candidate blocks found for rule deletion
2020-12-11 07:25:12,407:140038718158592:INFO:BaseWorkerThread:RucioInjectorPoller took 98.819 secs to execute
2020-12-11 07:30:12,827:140038718158592:INFO:RucioInjectorPoller:Running Rucio injector poller algorithm...
2020-12-11 07:30:12,834:140038718158592:INFO:RucioInjectorPoller:Preparing to insert containers into Rucio...
2020-12-11 07:30:17,940:140038718158592:INFO:RucioInjectorPoller:Container /TTToSemiLeptonic_TuneCP5down_13TeV-powheg-pythia8/RunIISummer19UL18NanoAODv2-106X_upgrade2018_realistic_v15_L1v1-v1/NANOAODSIM inserted into Rucio
2020-12-11 07:30:26,951:140038718158592:INFO:RucioInjectorPoller:Container /DY3JetsToLL_M-50_TuneCP5_13TeV-madgraphMLM-pythia8/RunIISummer19UL17NanoAODv2-106X_mc2017_realistic_v8-v1/NANOAODSIM inserted into Rucio
2020-12-11 07:30:35,974:140038718158592:INFO:RucioInjectorPoller:Container /QCD_HT300to500_TuneCP5_PSWeights_13TeV-madgraphMLM-pythia8/RunIISummer19UL17NanoAODv2-106X_mc2017_realistic_v8-v1/NANOAODSIM inserted into Rucio
2020-12-11 07:30:35,975:140038718158592:INFO:RucioInjectorPoller:Preparing to insert blocks into Rucio...
2020-12-11 07:30:54,050:140038718158592:INFO:RucioInjectorPoller:Block /TTToSemiLeptonic_TuneCP5down_13TeV-powheg-pythia8/RunIISummer19UL18NanoAODv2-106X_upgrade2018_realistic_v15_L1v1-v1/NANOAODSIM#5752c59b-b6d0-4ba3-a2f1-937a64e6f281 inserted into Rucio
2020-12-11 07:31:15,126:140038718158592:INFO:RucioInjectorPoller:Block /DY3JetsToLL_M-50_TuneCP5_13TeV-madgraphMLM-pythia8/RunIISummer19UL17NanoAODv2-106X_mc2017_realistic_v8-v1/NANOAODSIM#27f0b17c-be0a-44c0-a996-15ec820b6175 inserted into Rucio
```


Changed files:
src/python/WMComponent/RucioInjector/RucioInjectorPoller.py, src/python/WMCore/Services/Rucio/Rucio.py

diff --git a/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py b/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
index 0aa7a8ce4..d8f625fb0 100644
--- a/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
+++ b/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
@@ -287,6 +287,10 @@ class RucioInjectorPoller(BaseWorkerThread):
             if not self._isBlockTierAllowed(item['blockname']):
                 logging.debug("Component configured to skip block rule for: %s", item['blockname'])
                 continue
+            # first, check if the block has already been created in Rucio
+            if not self.rucio.didExist(item['blockname']):
+                logging.warning("Block: %s not yet in Rucio. Retrying later..", item['blockname'])
+                continue
             kwargs = dict(activity="Production Output", account=self.rucioAcct,
                           grouping="DATASET", comment="WMAgent automatic container rule",
                           ignore_availability=True, meta=self.metaData)
@@ -519,6 +523,10 @@ class RucioInjectorPoller(BaseWorkerThread):
                              container, rseName)
                 subscriptionsMade.append(subInfo['id'])
                 continue
+            # then check if the container has already been created in Rucio
+            if not self.rucio.didExist(container):
+                logging.warning("Container: %s not yet in Rucio. Retrying later..", container)
+                continue
 
             ruleKwargs = dict(ask_approval=False,
                               activity=self._activityMap(rseName),
diff --git a/src/python/WMCore/Services/Rucio/Rucio.py b/src/python/WMCore/Services/Rucio/Rucio.py
index e0b71bfab..fd999e8b6 100644
--- a/src/python/WMCore/Services/Rucio/Rucio.py
+++ b/src/python/WMCore/Services/Rucio/Rucio.py
@@ -842,6 +842,19 @@ class Rucio(object):
             self.logger.error("Data identifier not found in Rucio: %s. Error: %s", didName, str(exc))
         return response
 
+    def didExist(self, didName, scope='cms'):
+        """
+        Provided a given DID, check whether it's already in the Rucio server.
+        Any kind of exception will return False (thus, data not yet in Rucio).
+        :param didName: a string with the DID name (container, block, or file)
+        :return: True if DID has been found, False otherwise
+        """
+        try:
+            response = self.cli.get_did(scope=scope, name=didName)
+        except Exception:
+            response = dict()
+        return response.get("name") == didName
+
     # FIXME we can likely delete this method (replaced by another implementation)
     def getDataLockedAndAvailable_old(self, **kwargs):
         """