Repository: dmwm/wmcore
Issue: Rucio DIDs that failed to get attached to their parent DID are not retried

**Impact of the bug**
WMAgent

**Describe the bug**
A bad bug! If there is any exception while attaching a DID to its parent - like attaching a block to its container - that operation isn't retried again. Which means, the block + files created doesn't belong to any CMS dataset...

**How to reproduce it**
Cause an exception when attaching DIDs

**Expected behavior**
If file or block fails to get attached to its parent DID, we should make sure the same operation gets retried in the next cycle.

**Additional context and error message**
Issue reported in this JIRA ticket:
https://its.cern.ch/jira/browse/CMSCOMPPR-17508

corresponding to this log in vocms0280:
```
2020-09-29 20:45:29,845:139990532335360:ERROR:Rucio:Exception attaching dids: [{'scope': 'cms', 'name': '/Zprime_NonUniversalSSM_M4000_gl0p5_gh0p5_mg_pythia8_TuneCP5/RunIIAutumn18NanoAODv7-Nano02Apr2020_102X_upgrade2018_realistic_v21-v
1/NANOAODSIM#1a2e32c5-b350-4140-adce-cba0cb2c2019'}] to: /Zprime_NonUniversalSSM_M4000_gl0p5_gh0p5_mg_pythia8_TuneCP5/RunIIAutumn18NanoAODv7-Nano02Apr2020_102X_upgrade2018_realistic_v21-v1/NANOAODSIM. Error: An unknown exception occurr
ed.
Details: no error information passed (http status code: 502 ('bad_gateway',))
2020-09-29 20:45:29,845:139990532335360:ERROR:RucioInjectorPoller:Failed to create block: /Zprime_NonUniversalSSM_M4000_gl0p5_gh0p5_mg_pythia8_TuneCP5/RunIIAutumn18NanoAODv7-Nano02Apr2020_102X_upgrade2018_realistic_v21-v1/NANOAODSIM#1a
2e32c5-b350-4140-adce-cba0cb2c2019
...
2020-09-29 21:06:06,411:139990532335360:INFO:RucioInjectorPoller:Successfully inserted 1 files on block /Zprime_NonUniversalSSM_M4000_gl0p5_gh0p5_mg_pythia8_TuneCP5/RunIIAutumn18NanoAODv7-Nano02Apr2020_102X_upgrade2018_realistic_v21-v1/NANOAODSIM#1a2e32c5-b350-4140-adce-cba0cb2c2019
...
2020-09-29 21:15:47,156:139990532335360:INFO:RucioInjectorPoller:Closing block: /Zprime_NonUniversalSSM_M4000_gl0p5_gh0p5_mg_pythia8_TuneCP5/RunIIAutumn18NanoAODv7-Nano02Apr2020_102X_upgrade2018_realistic_v21-v1/NANOAODSIM#1a2e32c5-b350-4140-adce-cba0cb2c2019
```

Changed files:
src/python/WMComponent/RucioInjector/RucioInjectorPoller.py, src/python/WMCore/Services/Rucio/Rucio.py

diff --git a/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py b/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
index 0f917356a..f4c053f52 100644
--- a/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
+++ b/src/python/WMComponent/RucioInjector/RucioInjectorPoller.py
@@ -317,6 +317,10 @@ class RucioInjectorPoller(BaseWorkerThread):
             rseName = "%s_Test" % location if self.testRSEs else location
             for container in uninjectedData[location]:
                 for block in uninjectedData[location][container]:
+                    if block not in self.blocksCache:
+                        logging.warning("Skipping %d file injection for block that failed to be added into Rucio: %s",
+                                        len(uninjectedData[location][container][block]['files']), block)
+                        continue
                     injectData = []
                     listLfns = []
                     for fileInfo in uninjectedData[location][container][block]['files']:
diff --git a/src/python/WMCore/Services/Rucio/Rucio.py b/src/python/WMCore/Services/Rucio/Rucio.py
index 7e412155a..6eef18062 100644
--- a/src/python/WMCore/Services/Rucio/Rucio.py
+++ b/src/python/WMCore/Services/Rucio/Rucio.py
@@ -414,8 +414,17 @@ class Rucio(object):
         try:
             # add_replicas(rse, files, ignore_availability=True)
             response = self.cli.add_replicas(rse, files, ignoreAvailability)
-        except Exception as ex:
-            self.logger.error("Failed to add replicas for: %s and block: %s. Error: %s", files, block, str(ex))
+        except DataIdentifierAlreadyExists as exc:
+            if len(files) == 1:
+                self.logger.debug("File replica already exists in Rucio: %s", files[0]['name'])
+                response = True
+            else:
+                # FIXME: I think we would have to iterate over every single file and add then one by one
+                errorMsg = "Failed to insert replicas for: {} and block: {}".format(files, block)
+                errorMsg += " Some/all DIDs already exist in Rucio. Error: {}".format(str(exc))
+                self.logger.error(errorMsg)
+        except Exception as exc:
+            self.logger.error("Failed to add replicas for: %s and block: %s. Error: %s", files, block, str(exc))
 
         if response:
             files = [item['name'] for item in files]