Repository: rucio/rucio

Issue:
Incorrect use of arguments in Dark Reaper
Motivation
----------

I am further confused by commit f5578416bdfc071ebac14ca3cd2f64b669993597. `list_rses()` returns a list of strings, each containing an RSE ID (previously the RSE name instead). Yet the `reaper()` function seems to have been modified to expect a list of dictionaries.

Modification
------------





Discussion:
This is sufficient to get it working, but I am not sure if it’s what was intended.

```
--- dark_reaper.py.orig 2020-09-29 15:05:29.639185487 +0200
+++ dark_reaper.py      2020-09-29 15:02:08.044885938 +0200
@@ -75,13 +75,13 @@
     :param once: If True, only runs one iteration of the main loop.
     :param scheme: Force the reaper to use a particular protocol, e.g., mock.
     """
-    logging.info('Starting Dark Reaper %s-%s: Will work on RSEs: %s', worker_number, total_workers, ', '.join([rse['rse'] for rse in rses]))
+    logging.info('Starting Dark Reaper %s-%s: Will work on RSEs: %s', worker_number, total_workers, ', '.join(rses))
 
     pid = os.getpid()
     thread = threading.current_thread()
     hostname = socket.gethostname()
     executable = ' '.join(sys.argv)
-    hash_executable = hashlib.sha256(sys.argv[0] + ''.join([rse['rse'] for rse in rses])).hexdigest()
+    hash_executable = hashlib.sha256(sys.argv[0] + ''.join(rses)).hexdigest()
     sanity_check(executable=None, hostname=hostname)
 
     while not GRACEFUL_STOP.is_set():
@@ -92,14 +92,13 @@
             nothing_to_do = True
 
             random.shuffle(rses)
-            for rse in rses:
-                rse_id = rse['id']
-                rse = rse['rse']
+            for rse_id in rses:
                 replicas = list_quarantined_replicas(rse_id=rse_id,
                                                      limit=chunk_size, worker_number=worker_number,
                                                      total_workers=total_workers)
 
                 rse_info = rsemgr.get_rse_info(rse_id=rse_id)
+                rse = rse_info['rse']
                 prot = rsemgr.create_protocol(rse_info, 'delete', scheme=scheme)
                 deleted_replicas = []
                 try:

```
Hi @dchristidis, I suspect this was probably due to me confusing `quarantined_replica.list_rses` (which returns a list of strings and is used by the dark_reaper) with `core.list_rses` (which returns a list of dictionaries and is used elsewhere) while I was making changes to all the different reapers to accept the VO. As far as I know there isn't a test for dark_reaper in the test suite which would explain why this went undetected?

The changes proposed look good to me, as the handling of VO for multi_vo mode isn't affected. Hopefully there isn't any further issues with this commit, though if there are and I might be able to help please do let me know. Unfortunately I'm now working on a different project for STFC/RAL (not Rucio), but I'm more than happy to have a quick look over any issues related to multi_vo or my changes. Would it be OK for you to create the PR etc. for this change?
Thanks a lot, that would explain the confusion. I’ll take it from here.

Resolving patch:
diff --git a/lib/rucio/daemons/reaper/dark_reaper.py b/lib/rucio/daemons/reaper/dark_reaper.py
--- a/lib/rucio/daemons/reaper/dark_reaper.py
+++ b/lib/rucio/daemons/reaper/dark_reaper.py
@@ -23,6 +23,7 @@
 # - Brandon White <bjwhite@fnal.gov>, 2019
 # - Patrick Austin <patrick.austin@stfc.ac.uk>, 2020
 # - Benedikt Ziemons <benedikt.ziemons@cern.ch>, 2020
+# - Dimitrios Christidis <dimitrios.christidis@cern.ch>, 2020
 
 '''
 Dark Reaper is a daemon to manage quarantined file deletion.
@@ -76,13 +77,13 @@ def reaper(rses=[], worker_number=0, total_workers=1, chunk_size=100, once=False
     :param once: If True, only runs one iteration of the main loop.
     :param scheme: Force the reaper to use a particular protocol, e.g., mock.
     """
-    logging.info('Starting Dark Reaper %s-%s: Will work on RSEs: %s', worker_number, total_workers, ', '.join([rse['rse'] for rse in rses]))
+    logging.info('Starting Dark Reaper %s-%s: Will work on RSEs: %s', worker_number, total_workers, ', '.join(rses))
 
     pid = os.getpid()
     thread = threading.current_thread()
     hostname = socket.gethostname()
     executable = ' '.join(sys.argv)
-    hash_executable = hashlib.sha256(sys.argv[0] + ''.join([rse['rse'] for rse in rses])).hexdigest()
+    hash_executable = hashlib.sha256(sys.argv[0] + ''.join(rses)).hexdigest()
     sanity_check(executable=None, hostname=hostname)
 
     while not GRACEFUL_STOP.is_set():
@@ -93,14 +94,13 @@ def reaper(rses=[], worker_number=0, total_workers=1, chunk_size=100, once=False
             nothing_to_do = True
 
             random.shuffle(rses)
-            for rse in rses:
-                rse_id = rse['id']
-                rse = rse['rse']
+            for rse_id in rses:
                 replicas = list_quarantined_replicas(rse_id=rse_id,
                                                      limit=chunk_size, worker_number=worker_number,
                                                      total_workers=total_workers)
 
                 rse_info = rsemgr.get_rse_info(rse_id=rse_id)
+                rse = rse_info['rse']
                 prot = rsemgr.create_protocol(rse_info, 'delete', scheme=scheme)
                 deleted_replicas = []
                 try:


Test patch: