Repository: rucio/rucio

Issue:
Default multihop tombstone is effectively None
Motivation
----------
In
https://github.com/rucio/rucio/blob/d59c237f533f40116026dc9f347f4fc1297f1ff0/lib/rucio/core/transfer.py#L103
a `DEFAULT_MULTIHOP_TOMBSTONE_DELAY` is set, and then in 
https://github.com/rucio/rucio/blob/d59c237f533f40116026dc9f347f4fc1297f1ff0/lib/rucio/core/transfer.py#L1440
the default value is passed if no runtime configuration is set. The problem is, all cache values are converted to string in
https://github.com/rucio/rucio/blob/d59c237f533f40116026dc9f347f4fc1297f1ff0/lib/rucio/core/config.py#L191
which means that the actual default value ends up being the string `'2:00:00'`, and later on in `tombstone_from_delay` we get a ValueError which is converted to `None`
https://github.com/rucio/rucio/blob/d59c237f533f40116026dc9f347f4fc1297f1ff0/lib/rucio/core/replica.py#L1424-L1428
hence no tombstone unless `transfers.multihop_tombstone_delay` is explicitly set (or if the RSE has a `multihop_tombstone_delay` attribute)


Modification
------------
Set
```python
DEFAULT_MULTIHOP_TOMBSTONE_DELAY = int(datetime.timedelta(hours=2).total_seconds())
```




Discussion:
thank you for the investigation. I think the real problem is that we write the default to cache. Also, setting to none on conversion error isn't the best idea. @bari12 , WDYT?
Writing the default to cache is good in principle, this way we still minimise having to do database interaction in case the default is used. In principle dogpile supports writing the actual value (with type) to the cache and also retrieve it like that. I am not exactly sure why we cast it to string here. But I am a little bit concerned of changing this now and what side-effects this could result in.
For now I think @nsmith- suggestion is good. In a second step we should perhaps evaluate writing the right types to the cache.
A workaround is to set `rucio-admin config set --section transfers --option multihop_tombstone_delay --value 7200`

Resolving patch:
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -1421,11 +1421,7 @@ def tombstone_from_delay(tombstone_delay):
     if not tombstone_delay:
         return None
 
-    if not isinstance(tombstone_delay, timedelta):
-        try:
-            tombstone_delay = timedelta(seconds=int(tombstone_delay))
-        except ValueError:
-            return None
+    tombstone_delay = timedelta(seconds=int(tombstone_delay))
 
     if not tombstone_delay:
         return None
diff --git a/lib/rucio/core/transfer.py b/lib/rucio/core/transfer.py
--- a/lib/rucio/core/transfer.py
+++ b/lib/rucio/core/transfer.py
@@ -100,7 +100,7 @@
                                        arguments={'url': config_get('cache', 'url', False, '127.0.0.1:11211'), 'distributed_lock': True})
 WEBDAV_TRANSFER_MODE = config_get('conveyor', 'webdav_transfer_mode', False, None)
 
-DEFAULT_MULTIHOP_TOMBSTONE_DELAY = datetime.timedelta(hours=2)
+DEFAULT_MULTIHOP_TOMBSTONE_DELAY = int(datetime.timedelta(hours=2).total_seconds())
 
 
 class RseData:
@@ -1511,10 +1511,14 @@ def create_missing_replicas_and_requests(
         if rws.request_id:
             continue
 
-        if 'multihop_tombstone_delay' in rws.dest_rse.attributes:
-            tombstone = tombstone_from_delay(rws.dest_rse.attributes['multihop_tombstone_delay'])
-        else:
-            tombstone = tombstone_from_delay(default_tombstone_delay)
+        tombstone_delay = rws.dest_rse.attributes.get('multihop_tombstone_delay', default_tombstone_delay)
+        try:
+            tombstone = tombstone_from_delay(tombstone_delay)
+        except ValueError:
+            logger(logging.ERROR, "%s: Cannot parse multihop tombstone delay %s", initial_request_id, tombstone_delay)
+            creation_successful = False
+            break
+
         files = [{'scope': rws.scope,
                   'name': rws.name,
                   'bytes': rws.byte_count,


Test patch: