Repository: rucio/rucio

Issue:
Make suspicious-replica-recoveror daemon VO-agnostic
### Description

We're evaluating this daemon for CMS use and I've realized that there is an RSE expression hard-coded [1] which appears to be ATLAS specific. Can we make it configurable?

There could be other parts to be changed in this regard that I didn't realize. Do the experts see any other change required in this context?

[1] https://github.com/rucio/rucio/blob/0290623118d0890e799b848d72f2c1a3b18b2b04/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py#L294

### Motivation

To be adapted by CMS

### Change

Make all VO-specific parts of this daemon VO-agnostic.

@ericvaandering  @Panos512  FYI


Discussion:
I've found another reason why this daemon wouldn't work for CMS out of the box: It requires dids to have `datatype` metadata and it ignores the files which don't have it [1]. CMS doesn't use this metadata field. I don't know what's the best strategy here: Use another metadata field to apply policies (fileName patterns?)

[1] https://github.com/rucio/rucio/blob/0290623118d0890e799b848d72f2c1a3b18b2b04/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py#L367 
Hi,

I think the rse_expression should be easy to adapt. For the fixed datatype field, I am not sure. @ChristophAmes and @cserf can comment.
@haozturk and I talked about solutions for this - We agreed that the most straight forward solution is to make both [1] and [2] configurable via the config file, with defaults that don't impact ATLAS

[1] Would be something like 
Config: 
```
[replicarecoverer]
rse_expression = rse_expression
```
`suspicious_replica_recoverer.py`: 
```
rse_expression = config_get('replicarecoverer', 'rse_expression', raise_exception=False, default='type=SCRATCHDISK')
 add_rule(dids=dids_nattempts_1, account=InternalAccount('root', vo=vo), copies=nattempts, rse_expression=rse_expression, grouping=None, weight=None, lifetime=24 * 3600, locked=False, subscription_id=None) 
```

[2]
Config: 

```
[replicarecoverer] 
use_file_metadata = false 
datatype_expression = {matching regex}
```

`suspicious_replica_recoverer.py`: 
```
use_file_metadata = config_get_bool("replicarecoverer","use_file_metadata", raise_exception=False, default=True)
if use_file_metadata: 
    file_metadata_datatype = str(file_metadata["datatype"]) 
else: 
       name_regex = config_get('replicarecoverer', 'datatype_expression', raise_exception=True)
       file_metadata_datatype = re.compile(name_regex).match(file_name) 
```

[1]
https://github.com/rucio/rucio/blob/0290623118d0890e799b848d72f2c1a3b18b2b04/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py#L294
[2]
https://github.com/rucio/rucio/blob/0290623118d0890e799b848d72f2c1a3b18b2b04/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py#L367

One minor comment. Maybe "did_name_expression" rather than "datatype_expression" so it's clear what we're proposing to act on. (Assuming that's right, we will just look at the DID name).

That leaves open the possibility for adding other types of selections for other use cases.
@bari12 Is this proposed solution okay? 

@ChristophAmes has there been any work on this? If not I can gladly take it over. 
For me this is ok, but @cserf or @ChristophAmes should really comment on this, as the experts. 🔔 
The proposed solutions look fine to us. We haven't done any work on this topic yet.

Resolving patch:
diff --git a/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py b/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py
--- a/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py
+++ b/lib/rucio/daemons/replicarecoverer/suspicious_replica_recoverer.py
@@ -25,15 +25,16 @@
 import re
 import threading
 import time
+from configparser import NoOptionError, NoSectionError
 from datetime import datetime, timedelta
 from typing import TYPE_CHECKING, Any, Optional
 
 import rucio.db.sqla.util
-from rucio.common.config import config_get_bool
+from rucio.common.config import config_get, config_get_bool
 from rucio.common.constants import SuspiciousAvailability
 from rucio.common.exception import DatabaseException, DuplicateRule, VONotFound
 from rucio.common.logging import setup_logging
-from rucio.common.types import InternalAccount
+from rucio.common.types import InternalAccount, LoggerFunction
 from rucio.core.did import get_metadata
 from rucio.core.replica import (
     add_bad_pfns,
@@ -49,6 +50,7 @@
 from rucio.db.sqla.util import get_db_time
 
 if TYPE_CHECKING:
+    from collections.abc import Sequence
     from types import FrameType
 
 GRACEFUL_STOP = threading.Event()
@@ -80,7 +82,54 @@ def check_suspicious_policy(policy: dict[str, str], file_metadata_datatype: str,
     return action
 
 
-def declare_suspicious_replicas_bad(once: bool = False, younger_than: int = 5, nattempts: int = 5, vos: Optional[list[str]] = None, limit_suspicious_files_on_rse: int = 5, json_file_name: str = "/opt/rucio/etc/suspicious_replica_recoverer.json", sleep_time: int = 3600, active_mode: bool = False) -> None:
+def parse_replica_datatype(
+        did_name_expression: Optional[str],
+        file_name: str,
+        rse_key: str,
+        logger: LoggerFunction) -> Optional[str]:
+
+    file_metadata_datatype = None
+    if did_name_expression is not None:
+        logger(logging.WARNING, f'Using {did_name_expression} as regex to determine file datatype for file: {file_name} on rse: {rse_key}')
+        try:
+            file_metadata_datatype_reg = re.compile(did_name_expression).search(file_name)
+            if file_metadata_datatype_reg is not None:
+                file_metadata_datatype = file_metadata_datatype_reg[0]
+        except (re.error, TypeError) as e:
+            logger(logging.WARNING, f'Could not determine datatype using regex {did_name_expression} on file: {file_name}, ignoring: {e}')
+
+    return file_metadata_datatype
+
+
+def _read_from_config(logger: LoggerFunction) -> tuple[str, bool, Optional[str]]:
+    rule_suspicious_rse_expression = config_get("replicarecoverer", "rule_rse_expression", raise_exception=False, default='type=SCRATCHDISK')
+    logger(logging.INFO, f"Recovering with the rse expression: {rule_suspicious_rse_expression}")
+
+    use_file_metadata = config_get_bool("replicarecoverer", "use_file_metadata", raise_exception=False, default=True)
+    did_name_expression = None
+
+    if not use_file_metadata:
+        try:
+            did_name_expression = config_get('replicarecoverer', 'did_name_expression', raise_exception=True)
+            logger(logging.INFO, f"Setting replica datatype from did name with regex expression: {did_name_expression}")
+
+        except (NoOptionError, NoSectionError) as e:
+            logger(logging.WARNING, f"Cannot use option 'use_file_metadata' without setting a replacement expression with 'did_name_expression'- {e}")
+            use_file_metadata = True
+
+    logger(logging.INFO, f"Recovering with the rse expression: {rule_suspicious_rse_expression}")
+    return rule_suspicious_rse_expression, use_file_metadata, did_name_expression
+
+
+def declare_suspicious_replicas_bad(
+        once: bool = False,
+        younger_than: int = 5,
+        nattempts: int = 5,
+        vos: "Optional[Sequence[str]]" = None,
+        limit_suspicious_files_on_rse: int = 5,
+        json_file_name: str = "/opt/rucio/etc/suspicious_replica_recoverer.json",
+        sleep_time: int = 3600,
+        active_mode: bool = False) -> None:
     """
     Main loop to check for available replicas which are labeled as suspicious.
 
@@ -123,10 +172,10 @@ def declare_suspicious_replicas_bad(once: bool = False, younger_than: int = 5, n
     )
 
 
-def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Optional[list[str]], limit_suspicious_files_on_rse: int, json_file_name: str, active_mode: int, **_kwargs) -> bool:
-
-    worker_number, total_workers, logger = heartbeat_handler.live()
+def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: "Optional[Sequence[str]]", limit_suspicious_files_on_rse: int, json_file_name: str, active_mode: int, **_kwargs) -> bool:
 
+    _, _, logger = heartbeat_handler.live()
+    rule_suspicious_rse_expression, use_file_metadata, did_name_expression = _read_from_config(logger=logger)
     vos = vos or []
 
     multi_vo = config_get_bool('common', 'multi_vo', raise_exception=False, default=False)
@@ -302,7 +351,7 @@ def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Opt
         logger(logging.INFO, 'All RSEs have been checked for suspicious replicas. Total time: %s seconds.', time.time() - start)
 
         # Checking that everything is still working properly
-        worker_number, total_workers, logger = heartbeat_handler.live()
+        _, _, logger = heartbeat_handler.live()
 
         logger(logging.INFO, 'Create rules for replicas with nattempts=1.')
 
@@ -340,11 +389,13 @@ def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Opt
                         action = ""
                     else:
                         # Deal with replicas based on their metadata.
-                        if file_metadata["datatype"] is None:  # "None" type has no function "split()"
+                        if (file_metadata["datatype"] is None) and (use_file_metadata):  # "None" type has no function "split()"
                             logger(logging.WARNING, "RSE: %s, replica name %s, surl %s: Replica does not have a data type associated with it. No action will be taken.",
                                    rse_key, replica_key, replicas_nattempts_1[vo][rse_key][replica_key]['surl'])
                             continue
                         file_metadata_datatype = str(file_metadata["datatype"])
+                        if not use_file_metadata:
+                            file_metadata_datatype = parse_replica_datatype(did_name_expression, file_name, rse_key, logger)
                         file_metadata_scope = str(file_metadata["scope"])
                         action = ""
                         if file_metadata_datatype:
@@ -361,7 +412,7 @@ def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Opt
             if active_mode:
                 if len(dids_nattempts_1) > 0:
                     try:
-                        add_rule(dids=dids_nattempts_1, account=InternalAccount('root', vo=vo), copies=nattempts, rse_expression='type=SCRATCHDISK', grouping=None, weight=None, lifetime=5 * 24 * 3600, locked=False, subscription_id=None)
+                        add_rule(dids=dids_nattempts_1, account=InternalAccount('root', vo=vo), copies=nattempts, rse_expression=rule_suspicious_rse_expression, grouping=None, weight=None, lifetime=5 * 24 * 3600, locked=False, subscription_id=None)
                         logger(logging.INFO, 'Rules have been created for %i replicas on %s.', len(dids_nattempts_1), rse_key)
                     except DuplicateRule:
                         logger(logging.INFO, 'Tried to create rules on %s, but it already exists.', rse_key)
@@ -403,7 +454,7 @@ def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Opt
             logger(logging.INFO, "%s", rse)
 
         # Checking that everything is still working properly
-        worker_number, total_workers, logger = heartbeat_handler.live()
+        _, _, logger = heartbeat_handler.live()
 
         auditor = 0
         checksum = 0
@@ -451,13 +502,15 @@ def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Opt
                             del recoverable_replicas[vo][rse_key][replica_key]
                         else:
                             # Deal with replicas based on their metadata.
-                            if file_metadata["datatype"] is None:  # "None" type has no function "split()"
+                            if (file_metadata["datatype"] is None) and use_file_metadata:  # "None" type has no function "split()"
                                 files_to_be_ignored.append(recoverable_replicas[vo][rse_key][replica_key])
                                 logger(logging.WARNING, "RSE: %s, replica name %s, surl %s: Replica does not have a data type associated with it. No action will be taken.",
                                        rse_key, replica_key, recoverable_replicas[vo][rse_key][replica_key]['surl'])
                                 continue
 
                             file_metadata_datatype = str(file_metadata["datatype"])
+                            if not use_file_metadata:
+                                file_metadata_datatype = parse_replica_datatype(did_name_expression, file_name, rse_key, logger)
                             file_metadata_scope = str(file_metadata["scope"])
                             action = ""
                             if file_metadata_datatype:
@@ -524,7 +577,15 @@ def run_once(heartbeat_handler: Any, younger_than: int, nattempts: int, vos: Opt
     return must_sleep
 
 
-def run(once: bool = False, younger_than: int = 5, nattempts: int = 5, vos: list[str] = None, limit_suspicious_files_on_rse: int = 5, json_file_name: str = "/opt/rucio/etc/suspicious_replica_recoverer.json", sleep_time: int = 3600, active_mode: bool = False) -> None:
+def run(
+        once: bool = False,
+        younger_than: int = 5,
+        nattempts: int = 5,
+        vos: "Optional[Sequence[str]]" = None,
+        limit_suspicious_files_on_rse: int = 5,
+        json_file_name: str = "/opt/rucio/etc/suspicious_replica_recoverer.json",
+        sleep_time: int = 3600,
+        active_mode: bool = False) -> None:
     """
     Starts up the Suspicious-Replica-Recoverer threads.
     """


Test patch:
diff --git a/tests/test_replica_recoverer.py b/tests/test_replica_recoverer.py
--- a/tests/test_replica_recoverer.py
+++ b/tests/test_replica_recoverer.py
@@ -520,3 +520,76 @@ def test_replica_recoverer(self, vo):
         assert (self.tmp_file11.name, self.rse4recovery_id, BadFilesStatus.BAD) not in bad_checklist
         assert (self.tmp_file12.name, self.rse4recovery_id, BadFilesStatus.BAD) not in bad_checklist
         assert (self.tmp_file13.name, self.rse4recovery_id, BadFilesStatus.BAD) not in bad_checklist
+
+
+@pytest.mark.parametrize(
+    "file_config_mock", [{"overrides": [
+        ("replicarecoverer", "rule_rse_expression", "type=SCRATCHDISK"),
+        ('replicarecoverer', 'use_file_metadata', 'False'),
+        ('replicarecoverer', 'did_name_expression', '^.*')]}],
+    indirect=True)
+def test_vo_agnostic_rules(file_config_mock, replica_client, rse_factory, scope_factory, file_factory, vo):
+    """
+    Runs only a single time, make sure it makes a new rule with this config.
+
+    Examining files without a datatype, but because the "use_file_metadata" is turned off,
+    files without a datatype can be acted on.
+    """
+    rse4suspicious, rse4suspicious_id = rse_factory.make_posix_rse(deterministic=True, vo=vo)
+    rse4recovery, rse4recovery_id = rse_factory.make_posix_rse(deterministic=True, vo=vo)
+
+    add_rse_attribute(rse4suspicious_id, "enable_suspicious_file_recovery", True)
+    add_rse_attribute(rse4recovery_id, "enable_suspicious_file_recovery", True)
+
+    # Create new scopes
+    _, [scope_declarebad] = scope_factory(vos=[vo])
+
+    tmp_file_delare_bad = file_factory.file_generator()
+
+    replicas_without_types = [{'scope': scope_declarebad, 'name': tmp_file_delare_bad.name, 'type': DIDType.FILE}]
+    for rse in [rse4suspicious, rse4recovery]:
+        cmd = f'rucio -v upload --rse {rse} --scope {scope_declarebad} {tmp_file_delare_bad}'
+        exitcode, _, _ = execute(cmd)
+        assert exitcode == 0
+
+    remove(tmp_file_delare_bad)
+    rse_expression_parser.REGION.invalidate()
+
+    update_replica_state(rse4recovery_id, scope_declarebad, tmp_file_delare_bad.name, ReplicaState.UNAVAILABLE)
+    replical_declare_bad = list(list_replicas(dids=replicas_without_types))
+
+    for replica in replical_declare_bad:
+        suspicious_pfns = replica['rses'][rse4suspicious_id]
+        for _ in range(3):
+            # The reason must contain the word "checksum", so that the replica can be declared bad.
+            replica_client.declare_suspicious_file_replicas([suspicious_pfns[0], ], 'checksum')
+            sleep(1)
+
+    recovery_policy = [{"action": "declare bad", "datatype": [str(tmp_file_delare_bad.name)], "scope": [str(scope_declarebad)]}]
+    recovery_policy_json = "/opt/rucio/etc/test_replica_recoverer_vo_agonistism.json"
+    json.dump(recovery_policy, open(recovery_policy_json, mode="w"))
+
+    try:
+        run(
+            once=True,
+            younger_than=1,
+            nattempts=2,
+            limit_suspicious_files_on_rse=2,
+            json_file_name=recovery_policy_json,
+            sleep_time=0,
+            active_mode=True
+        )
+    except KeyboardInterrupt:
+        stop()
+
+    # dids without types now have a different behavior
+
+    for replica in replicas_without_types:
+        # Files that are now acted on now the datatype can be found.
+        if replica['name'] == tmp_file_delare_bad.name:
+            # tmp_file11 should no longer be ignored.
+            assert not replica.get('states')
+
+    bad_replicas_list = list_bad_replicas_status(rse_id=rse4suspicious_id, vo=vo)
+    bad_checklist = [(badf['name'], badf['rse_id'], badf['state']) for badf in bad_replicas_list]
+    assert (tmp_file_delare_bad.name, rse4suspicious_id, BadFilesStatus.BAD) in bad_checklist