Repository: rucio/rucio

Issue:
Listing replicas: Support storage that doesn't support multirange-byte requests
### Description

During a multi-range query, the client requests multiple contiguous chunks of a file. Not all servers support this, however. Some will understand the request and return a proper multi-chunk response, some will return the entire file, some will only give you the first range, and some will return an error.

If you know that a server does not support multi-range, you can instruct davix to jump straight to simulation by using a URL fragment parameter.

`https://example.org/somefile&#multirange=false`

Davix will try to lessen this performance impact by using multiple parallel HTTP connections. The default number is 10 connections, but you can override it with a URL fragment parameter. (Notice the weird `&#` construct.)

`https://example.org/somefile&#multirange=false&nconnections=30`

### Motivation

This is needed to support Google Cloud Storage for analysis with direct IO. We need to simulate the multirange byte request.

### Change

Extend `list_replicas()` (Client, REST, API, Core) to support an additional parameter, passing through the nr parallels, e.g.,

`rucio list-file-replicas .... --simulate-multirange=10`

`def list_replicas(....., simulate_multirange=10):`

(other name proposals welcome).


Such that the returned PFNs then look like:

`https://hostname:443/prefix/filename.txt&#multirange=false&nconnections=10`


(+ unit tests :-)


Discussion:
Do we need to store if an RSE supports multi-range responses? If not, why not just do this client-side?
Hmmm, yes, that's an RSE "feature", but the nr_connections are client-specific.
Then it's probably easier to do that client-sided.
Thus still have the `rucio list-file-replicas .... --simulate-multirange=10` command, but actually not pass this through to the server, but just append it client-sided for all `https://` urls?
Hmm, in addition the pilot calls `replicaclient.list_replicas(...)`.... are you suggesting that this should be done purely by the rucio copytool in the pilot? 
After discussing with @PalNilsson we decided that giving clients the choice of the concurrency is maybe not the best idea.

So instead the implementation should look like this:

If an RSE has an RSE Attribute `multirange`:`<int>`, then apply the workaround when calling `list_replicas()`. No need for the API/client to change anything.

Resolving patch:
diff --git a/lib/rucio/core/credential.py b/lib/rucio/core/credential.py
--- a/lib/rucio/core/credential.py
+++ b/lib/rucio/core/credential.py
@@ -37,7 +37,7 @@
 METRICS = MetricManager(module=__name__)
 
 
-def get_signed_url(rse_id, service, operation, url, lifetime=600):
+def get_signed_url(rse_id: str, service: str, operation: str, url: str, lifetime=600) -> str:
     """
     Get a signed URL for a particular service and operation.
 
@@ -69,7 +69,6 @@ def get_signed_url(rse_id, service, operation, url, lifetime=600):
             except:
                 raise UnsupportedOperation('Lifetime must be convertible to numeric.')
 
-    signed_url = None
     if service == 'gcs':
         if not CREDS_GCS:
             CREDS_GCS = Credentials.from_service_account_file(config_get('credentials', 'gcs',
@@ -102,11 +101,11 @@ def get_signed_url(rse_id, service, operation, url, lifetime=600):
         signature = urlencode({'': base64.b64encode(CREDS_GCS.sign_bytes(to_sign))})[1:]
 
         # assemble final signed URL
-        signed_url = 'https://%s%s?GoogleAccessId=%s&Expires=%s&Signature=%s' % (host,
-                                                                                 path,
-                                                                                 CREDS_GCS.service_account_email,
-                                                                                 lifetime,
-                                                                                 signature)
+        signed_url = (
+            f'https://{host}{path}'
+            f'?GoogleAccessId={CREDS_GCS.service_account_email}'
+            f'&Expires={lifetime}&Signature={signature}'
+        )
 
     elif service == 's3':
 
@@ -169,16 +168,17 @@ def get_signed_url(rse_id, service, operation, url, lifetime=600):
                 s3_url_style = "virtual"
 
             s3 = boto3.client(service_name='s3',
-                              endpoint_url='https://' + host + ':' + port,
+                              endpoint_url=f'https://{host}:{port}',
                               aws_access_key_id=access_key,
                               aws_secret_access_key=secret_key,
                               config=Config(signature_version=signature_version,
                                             region_name=region_name,
                                             s3={"addressing_style": s3_url_style}))
 
-            signed_url = s3.generate_presigned_url(s3op, Params={'Bucket': bucket, 'Key': key}, ExpiresIn=lifetime)
+            signed_url: str = s3.generate_presigned_url(
+                s3op, Params={'Bucket': bucket, 'Key': key}, ExpiresIn=lifetime)
 
-    elif service == 'swift':
+    else:  # service == 'swift'
         # split URL to get hostname and path
         components = urlparse(url)
         host = components.netloc
@@ -213,6 +213,6 @@ def get_signed_url(rse_id, service, operation, url, lifetime=600):
             hmac_body = u'%s\n%s\n%s' % (swiftop, expires, components.path)
             # Python 3 hmac only accepts bytes or bytearray
             sig = hmac.new(bytearray(tempurl_key, 'utf-8'), bytearray(hmac_body, 'utf-8'), sha1).hexdigest()
-            signed_url = 'https://' + host + components.path + '?temp_url_sig=' + sig + '&temp_url_expires=' + str(expires)
+            signed_url = f'https://{host}{components.path}?temp_url_sig={sig}&temp_url_expires={expires}'
 
     return signed_url
diff --git a/lib/rucio/core/replica.py b/lib/rucio/core/replica.py
--- a/lib/rucio/core/replica.py
+++ b/lib/rucio/core/replica.py
@@ -804,6 +804,8 @@ def _build_list_replicas_pfn(
         sign_urls: bool,
         signature_lifetime: int,
         client_location: "Dict[str, Any]",
+        logger=logging.log,
+        *,
         session: "Session",
 ) -> str:
     """
@@ -811,9 +813,9 @@ def _build_list_replicas_pfn(
     If needed, sign the PFN url
     If relevant, add the server-side root proxy to te pfn url
     """
-    pfn = list(protocol.lfns2pfns(lfns={'scope': scope.external,
-                                        'name': name,
-                                        'path': path}).values())[0]
+    pfn: str = list(protocol.lfns2pfns(lfns={'scope': scope.external,
+                                             'name': name,
+                                             'path': path}).values())[0]
 
     # do we need to sign the URLs?
     if sign_urls and protocol.attributes['scheme'] == 'https':
@@ -839,7 +841,7 @@ def _build_list_replicas_pfn(
                     # print('filename', name)
                     selected_prefix = get_multi_cache_prefix(cache_site, name)
                     if selected_prefix:
-                        pfn = 'root://' + selected_prefix + '//' + pfn.replace('davs://', 'root://')
+                        pfn = f"root://{selected_prefix}//{pfn.replace('davs://', 'root://')}"
                 else:
                     # print('site:', client_location['site'], 'has no cache')
                     # print('lets check if it has defined an internal root proxy ')
@@ -855,7 +857,22 @@ def _build_list_replicas_pfn(
                             pass  # ATLAS HACK
                         else:
                             # don't forget to mangle gfal-style davs URL into generic https URL
-                            pfn = 'root://' + root_proxy_internal + '//' + pfn.replace('davs://', 'https://')
+                            pfn = f"root://{root_proxy_internal}//{pfn.replace('davs://', 'https://')}"
+
+    simulate_multirange = get_rse_attribute(rse_id, 'simulate_multirange')
+
+    if simulate_multirange is not None:
+        try:
+            # cover values that cannot be cast to int
+            simulate_multirange = int(simulate_multirange)
+        except ValueError:
+            simulate_multirange = 1
+            logger(logging.WARNING, 'Value encountered when retrieving RSE attribute "simulate_multirange" not compatible with "int", used default value "1".')
+        if simulate_multirange <= 0:
+            logger(logging.WARNING, f'Value {simulate_multirange} encountered when retrieving RSE attribute "simulate_multirange" is <= 0, used default value "1".')
+            simulate_multirange = 1
+        pfn += f'&#multirange=false&nconnections={simulate_multirange}'
+
     return pfn
 
 


Test patch:
diff --git a/lib/rucio/tests/temp_factories.py b/lib/rucio/tests/temp_factories.py
--- a/lib/rucio/tests/temp_factories.py
+++ b/lib/rucio/tests/temp_factories.py
@@ -31,6 +31,10 @@
 from rucio.tests.common_server import cleanup_db_deps
 from sqlalchemy import and_, or_, delete
 
+from typing import TYPE_CHECKING
+if TYPE_CHECKING:
+    from typing import Tuple
+
 
 class TemporaryRSEFactory:
     """
@@ -68,7 +72,7 @@ def _cleanup_rses(self):
             # So running test in parallel results in some tests failing on foreign key errors.
             rse_core.del_rse(rse_id)
 
-    def _make_rse(self, scheme, protocol_impl, parameters=None, add_rse_kwargs=None):
+    def _make_rse(self, scheme, protocol_impl, parameters=None, add_rse_kwargs=None) -> "Tuple[str, str]":
         rse_name = self.name_prefix + ''.join(choice(ascii_uppercase) for _ in range(6))
         if add_rse_kwargs and 'vo' in add_rse_kwargs:
             rse_id = rse_core.add_rse(rse_name, **add_rse_kwargs)
diff --git a/lib/rucio/tests/test_replica.py b/lib/rucio/tests/test_replica.py
--- a/lib/rucio/tests/test_replica.py
+++ b/lib/rucio/tests/test_replica.py
@@ -46,6 +46,10 @@
 from rucio.rse import rsemanager as rsemgr
 from rucio.tests.common import execute, headers, auth, Mime, accept, did_name_generator
 
+from typing import TYPE_CHECKING
+if TYPE_CHECKING:
+    from rucio.tests.temp_factories import TemporaryRSEFactory
+
 
 def mocked_VP_requests_get(*args, **kwargs):
     """This method will be used by the mock to replace requests.get to VP server."""
@@ -196,6 +200,46 @@ def test_add_list_replicas(self, rse_factory, mock_scope, root_account):
 
         assert nbfiles == replica_cpt
 
+    @pytest.mark.parametrize(
+        "params",
+        [
+            (False, None, None),
+            (True, 10, 10),
+            (True, -2, 1),
+            (True, 0, 1),
+            (True, 56.7, 1),
+            (True, "oops", 1)
+        ]
+    )
+    def test_list_replicas_sim_multirange(
+        self, params, rse_factory: "TemporaryRSEFactory", mock_scope, root_account
+    ):
+        """ REPLICA (CORE): List file replicas for storage that does not support multirange-byte requests should add suffix to PFN"""
+
+        # bear in mind that if a bad value is entered as rse_attribute, the default value is 1
+        simulate_multirange, nconns, expectedconns = params
+
+        _, rse_id = rse_factory.make_mock_rse()
+
+        if simulate_multirange:
+            add_rse_attribute(
+                rse_id=rse_id, key='simulate_multirange', value=str(nconns)
+            )
+
+        # add files
+        file_name = did_name_generator('testfiles')
+        add_replica(rse_id, mock_scope, file_name, 2, root_account)
+
+        dids = [{'scope': mock_scope, 'name': file_name, 'type': DIDType.FILE}]
+        file = next(list_replicas(dids=dids))
+        pfn = list(file['pfns'].keys())[0]
+        assert pfn[:7] == 'mock://'
+        assert file_name in pfn
+        if simulate_multirange:
+            assert f'&#multirange=false&nconnections={expectedconns}' in pfn
+        else:
+            assert '&#multirange=false&nconnections' not in pfn
+
     @pytest.mark.parametrize("file_config_mock", [
         # Run test twice: with, and without, temp tables
         {"overrides": [('core', 'use_temp_tables', 'True')]},