Repository: rucio/rucio

Issue:
Relative transfer timeout for downloads
   Dear all,

Please, can we have the --transfer-timeout option by default in rucio, if the file size is larger than i.e 4 G. This would be super useful, as (in the case of large files) rucio fails as shown below:

rucio get mc16_13TeV:DAOD_EGAM1.23267689._000140.pool.root.1
2021-03-02 15:34:57,432 INFO    Processing 1 item(s) for input
2021-03-02 15:34:57,432 INFO    Getting sources of DIDs
2021-03-02 15:34:58,100 INFO    Using main thread to download 1 file(s)
2021-03-02 15:34:58,100 INFO    Preparing download of mc16_13TeV:DAOD_EGAM1.23267689._000140.pool.root.1
2021-03-02 15:34:58,236 INFO    Trying to download with gsiftp from TOKYO-LCG2_DATADISK: mc16_13TeV:DAOD_EGAM1.23267689._000140.pool.root.1
2021-03-02 15:41:24,421 WARNING Download attempt failed. Try 1/2
2021-03-02 15:47:40,551 WARNING Download attempt failed. Try 2/2
2021-03-02 15:47:40,601 INFO    Trying to download with root from TOKYO-LCG2_DATADISK: mc16_13TeV:DAOD_EGAM1.23267689._000140.pool.root.1
2021-03-02 15:55:21,619 WARNING Download attempt failed. Try 1/2
2021-03-02 16:02:29,055 WARNING Download attempt failed. Try 2/2
2021-03-02 16:02:29,109 INFO    Trying to download with davs from TOKYO-LCG2_DATADISK: mc16_13TeV:DAOD_EGAM1.23267689._000140.pool.root.1
2021-03-02 16:08:47,471 WARNING Download attempt failed. Try 1/2
2021-03-02 16:15:15,489 WARNING Download attempt failed. Try 2/2
2021-03-02 16:15:15,544 ERROR   Failed to download file mc16_13TeV:DAOD_EGAM1.23267689._000140.pool.root.1
2021-03-02 16:15:15,545 ERROR   None of the requested files have been downloaded.


This was discussed in the DAST group:
https://groups.cern.ch/group/hn-atlas-dist-analysis-help/Lists/Archive/Flat.aspx?RootFolder=%2Fgroup%2Fhn%2Datlas%2Ddist%2Danalysis%2Dhelp%2FLists%2FArchive%2FCan%20not%20download%20one%20dataset%20using%20rucio&FolderCTID=0x01200200FB8F64B0E343CB4680DEAC546A9DF813

Thank you very much!
Otilia (DAST shifter)



Discussion:
I don't have a good idea on how to tackle this issue. If we want to add new client/CLI/config options, I think that the best solution is to resolve #4153 and then implement a mechanism similar to the one in curl [low_speed_time](https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html)  + [low_speed_limit](https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html) on top of it. This would allow to set the default timeout to 0 (do not timeout) in all rucio clients while still relying on these two options to detect stale downloads.
However, this doesn't seem trivial to implement.

I don't like the idea of automatically overriding the timeout set in a config file or on the command line for big files. If a timeout is requested, it must be respected as-is, without modifications.

A compromise solution, which I like and avoids adding any new client/CLI options: if timeout is not requested explicitly, compute a default timeout which depends on the file size and some internal "min_speed_limit" constant. This way, we don't add any "not-very-well-designed" cli/config options which we would have to keep supporting in the future. 
For example: 
```
if timeout is None:
    timeout = max(360, file_size/ 1Mbps) 
```
360 is the default timeout [used now in the CLI](https://github.com/rucio/rucio/blob/afad482b259bf8a5d045480f92a8d6cb26b4d9d2/bin/rucio#L2121). We can distinguish between "timeout=0" and timeout=None. In the first case, we wouldn't compute depending on file size.

Any opinions ? The last solution changes the behavior for clients instantiating the DownloadClient directly. For example, pilot will have a timeout enforced in places where it doesn't explicitly set this option.
I agree.
The first one is not trivial to implement, so I would also first work with what's there.

In principle I like your proposal, but I wonder if we need to be able to let the user/external application influence this more. We could offer two parameters
`--transfer-timeout` which does what it already does.
`--transfer-speed-timeout` (in Mbps) which does basically the maximum timeout calculation like you suggested.

Per default we only set the second one to 1Mbps. 
If both are present, `--transfer-timeout` has precedence.

I do think though that the default should also be used in the DownloadClient.
So we'll have a `--transfer-timeout` defaulting to "no timeout" and `--transfer-speed-timeout` defaulting to some constant (1mbps for example). If the first one is set, it overrides the second. If not, the internal timeout is computed from the second? 
Looks OK for me: it is my proposal, but with the possibility to set the desired speed.
Do we want to maintain the current behavior and enforce a 360second minimum on the value computed from `--transfer-speed-timeout`? Seems high, I'd vote to lower it a bit. 
Exactly. It's not perfect, but if we document the precedence well I think it should be fine.
360 seems a bit high to me, but we need to have some kind of minimum. Maybe 60s? There is some overheads with establishing the connection etc. we need to keep in mind for very small files. 
I'd even go lower. Something like 10s, but you know much better than me how big the static overhead can be. 

Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -275,9 +275,7 @@ def download_dids(self, items, num_threads=2, trace_custom_fields={}, traces_cop
         trace_custom_fields['uuid'] = generate_uuid()
 
         logger(logging.INFO, 'Processing %d item(s) for input' % len(items))
-        download_info = self._resolve_and_merge_input_items(copy.deepcopy(items))
-        did_to_options = download_info['did_to_options']
-        merged_items = download_info['merged_items']
+        did_to_options, merged_items = self._resolve_and_merge_input_items(copy.deepcopy(items))
 
         self.logger(logging.DEBUG, 'num_unmerged_items=%d; num_dids=%d; num_merged_items=%d' % (len(items), len(did_to_options), len(merged_items)))
 
@@ -428,6 +426,34 @@ def _download_worker(self, input_queue, output_queue, trace_custom_fields, trace
                 logger(logging.ERROR, '%sFailed to download item' % log_prefix)
                 logger(logging.DEBUG, error)
 
+    @staticmethod
+    def _compute_actual_transfer_timeout(item):
+        """
+        Merge the two options related to timeout into the value which will be used for protocol download.
+        :param item: dictionary that describes the item to download
+        :return: timeout in seconds
+        """
+        default_transfer_timeout = 360
+        default_transfer_speed_timeout = 1000000  # 1Mbps
+        # Static additive increment of the speed timeout. To include the static cost of
+        # establishing connections and download of small files
+        transfer_speed_timeout_static_increment = 60
+
+        transfer_timeout = item.get('merged_options', {}).get('transfer_timeout')
+        if transfer_timeout is not None:
+            return transfer_timeout
+
+        transfer_speed_timeout = item.get('merged_options', {}).get('transfer_speed_timeout')
+        bytes = item.get('bytes')
+        if not bytes or transfer_speed_timeout is None:
+            return default_transfer_timeout
+
+        if not transfer_speed_timeout > 0:
+            transfer_speed_timeout = default_transfer_speed_timeout
+
+        timeout = bytes // transfer_speed_timeout + transfer_speed_timeout_static_increment
+        return timeout
+
     def _download_item(self, item, trace, traces_copy_out, log_prefix=''):
         """
         Downloads the given item and sends traces for success/failure.
@@ -576,7 +602,7 @@ def _download_item(self, item, trace, traces_copy_out, log_prefix=''):
                 start_time = time.time()
 
                 try:
-                    protocol.get(pfn, temp_file_path, transfer_timeout=item.get('merged_options', {}).get('transfer_timeout'))
+                    protocol.get(pfn, temp_file_path, transfer_timeout=self._compute_actual_transfer_timeout(item))
                     success = True
                 except Exception as error:
                     logger(logging.DEBUG, error)
@@ -705,9 +731,7 @@ def download_aria2c(self, items, trace_custom_fields={}, filters={}):
             item['force_scheme'] = ['https', 'davs']
 
         logger(logging.INFO, 'Processing %d item(s) for input' % len(items))
-        download_info = self._resolve_and_merge_input_items(copy.deepcopy(items))
-        did_to_options = download_info['did_to_options']
-        merged_items = download_info['merged_items']
+        did_to_options, merged_items = self._resolve_and_merge_input_items(copy.deepcopy(items))
 
         self.logger(logging.DEBUG, 'num_unmerged_items=%d; num_dids=%d; num_merged_items=%d' % (len(items), len(did_to_options), len(merged_items)))
 
@@ -952,6 +976,34 @@ def _download_items_aria2c(self, items, aria_rpc, rpc_auth, trace_custom_fields=
 
         return items
 
+    def _resolve_one_item_dids(self, item):
+        """
+        Resolve scopes or wildcard DIDs to lists of full did names:
+        :param item: One input item
+        """
+        dids = item.get('did')
+        filters = item.get('filters', {})
+        if filters:
+            filters = copy.copy(filters)
+
+        if dids is None:
+            self.logger(logging.DEBUG, 'Resolving DIDs by using filter options')
+            scope = filters.pop('scope')
+            yield scope, list(self.client.list_dids(scope, filters=filters, type='all'))
+            return
+
+        if not isinstance(dids, list):
+            dids = [dids]
+
+        for did_str in dids:
+            scope, did_name = self._split_did_str(did_str)
+            if '*' in did_name:
+                filters['name'] = did_name
+                resolved_dids = list(self.client.list_dids(scope, filters=filters, type='all'))
+                yield scope, resolved_dids
+            else:
+                yield scope, [did_name]
+
     def _resolve_and_merge_input_items(self, items):
         """
         This function takes the input items given to download_dids etc. and merges them
@@ -960,7 +1012,7 @@ def _resolve_and_merge_input_items(self, items):
 
         :param items: List of dictionaries. Each dictionary describing an input item
 
-        :returns: a dictionary with a dictionary that maps the input DIDs to options
+        :returns: a dictionary that maps the input DIDs to options
                   and a list with a dictionary for each merged download item
 
         :raises InputValidationError: if one of the input items is in the wrong format
@@ -973,70 +1025,51 @@ def _resolve_and_merge_input_items(self, items):
                 logger(logging.WARNING, 'resolve_archives option is deprecated and will be removed in a future release.')
                 item.setdefault('no_resolve_archives', not item.pop('resolve_archives'))
 
-            did = item.get('did', [])
-            if len(did) == 0:
+            if not item.get('did'):
                 if not item.get('filters', {}).get('scope'):
                     logger(logging.DEBUG, item)
                     raise InputValidationError('Item without did and filter/scope')
-                item['did'] = [None]
-            elif not isinstance(did, list):
-                item['did'] = [did]
 
         distinct_keys = ['rse', 'force_scheme', 'nrandom']
         all_resolved_did_strs = set()
 
         did_to_options = {}
         merged_items = []
-        download_info = {'did_to_options': did_to_options,
-                         'merged_items': merged_items}
-
-        while len(items) > 0:
-            item = items.pop()
-
-            filters = item.get('filters', {})
-            item_dids = item.pop('did')
-            if item_dids[0] is None:
-                logger(logging.DEBUG, 'Resolving DIDs by using filter options')
-                item_dids = []
-                scope = filters.pop('scope')
-                for did_name in self.client.list_dids(scope, filters=filters, type='all'):
-                    item_dids.append('%s:%s' % (scope, did_name))
 
+        for item in items:
             base_dir = item.pop('base_dir', '.')
             no_subdir = item.pop('no_subdir', False)
             ignore_checksum = item.pop('ignore_checksum', False)
             new_transfer_timeout = item.pop('transfer_timeout', None)
+            new_transfer_speed_timeout = item.pop('transfer_speed_timeout', None)
+
             resolved_dids = item.setdefault('dids', [])
-            for did_str in item_dids:
-                did_scope, did_name = self._split_did_str(did_str)
-                tmp_did_names = []
-                if '*' in did_name:
-                    filters['name'] = did_name
-                    tmp_did_names = list(self.client.list_dids(did_scope, filters=filters, type='all'))
-                else:
-                    tmp_did_names = [did_name]
 
-                for did_name in tmp_did_names:
+            for did_scope, did_names in self._resolve_one_item_dids(item):
+                for did_name in did_names:
                     resolved_did_str = '%s:%s' % (did_scope, did_name)
                     options = did_to_options.setdefault(resolved_did_str, {})
                     options.setdefault('destinations', set()).add((base_dir, no_subdir))
 
-                    if resolved_did_str in all_resolved_did_strs:
-                        # in this case the DID was already given in another item
-                        # the options of this DID will be ignored and the options of the first item that contained the DID will be used
-                        # another approach would be to compare the options and apply the more relaxed options
-                        logger(logging.DEBUG, 'Ignoring further options of DID: %s' % resolved_did_str)
-                        continue
-
+                    # Merge some options
+                    # The other options of this DID will be inherited from the first item that contained the DID
                     options['ignore_checksum'] = (options.get('ignore_checksum') or ignore_checksum)
+
                     cur_transfer_timeout = options.setdefault('transfer_timeout', None)
                     if cur_transfer_timeout is not None and new_transfer_timeout is not None:
                         options['transfer_timeout'] = max(int(cur_transfer_timeout), int(new_transfer_timeout))
                     elif new_transfer_timeout is not None:
                         options['transfer_timeout'] = int(new_transfer_timeout)
 
-                    resolved_dids.append({'scope': did_scope, 'name': did_name})
-                    all_resolved_did_strs.add(resolved_did_str)
+                    cur_transfer_speed_timeout = options.setdefault('transfer_speed_timeout', None)
+                    if cur_transfer_speed_timeout is not None and new_transfer_speed_timeout is not None:
+                        options['transfer_speed_timeout'] = min(int(cur_transfer_speed_timeout), int(new_transfer_speed_timeout))
+                    elif new_transfer_speed_timeout is not None:
+                        options['transfer_speed_timeout'] = int(new_transfer_speed_timeout)
+
+                    if resolved_did_str not in all_resolved_did_strs:
+                        resolved_dids.append({'scope': did_scope, 'name': did_name})
+                        all_resolved_did_strs.add(resolved_did_str)
 
             if len(resolved_dids) == 0:
                 logger(logging.WARNING, 'An item didnt have any DIDs after resolving the input. Ignoring it.')
@@ -1052,7 +1085,7 @@ def _resolve_and_merge_input_items(self, items):
             if not was_merged:
                 item['dids'] = resolved_dids
                 merged_items.append(item)
-        return download_info
+        return did_to_options, merged_items
 
     def _get_sources(self, merged_items, resolve_archives=True):
         """


Test patch:
diff --git a/lib/rucio/tests/test_bin_rucio.py b/lib/rucio/tests/test_bin_rucio.py
--- a/lib/rucio/tests/test_bin_rucio.py
+++ b/lib/rucio/tests/test_bin_rucio.py
@@ -733,6 +733,32 @@ def test_download_filter(self):
         print(out, err)
         assert re.search(tmp_file1[5:], out) is not None
 
+    def test_download_timeout_options_accepted(self):
+        """CLIENT(USER): Rucio download timeout options """
+        tmp_file1 = file_generator()
+        # add files
+        cmd = 'rucio upload --rse {0} --scope {1} {2}'.format(self.def_rse, self.user, tmp_file1)
+        print(self.marker + cmd)
+        exitcode, out, err = execute(cmd)
+        print(out, err)
+        # download files
+        cmd = 'rucio download --dir /tmp --transfer-timeout 3 --transfer-speed-timeout 1000 {0}:{1}'.format(self.user, tmp_file1[5:])  # triming '/tmp/' from filename
+        print(self.marker + cmd)
+        exitcode, out, err = execute(cmd)
+        print(out, err)
+        assert not err
+        # search for the files with ls
+        cmd = 'ls /tmp/'    # search in /tmp/
+        print(self.marker + cmd)
+        exitcode, out, err = execute(cmd)
+        print(out, err)
+
+        # Check that PFN the transfer-speed-timeout option is not accepted for --pfn
+        cmd = 'rucio -v download --rse {0} --transfer-speed-timeout 1 --pfn http://a.b.c/ {1}:{2}'.format(self.def_rse, self.user, tmp_file1)
+        exitcode, out, err = execute(cmd)
+        print(out, err)
+        assert "Download with --pfn doesn't support --transfer-speed-timeout" in err
+
     def test_download_metalink_file(self):
         """CLIENT(USER): Rucio download with metalink file"""
         metalink_file_path = generate_uuid()
diff --git a/lib/rucio/tests/test_download.py b/lib/rucio/tests/test_download.py
--- a/lib/rucio/tests/test_download.py
+++ b/lib/rucio/tests/test_download.py
@@ -25,6 +25,7 @@
 
 import logging
 import shutil
+from unittest.mock import patch, MagicMock, ANY
 from tempfile import TemporaryDirectory
 from zipfile import ZipFile
 
@@ -392,3 +393,61 @@ def test_norandom_respected(rse_factory, file_factory, download_client, root_acc
         nrandom = 2
         result = download_client.download_dids([{'did': dataset_did_str, 'nrandom': nrandom, 'base_dir': tmp_dir}])
         assert len(result) == nrandom
+
+
+def test_transfer_timeout(rse_factory, file_factory, download_client):
+    rse, _ = rse_factory.make_posix_rse()
+    did = file_factory.upload_test_file(rse)
+    did_str = '%s:%s' % (did['scope'], did['name'])
+
+    mocks_get = []
+
+    # Wraps PosixProtocol and allows to verify with which parameters get() was called
+    class CallCounterPosixProtocol(PosixProtocol):
+        def __init__(self, *args, mocks_get=mocks_get, **kwargs):
+            super(CallCounterPosixProtocol, self).__init__(*args, **kwargs)
+            # Every instance of the class will now have its `get` method wrapped in a MagicMock
+            self.get = MagicMock(wraps=self.get)
+            mocks_get.append(self.get)
+
+    with patch('rucio.rse.protocols.posix.Default', CallCounterPosixProtocol):
+        # if none of timeout parameters set, the default value is used
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=360)
+
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir, 'transfer_timeout': 10}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=10)
+
+        # transfer_timeout set. transfer_speed_timeout is ignored.
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir, 'transfer_timeout': 5, 'transfer_speed_timeout': 1}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=5)
+
+        # 60s static + 2bytes(file size) at 1bps = 62s
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir, 'transfer_speed_timeout': 1}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=62)
+
+        # 60s static + 2bytes(file size) at high speed = 60s
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir, 'transfer_speed_timeout': 10000}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=60)
+
+        # transfer_timeout=0 means no timeout
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir, 'transfer_timeout': 0}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=0)
+
+        # transfer_speed_timeout=0 is ignored
+        with TemporaryDirectory() as tmp_dir:
+            mocks_get.clear()
+            download_client.download_dids([{'did': did_str, 'base_dir': tmp_dir, 'transfer_speed_timeout': 0}])
+            mocks_get[0].assert_called_with(ANY, ANY, transfer_timeout=60)