Repository: rucio/rucio

Issue:
Rucio download is not able to download files if the scope or name contains "/"
Motivation
----------
 rucio download is not able to download files if the scope or name contains "/"







Discussion:
You skipped the 'modifications' section in the description :P
So how should this be resolved?
Is it related to #1393 ?
Using urllib.quote_plus() for all but containers?

I don't think thjis is related to #1393 - There it is specifically if the trailing `/` is mentioned in the did name, like ATLAS prodsys was doing this in the past. Here the error must be something else. 
I think this is rather an list_replicas issue then. But if the physical filename or the dataset name contains a "/" it will be problematic because it's the directory separator. We could change the download client to replace it by another character or create a subdirectory for each "/" but I think the best thing would be not to have "/" in the LFNs :sweat_smile: 
Creating subdirectories is fine. This is in fact how these files are stored on storage. The LFN just includes these structural directories. There is nothing wrong with having `/` in the LFN.
I would suggest we just create subdirectories. @TWAtGH can you please try and confirm the exact issue. If it is in list_replicas then this goes over to @mlassnig - but we should narrow down what the actual issue is.
Looks like the download is already implemented like this. But it's difficult to try because the upload doesn't support this. I assume that most client functions that do an REST API call won't work using a DID containing a slash.
A little bit more info about the exact issue would be nice @cserf and how can I create a DID containing a slash?
Also how would double slashes be resolved? Just a single directory I guess.
Ok in fact it was the server which rejected the upload due to a missing `AllowEncodedSlashes On`
But the server still don't allow to insert slashed DIDs. I think it happens in the `add_replicas` call:
```
2019-11-19 17:25:17,488	DEBUG	File DID does not exist
2019-11-19 17:25:19,032	ERROR	Provided object does not match schema.
Details: Problem validating dids : u'slash/lfn01' does not match '^[A-Za-z0-9][A-Za-z0-9\\.\\-\\_]{1,250}$'

Failed validating 'pattern' in schema['items']['properties']['name']:
    {'description': 'Data Identifier name',
     'pattern': '^[A-Za-z0-9][A-Za-z0-9\\.\\-\\_]{1,250}$',
     'type': 'string'}

On instance[0]['name']:
    u'slash/lfn01'
```
Yes, you have to try this with a schema which allows slashes. Eg. the `belleii` schema.
@TWAtGH is there any news on this? 
Traceback :
```
rucio -v download test:/grid/belle/ddm/functional_tests/release-0x-0y-0z/DB00000xxx/2020-02-11/MCyy/26903/dst
2020-02-11 05:54:32,806 INFO    Processing 1 item(s) for input
2020-02-11 05:54:32,806 DEBUG   num_unmerged_items=1; num_dids=1; num_merged_items=1
2020-02-11 05:54:32,806 INFO    Getting sources of DIDs
2020-02-11 05:54:32,806 DEBUG   schemes: None
2020-02-11 05:54:32,806 DEBUG   rse_expression: *\istape=true
2020-02-11 05:54:32,806 DEBUG   num DIDs for list_replicas call: 1
2020-02-11 05:54:34,649 DEBUG   num resolved files: 2
2020-02-11 05:54:34,655 DEBUG   "unzip -v" returned with exitcode 0
2020-02-11 05:54:34,661 DEBUG   "tar --version" returned with exitcode 0
2020-02-11 05:54:34,661 DEBUG   num list_replicas calls: 1
2020-02-11 05:54:34,661 DEBUG   Queueing file: test:/grid/belle/ddm/functional_tests/release-0x-0y-0z/DB00000xxx/2020-02-11/MCyy/26903/dst/bf55cdc6f73e473396c3476ca7047384
2020-02-11 05:54:34,661 DEBUG   real parents: set(['test:/grid/belle/ddm/functional_tests/release-0x-0y-0z/DB00000xxx/2020-02-11/MCyy/26903/dst'])
2020-02-11 05:54:34,661 DEBUG   options: {'test:/grid/belle/ddm/functional_tests/release-0x-0y-0z/DB00000xxx/2020-02-11/MCyy/26903/dst': {'ignore_checksum': False, 'transfer_timeout': 3600, 'destinations': set([('.', False)])}}
2020-02-11 05:54:34,663 DEBUG   Traceback (most recent call last):
  File "/usr/bin/rucio", line 159, in new_funct
    return function(*args, **kwargs)
  File "/usr/bin/rucio", line 975, in download
    result = download_client.download_dids(items, args.ndownloader, trace_pattern)
  File "/usr/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 280, in download_dids
    input_items = self._prepare_items_for_download(did_to_options, merged_items_with_sources, resolve_archives=resolve_archives)
  File "/usr/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 1158, in _prepare_items_for_download
    paths = [os.path.join(self._prepare_dest_dir(dest[0], dataset_name, file_did_name, dest[1]), file_did_name) for dest in destinations]
  File "/usr/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 1395, in _prepare_dest_dir
    os.makedirs(dest_dir_path)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/grid'

2020-02-11 05:54:34,663 ERROR   [Errno 13] Permission denied: '/grid'
2020-02-11 05:54:34,663 ERROR
Rucio exited with an unexpected/unknown error.
```
@cserf is this still an issue? How did you fix this with belle2?
Still an issue. Not fixed yet in Belle II. We have been using a separate tool for download until now, but at some point we plan to move to rucio download API.
Found the issue. It is due to the behaviour of `os.path.join`
```
Help on function join in module posixpath:

join(a, *p)
    Join two or more pathname components, inserting '/' as needed.
    If any component is an absolute path, all previous path components
    will be discarded.  An empty last part will result in a path that
    ends with a separator.
```


Resolving patch:
diff --git a/lib/rucio/client/downloadclient.py b/lib/rucio/client/downloadclient.py
--- a/lib/rucio/client/downloadclient.py
+++ b/lib/rucio/client/downloadclient.py
@@ -32,6 +32,7 @@
 # - Radu Carpa <radu.carpa@cern.ch>, 2021
 # - Rakshita Varadarajan <rakshitajps@gmail.com>, 2021
 # - David Población Criado <13998309+davidpob99@users.noreply.github.com>, 2021
+# - Cedric Serfon <cedric.serfon@cern.ch>, 2021
 
 from __future__ import division
 
@@ -50,11 +51,12 @@
 from threading import Thread
 
 from rucio.client.client import Client
+from rucio.common.config import config_get
 from rucio.common.exception import (InputValidationError, NoFilesDownloaded, NotAllFilesDownloaded, RucioException)
 from rucio.common.didtype import DIDType
 from rucio.common.pcache import Pcache
 from rucio.common.utils import adler32, detect_client_location, generate_uuid, parse_replicas_from_string, \
-    send_trace, sizefmt, execute, parse_replicas_from_file
+    send_trace, sizefmt, execute, parse_replicas_from_file, extract_scope
 from rucio.common.utils import GLOBALLY_SUPPORTED_CHECKSUMS, CHECKSUM_ALGO_DICT, PREFERRED_CHECKSUM
 from rucio.rse import rsemanager as rsemgr
 from rucio import version
@@ -181,6 +183,7 @@ def __init__(self, client=None, logger=None, tracing=True, check_admin=False, ch
         # tar -C <dest_dir_path> -xf <archive_file_path>  <did_name>
         extract_args = '-C %(dest_dir_path)s -xf %(archive_file_path)s %(file_to_extract)s'
         self.extraction_tools.append(BaseExtractionTool('tar', '--version', extract_args, logger=self.logger))
+        self.extract_scope_convention = config_get('common', 'extract_scope', False, None)
 
     def download_pfns(self, items, num_threads=2, trace_custom_fields={}, traces_copy_out=None):
         """
@@ -233,7 +236,10 @@ def download_pfns(self, items, num_threads=2, trace_custom_fields={}, traces_cop
             item['scope'] = did_scope
             item['name'] = did_name
             item['sources'] = [{'pfn': pfn, 'rse': rse}]
-            dest_file_path = os.path.join(dest_dir_path, did_name)
+            did_path_name = did_name
+            if self.extract_scope_convention and self.extract_scope_convention == 'belleii' and did_name.startswith('/'):
+                did_path_name = did_name[1:]
+            dest_file_path = os.path.join(dest_dir_path, did_path_name)
             item['dest_file_paths'] = [dest_file_path]
             item['temp_file_path'] = '%s.part' % dest_file_path
             options = item.setdefault('merged_options', {})
@@ -1245,7 +1251,10 @@ def _prepare_items_for_download(self, did_to_options, merged_items_with_sources,
 
                     destinations = options['destinations']
                     dataset_scope, dataset_name = self._split_did_str(dataset_did_str)
-                    paths = [os.path.join(self._prepare_dest_dir(dest[0], dataset_name, dest[1]), file_did_name) for dest in destinations]
+                    file_did_path = file_did_name
+                    if self.extract_scope_convention == 'belleii' and file_did_path.startswith('/'):
+                        file_did_path = file_did_name.split('/')[-1]
+                    paths = [os.path.join(self._prepare_dest_dir(dest[0], dataset_name, dest[1]), file_did_path) for dest in destinations]
                     if any(path in all_dest_file_paths for path in paths):
                         raise RucioException("Multiple file items with same destination file path")
 
@@ -1263,7 +1272,10 @@ def _prepare_items_for_download(self, did_to_options, merged_items_with_sources,
                         logger(logging.ERROR, 'No input options available for %s' % file_did_str)
                         continue
                     destinations = options['destinations']
-                    paths = [os.path.join(self._prepare_dest_dir(dest[0], file_did_scope, dest[1]), file_did_name) for dest in destinations]
+                    file_did_path = file_did_name
+                    if self.extract_scope_convention == 'belleii' and file_did_path.startswith('/'):
+                        file_did_path = file_did_name[1:]
+                    paths = [os.path.join(self._prepare_dest_dir(dest[0], file_did_scope, dest[1]), file_did_path) for dest in destinations]
                     if any(path in all_dest_file_paths for path in paths):
                         raise RucioException("Multiple file items with same destination file path")
                     all_dest_file_paths.update(paths)
@@ -1444,11 +1456,15 @@ def _split_did_str(self, did_str):
             did_scope = did[0]
             did_name = did[1]
         elif len(did) == 1:
-            did = did_str.split('.')
-            did_scope = did[0]
-            if did_scope == 'user' or did_scope == 'group':
-                did_scope = '%s.%s' % (did[0], did[1])
-            did_name = did_str
+            if self.extract_scope_convention == 'belleii':
+                scopes = [scope for scope in self.client.list_scopes()]
+                did_scope, did_name = extract_scope(did[0], scopes)
+            else:
+                did = did_str.split('.')
+                did_scope = did[0]
+                if did_scope == 'user' or did_scope == 'group':
+                    did_scope = '%s.%s' % (did[0], did[1])
+                did_name = did_str
         else:
             raise InputValidationError('%s is not a valid DID. To many colons.' % did_str)
 
@@ -1457,8 +1473,7 @@ def _split_did_str(self, did_str):
 
         return did_scope, did_name
 
-    @staticmethod
-    def _prepare_dest_dir(base_dir, dest_dir_name, no_subdir):
+    def _prepare_dest_dir(self, base_dir, dest_dir_name, no_subdir):
         """
         Builds the final destination path for a file and creates the
         destination directory if it's not existent.
@@ -1471,6 +1486,8 @@ def _prepare_dest_dir(base_dir, dest_dir_name, no_subdir):
         :returns: the absolut path of the destination directory
         """
         # append dest_dir_name, if subdir should be used
+        if self.extract_scope_convention == 'belleii' and dest_dir_name.startswith('/'):
+            dest_dir_name = dest_dir_name[1:]
         dest_dir_path = os.path.join(os.path.abspath(base_dir), '' if no_subdir else dest_dir_name)
 
         if not os.path.isdir(dest_dir_path):


Test patch: