Repository: nipy/nipype

Issue:
DataSink S3 support does not work
### Summary
The DataSink interface does not work.

### Actual behavior
Only the local copy works. Oddly, when used in a workflow, DataSink creates a local directory named `s3:/` instead.

### Expected behavior
DataSink creates objects in S3 buckets when base_directory starts with `s3://`.

### How to replicate the behavior
```
ds = DataSink()

ds.inputs.base_directory = 's3://kimsin-test'
ds.inputs.creds_path = os.path.abspath('aws.csv')
ds.inputs.encrypt_bucket_keys = True
ds.inputs.local_copy = os.path.abspath('log')
setattr(ds.inputs, 'out_s3', 'DELETE.me')

ds.run()
```
This basic S3 test code does not create the desired output in the S3 bucket.

### Platform details:
```
{'commit_hash': '%h',
 'commit_source': 'archive substitution',
 'networkx_version': '2.4',
 'nibabel_version': '2.5.1',
 'nipype_version': '1.3.0-rc1',
 'numpy_version': '1.17.3',
 'pkg_path': '/opt/miniconda-latest/envs/neuro/lib/python3.6/site-packages/nipype',
 'scipy_version': '1.3.1',
 'sys_executable': '/opt/miniconda-latest/envs/neuro/bin/python',
 'sys_platform': 'linux',
 'sys_version': '3.6.7 | packaged by conda-forge | (default, Jul  2 2019, '
                '02:18:42) \n'
                '[GCC 7.3.0]',
 'traits_version': '5.1.2'}
```

### Execution environment
- My python environment inside container


Discussion:
For things like this, I'll often debug as follows:

```Python
>>> from nipype.interfaces.io import *
>>> ds = DataSink()
>>> ds.inputs.base_directory = 's3://kimsin-test'
>>> ds.inputs.creds_path = os.path.abspath('aws.csv')
>>> ds.inputs.encrypt_bucket_keys = True
>>> ds.inputs.local_copy = os.path.abspath('log')
>>> ds.inputs.out_s3 = 'DELETE.me'
>>> self = ds
```

Then just manually walk through the `_list_outputs` method to see where things break down.

https://github.com/nipy/nipype/blob/1a869991adc024577536689d557fc748c764f15d/nipype/interfaces/io.py#L944-L1067

I'm sure there's a way to do this in pdb that will save you typing/pasting, if you prefer...
Hi,

this may be related to the use of `traits.Directory` in the input spec, as opposed to `Str`. Maybe `traits.Directory` is stripping out the double `/` in the S3 url.

Edit: Looks like pathlib.Path is the culprit:

https://github.com/nipy/nipype/blob/1a869991adc024577536689d557fc748c764f15d/nipype/interfaces/base/traits_extension.py#L120-L126

```
Python 3.7.4 (default, Aug 13 2019, 15:17:50) 

In[3]: from pathlib import Path
Path('s3://this/is/a/test')
Out[3]: PosixPath('s3:/this/is/a/test')
```

Good find!

I suppose the simplest fix would be changing `DataSink._check_s3_base_dir` to accommodate this quirk. The other option of editing `traits.Directory` has too much potential to affect other interfaces.

I will submit a quick PR after testing.
I think `base_directory` should not be a `traits.Directory`. Why not just changing it to `traits.Str`?
@oesteban I am not quite sure about all the implications, but doing so would skip whatever cleanup `pathlib.Path` does on DataSink base_directory. This notably includes various backslash shenanigans.

For now, I will leave the discussion on what format DataSink expects as inputs up to others.

Resolving patch:
diff --git a/nipype/interfaces/io.py b/nipype/interfaces/io.py
--- a/nipype/interfaces/io.py
+++ b/nipype/interfaces/io.py
@@ -212,12 +212,12 @@ class DataSinkInputSpec(DynamicTraitedSpec, BaseInterfaceInputSpec):
     """
 
     # Init inputspec data attributes
-    base_directory = Directory(desc="Path to the base directory for storing data.")
+    base_directory = Str(desc="Path to the base directory for storing data.")
     container = Str(desc="Folder within base directory in which to store output")
     parameterization = traits.Bool(
         True, usedefault=True, desc="store output in parametrized structure"
     )
-    strip_dir = Directory(desc="path to strip out of filename")
+    strip_dir = Str(desc="path to strip out of filename")
     substitutions = InputMultiPath(
         traits.Tuple(Str, Str),
         desc=(
@@ -440,7 +440,6 @@ def _check_s3_base_dir(self):
             is not a valid S3 path, defaults to '<N/A>'
         """
 
-        # Init variables
         s3_str = "s3://"
         bucket_name = "<N/A>"
         base_directory = self.inputs.base_directory
@@ -449,22 +448,10 @@ def _check_s3_base_dir(self):
             s3_flag = False
             return s3_flag, bucket_name
 
-        # Explicitly lower-case the "s3"
-        if base_directory.lower().startswith(s3_str):
-            base_dir_sp = base_directory.split("/")
-            base_dir_sp[0] = base_dir_sp[0].lower()
-            base_directory = "/".join(base_dir_sp)
-
-        # Check if 's3://' in base dir
-        if base_directory.startswith(s3_str):
-            # Expects bucket name to be 's3://bucket_name/base_dir/..'
-            bucket_name = base_directory.split(s3_str)[1].split("/")[0]
-            s3_flag = True
-        # Otherwise it's just a normal datasink
-        else:
-            s3_flag = False
+        s3_flag = base_directory.lower().startswith(s3_str)
+        if s3_flag:
+            bucket_name = base_directory[len(s3_str):].partition('/')[0]
 
-        # Return s3_flag
         return s3_flag, bucket_name
 
     # Function to return AWS secure environment variables
@@ -618,13 +605,12 @@ def _upload_to_s3(self, bucket, src, dst):
 
         from botocore.exceptions import ClientError
 
-        # Init variables
         s3_str = "s3://"
         s3_prefix = s3_str + bucket.name
 
         # Explicitly lower-case the "s3"
-        if dst[: len(s3_str)].lower() == s3_str:
-            dst = s3_str + dst[len(s3_str) :]
+        if dst.lower().startswith(s3_str):
+            dst = s3_str + dst[len(s3_str):]
 
         # If src is a directory, collect files (this assumes dst is a dir too)
         if os.path.isdir(src):


Test patch: