Repository: nipy/nipype

Issue:
Boto do not get the chance to connect to S3 bucket using external authentication
### Summary
When using IAM roles on EC2 instances, the DataSink interface does not try to use given IAM role access for the instance to access S3 buckets. It goes straight to an anonymous connection. However, there is no indication in the environment that the role is present, so it can only be accessed by trying to connect to the bucket.

It is a scenario that I had to deal with when running workflows on AWS Batch, by change the Docker image / nipype code manually (yikes!)

### Actual behavior
If AWS Access Key / Secret are not defined via a file or environment variables, it tries to connect anonymously.

### Expected behavior
If AWS Access Key / Secret are not defined via a file or environment variables, it should give a try to IAM Roles before setting the anonymous connection.

### How to replicate the behavior

https://gist.github.com/anibalsolon/f7586a85942792432e98470c512588b9

After creating this, you need to create an EC2 instance w/ this profile, and run a workflow w/ DataSink without informing Access Key & Secret. It will try to connect anonymously.

### Script/Workflow details
Just a DataSink node for S3 usage.

### Platform details:

```
{
  'pkg_path': '/home/anibalsolon/Documents/nipype/anibalsolon/nipype',
  'commit_source': 'repository',
  'commit_hash': 'ad93735',
  'nipype_version': '1.1.0-dev+gad93735',
  'sys_version': '3.6.4 |Anaconda custom (64-bit)| (default, Jan 16 2018, 18:10:19) \n[GCC 7.2.0]',
  'sys_executable': '/opt/anaconda3/bin/python',
  'sys_platform': 'linux',
  'numpy_version': '1.14.0',
  'scipy_version': '1.0.0',
  'networkx_version': '2.1',
  'nibabel_version': '2.2.1',
  'traits_version': '4.6.0'
}
1.1.0-dev+gad93735
```

### Execution environment

Choose one
- Container [Tag: ce73e99f008e / py36]
- My python environment outside container



Discussion:

Resolving patch:
diff --git a/nipype/interfaces/io.py b/nipype/interfaces/io.py
--- a/nipype/interfaces/io.py
+++ b/nipype/interfaces/io.py
@@ -123,6 +123,35 @@ def add_traits(base, names, trait_type=None):
     return base
 
 
+def _get_head_bucket(s3_resource, bucket_name):
+    """ Try to get the header info of a bucket, in order to 
+    check if it exists and its permissions
+    """
+
+    import botocore
+
+    # Try fetch the bucket with the name argument
+    try:
+        s3_resource.meta.client.head_bucket(Bucket=bucket_name)
+    except botocore.exceptions.ClientError as exc:
+        error_code = int(exc.response['Error']['Code'])
+        if error_code == 403:
+            err_msg = 'Access to bucket: %s is denied; check credentials'\
+                        % bucket_name
+            raise Exception(err_msg)
+        elif error_code == 404:
+            err_msg = 'Bucket: %s does not exist; check spelling and try '\
+                        'again' % bucket_name
+            raise Exception(err_msg)
+        else:
+            err_msg = 'Unable to connect to bucket: %s. Error message:\n%s'\
+                        % (bucket_name, exc)
+    except Exception as exc:
+        err_msg = 'Unable to connect to bucket: %s. Error message:\n%s'\
+                    % (bucket_name, exc)
+        raise Exception(err_msg)
+
+
 class IOBase(BaseInterface):
     def _run_interface(self, runtime):
         return runtime
@@ -536,43 +565,35 @@ def _fetch_bucket(self, bucket_name):
             session = boto3.session.Session(
                 aws_access_key_id=aws_access_key_id,
                 aws_secret_access_key=aws_secret_access_key)
-            s3_resource = session.resource('s3', use_ssl=True)
 
-        # Otherwise, connect anonymously
         else:
-            iflogger.info('Connecting to AWS: %s anonymously...', bucket_name)
+            iflogger.info('Connecting to S3 bucket: %s with IAM role...',
+                          bucket_name)
+
+            # Lean on AWS environment / IAM role authentication and authorization
             session = boto3.session.Session()
-            s3_resource = session.resource('s3', use_ssl=True)
-            s3_resource.meta.client.meta.events.register(
-                'choose-signer.s3.*', botocore.handlers.disable_signing)
 
-        # Explicitly declare a secure SSL connection for bucket object
-        bucket = s3_resource.Bucket(bucket_name)
+        s3_resource = session.resource('s3', use_ssl=True)
 
         # And try fetch the bucket with the name argument
         try:
-            s3_resource.meta.client.head_bucket(Bucket=bucket_name)
-        except botocore.exceptions.ClientError as exc:
-            error_code = int(exc.response['Error']['Code'])
-            if error_code == 403:
-                err_msg = 'Access to bucket: %s is denied; check credentials'\
-                          % bucket_name
-                raise Exception(err_msg)
-            elif error_code == 404:
-                err_msg = 'Bucket: %s does not exist; check spelling and try '\
-                          'again' % bucket_name
-                raise Exception(err_msg)
-            else:
-                err_msg = 'Unable to connect to bucket: %s. Error message:\n%s'\
-                          % (bucket_name, exc)
+            _get_head_bucket(s3_resource, bucket_name)
         except Exception as exc:
-            err_msg = 'Unable to connect to bucket: %s. Error message:\n%s'\
-                      % (bucket_name, exc)
-            raise Exception(err_msg)
+
+            # Try to connect anonymously
+            s3_resource.meta.client.meta.events.register(
+                'choose-signer.s3.*', botocore.handlers.disable_signing)
+
+            iflogger.info('Connecting to AWS: %s anonymously...', bucket_name)
+            _get_head_bucket(s3_resource, bucket_name)
+
+        # Explicitly declare a secure SSL connection for bucket object
+        bucket = s3_resource.Bucket(bucket_name)
 
         # Return the bucket
         return bucket
 
+
     # Send up to S3 method
     def _upload_to_s3(self, bucket, src, dst):
         '''
@@ -592,10 +613,8 @@ def _upload_to_s3(self, bucket, src, dst):
         s3_prefix = s3_str + bucket.name
 
         # Explicitly lower-case the "s3"
-        if dst.lower().startswith(s3_str):
-            dst_sp = dst.split('/')
-            dst_sp[0] = dst_sp[0].lower()
-            dst = '/'.join(dst_sp)
+        if dst[:len(s3_str)].lower() == s3_str:
+            dst = s3_str + dst[len(s3_str):]
 
         # If src is a directory, collect files (this assumes dst is a dir too)
         if os.path.isdir(src):


Test patch: