Repository: spotify/luigi

Issue:
S3's _get_s3_config() restricts the use of [DEFAULT] ConfigParser section
<!---
Currently, there are no strict procedures or guidelines for submitting issues.
In short, please just use common sense.

Common sense includes this at bare-minimum:

 * search for similar issues posted before creating a new issue.
 * Use markdown to format all code/logs. Issues which are hard to read
   when rendered on GitHub might be closed with a friendly reminder of this.

Also, add steps to reproduce the bug, if applicable. Sample code would be nice too :)

For more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->
## Problem
S3Client implementation of config options prevents the use of `ConfigParser`'s `[DEFAULT]` section.

Due to the implementation of `luigi/contrib/s3.py`'s auto-fill options using all key-value pairs existing in the `[s3]` section within one's specified luigi configuration file, errors are thrown when attempting to utilize Python's ConfigParser `[DEFAULT]` section.

Functionality appears to have been added in PR #500

### Presence of Bug
From what i can tell by searching the repo for `configuration.get_config().items`, S3 is the only culprit for this issue.

## Question (fbo solution)
I understand the motivation here is to be able to specify s3 boto connection options via the Luigi config, but i'm not aware of a best strategy to limit either a list of approved keys to keep or if there's a way for the ConfigParser to parse and interpolate only those keys within the explicit `[s3]` path (ignoring those in `[DEFAULT]`).

I'm happy to tackle this, but suggestions are appreciated and welcome.

## Source Code at Fault
```python
class S3Client(FileSystem):
    _s3 = None 
    
    def __init__(self, aws_access_key_id=None, aws_secret_access_key=None, **kwargs):
        from boto.s3.key import Key
        options = self._get_s3_config()  # pulls all s3 section vars, including DEFAULT
        options.update(kwargs)
        if aws_access_key_id:
            options['aws_access_key_id'] = aws_access_key_id
        if aws_secret_access_key:
            options['aws_secret_access_key'] = aws_secret_access_key
        
        self.Key = Key
        self._options = options  # all s3 and DEFAULT key/value pairs

    @property
    def s3(self):
        # only import boto when needed to allow top-lvl s3 module import
        import boto
        import boto.s3.connection

        options = dict(self._options)  # var of interest

        if self._s3:
            return self._s3

        aws_access_key_id = options.get('aws_access_key_id')
        aws_secret_access_key = options.get('aws_secret_access_key')

        # Removing key args would break backwards compability
        role_arn = options.get('aws_role_arn')
        role_session_name = options.get('aws_role_session_name')

        aws_session_token = None

        if role_arn and role_session_name:
            from boto import sts

            sts_client = sts.STSConnection()
            assumed_role = sts_client.assume_role(role_arn, role_session_name)
            aws_secret_access_key = assumed_role.credentials.secret_key
            aws_access_key_id = assumed_role.credentials.access_key
            aws_session_token = assumed_role.credentials.session_token

        for key in ['aws_access_key_id', 'aws_secret_access_key', 'aws_role_session_name', 'aws_role_arn']:
            if key in options:
                options.pop(key)
        self._s3 = boto.s3.connection.S3Connection(aws_access_key_id,
                                                   aws_secret_access_key,
                                                   security_token=aws_session_token,
                                                   **options)  # passes all key/value pairs (except auth) to boto - may cause error
        return self._s3
...

    def _get_s3_config(self, key=None):
        try:
            config = dict(configuration.get_config().items('s3'))  # pulls all k/v from s3 and DEFAULT
        except NoSectionError:
            return {}
        # So what ports etc can be read without us having to specify all dtypes
        for k, v in six.iteritems(config):
            try:
                config[k] = int(v)
            except ValueError:
                pass
        if key:
            return config.get(key)
        return config
...
```

### Erroneous Luigi Code
luigi.cfg
```
[DEFAULT]
path_base: /Users/dstadther

[core]
...
logging_conf_file: %(path_base)s/this_project/logging.conf

[bucket]
load: s3://my.s3.load.bucket/

[path]
sql: %(path_base)s/sql

[s3]
aws_access_key_id: foo
aws_secret_access_key: bar
calling_format: boto.s3.connection.OrdinaryCallingFormat

[scheduler]
state-path: %(path_base)s/out/luigid_state.pkl

...

```

luigi example script (`test_s3_default.py`)
```python
import luigi
from luigi import mock
from luigi.contrib import s3


class bucket(luigi.Config):
    load = luigi.Parameter()


class TestDefault(luigi.Task):
    client = s3.S3Client()

    def output(self):
        return mock.MockTarget('')

    def run(self):
        for i in self.client.listdir(bucket().load):
            print(i)
        self.output().open('w').close()


if __name__ == '__main__':
    luigi.run()

```

shell
```sh
cd /Users/dstadther/this_project/scripts; /Users/dstadther/venv/this_project/bin/python test_s3_default.py TestDefault
```

Error stack trace
```sh
Traceback (most recent call last):
  File "/Users/dstadther/venv/this_project/lib/python2.7/site-packages/luigi/worker.py", line 191, in run
    new_deps = self._run_get_new_deps()
  File "/Users/dstadther/venv/this_project/lib/python2.7/site-packages/luigi/worker.py", line 129, in _run_get_new_deps
    task_gen = self.task.run()
  File "test_s3_default.py", line 13, in run
    for i in self.client.listdir(bucket().load):
  File "/Users/dstadther/venv/this_project/lib/python2.7/site-packages/luigi/contrib/s3.py", line 498, in listdir
    s3_bucket = self.s3.get_bucket(bucket, validate=True)
  File "/Users/dstadther/venv/this_project/lib/python2.7/site-packages/luigi/contrib/s3.py", line 126, in s3
    **options)
TypeError: __init__() got an unexpected keyword argument 'path_base'
```



Discussion:
Proposal: Make the assumption that S3 connection options would NEVER be included in the `DEFAULT` section, but perhaps interpolation is necessary. If Key exists in both, ensure that values differ.

Something like:
```python
    def _get_s3_config(self, key=None):
        defaults = dict(configuration.get_config().defaults())
        try:
            config = dict(configuration.get_config().items('s3'))  # pulls all k/v from s3 and DEFAULT
        except NoSectionError:
            return {}
        # So what ports etc can be read without us having to specify all dtypes
        for k, v in six.iteritems(config):
            try:
                config[k] = int(v)
            except ValueError:
                pass
        if key:
            return config.get(key)

        section_only = {k: v for k, v in config.items() if k not in defaults or v != defaults[k]}
        return section_only
```

_Untested_

Thoughts?

Resolving patch:
diff --git a/luigi/contrib/s3.py b/luigi/contrib/s3.py
--- a/luigi/contrib/s3.py
+++ b/luigi/contrib/s3.py
@@ -565,6 +565,7 @@ def mkdir(self, path, parents=True, raise_if_exists=False):
         return self.put_string("", self._add_path_delimiter(path))
 
     def _get_s3_config(self, key=None):
+        defaults = dict(configuration.get_config().defaults())
         try:
             config = dict(configuration.get_config().items('s3'))
         except NoSectionError:
@@ -577,7 +578,8 @@ def _get_s3_config(self, key=None):
                 pass
         if key:
             return config.get(key)
-        return config
+        section_only = {k: v for k, v in config.items() if k not in defaults or v != defaults[k]}
+        return section_only
 
     def _path_to_bucket_and_key(self, path):
         (scheme, netloc, path, query, fragment) = urlsplit(path)


Test patch: