Repository: boto/boto

Issue:
Config not correctly parsing aws credentials file in python 3.5
Looks like there is some issue with how the custom `Config` class is overriding the `ConfigParser` class in python 3.5.1 on Mac OSX 10.10.5. I am not able to read my aws credentials from the default `~/.aws/credentials` file if I use the boto `Config` class, but am able to if I use the `boto.compat.ConfigParser` class.

```
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from boto.compat import ConfigParser
>>> c = ConfigParser()
>>> c.read("/Users/asgoel/.aws/credentials")
['/Users/asgoel/.aws/credentials']
>>> c.get("default", "aws_access_key_id")
RETURNS_AWS_ACCESS_KEY
>>> from boto.pyami.config import Config
>>> c1 = Config()
c1.read("/Users/ashu/.aws/credentials")
['/Users/ashu/.aws/credentials']
>>> print(c1.get("default", "aws_access_key_id"))
None
```

Makes running applications locally a little bit annoying but not a huge issue, just something I thought I would point out to see if anyone is having similar problems. I couple of others within my organization are facing the same problem.



Discussion:
I too, am experiencing this issue. It seems to be the upgrade to Python 3.5.1. It works fine in my Python 3.5.0 virtualenv. Is there a proposed fix and ETA?

I just spent some time debugging this issue after encountering it, too (I should have checked the issues first). It looks like Python 3.5. adds some default options to the `get` method of the `ConfigParser` class, and these confuse boto's `Config` (which derives from `ConfigParser`). For the record, here is a minimal example that will reproduce the issue (the class `Config` is set up in the same way that boto does things)

``` python
from configparser import ConfigParser


class Config(ConfigParser):

    def __init__(self):
        ConfigParser.__init__(self)

    def get(self, section, name, default=None):
        try:
            val = ConfigParser.get(self, section, name)
        except:
            val = default
        return val


c = Config()
c['FOO'] = {}
c['FOO']['bar'] = 'baz'

assert c.get('FOO', 'bar') == 'baz'
```

Internally, the call to `ConfigParser.get` raises a `TypeError` because of the aforementioned extra arguments, but this gets swallowed by the catch-all try/except around it so that `None` always gets returned.

I think the fix would be to add the new default arguments to the methods on boto's `Config` class, but I haven't tried this out. Also, it would probably be good to turn the blanket `except:` into an `except KeyError`.

I just debugged the issue, and arrived at the same conclusion. However, I think this should be fixed by replacing the inheritance with composition:

```
--- a/boto/pyami/config.py
+++ b/boto/pyami/config.py
@@ -49,12 +49,12 @@ elif 'BOTO_PATH' in os.environ:
         BotoConfigLocations.append(expanduser(path))


-class Config(ConfigParser):
+class Config(object):

     def __init__(self, path=None, fp=None, do_load=True):
         # We don't use ``super`` here, because ``ConfigParser`` still uses
         # old-style classes.
-        ConfigParser.__init__(self, {'working_dir': '/mnt/pyami',
+        self._impl = ConfigParser({'working_dir': '/mnt/pyami',
                                          'debug': '0'})
         if do_load:
             if path:
@@ -70,6 +70,9 @@ class Config(ConfigParser):
                 except IOError:
                     warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)

+    def __getattr__(self, name):
+        return getattr(self._impl, name)
+
     def load_credential_file(self, path):
         """Load a credential file as is setup like the Java utilities"""
         c_data = StringIO()
@@ -139,21 +142,21 @@ class Config(ConfigParser):

     def get(self, section, name, default=None):
         try:
-            val = ConfigParser.get(self, section, name)
+            val = self._impl.get(section, name)
         except:
             val = default
         return val

     def getint(self, section, name, default=0):
         try:
-            val = ConfigParser.getint(self, section, name)
+            val = self._impl.getint(section, name)
         except:
             val = int(default)
         return val

     def getfloat(self, section, name, default=0.0):
         try:
-            val = ConfigParser.getfloat(self, section, name)
+            val = self._impl.getfloat(section, name)
         except:
             val = float(default)
         return val
```

(Of course the naked `except`s should also be changed, but I didn't do it since it wasn't necessary to demonstrate the fix.)

> I think this should be fixed by replacing the inheritance with composition:

I agree, that is the way to go. 


Resolving patch:
diff --git a/boto/compat.py b/boto/compat.py
--- a/boto/compat.py
+++ b/boto/compat.py
@@ -61,12 +61,13 @@
     # StandardError was removed, so use the base exception type instead
     StandardError = Exception
     long_type = int
-    from configparser import ConfigParser
+    from configparser import ConfigParser, NoOptionError, NoSectionError
     unquote_str = unquote_plus
 else:
     StandardError = StandardError
     long_type = long
     from ConfigParser import SafeConfigParser as ConfigParser
+    from ConfigParser import NoOptionError, NoSectionError
 
     def unquote_str(value, encoding='utf-8'):
         # In python2, unquote() gives us a string back that has the urldecoded
diff --git a/boto/pyami/config.py b/boto/pyami/config.py
--- a/boto/pyami/config.py
+++ b/boto/pyami/config.py
@@ -26,7 +26,7 @@
 
 import boto
 
-from boto.compat import expanduser, ConfigParser, StringIO
+from boto.compat import expanduser, ConfigParser, NoOptionError, NoSectionError, StringIO
 
 
 # By default we use two locations for the boto configurations,
@@ -49,13 +49,11 @@
         BotoConfigLocations.append(expanduser(path))
 
 
-class Config(ConfigParser):
+class Config(object):
 
     def __init__(self, path=None, fp=None, do_load=True):
-        # We don't use ``super`` here, because ``ConfigParser`` still uses
-        # old-style classes.
-        ConfigParser.__init__(self, {'working_dir': '/mnt/pyami',
-                                         'debug': '0'})
+        self._parser = ConfigParser({'working_dir': '/mnt/pyami',
+                                     'debug': '0'})
         if do_load:
             if path:
                 self.load_from_path(path)
@@ -70,6 +68,21 @@ def __init__(self, path=None, fp=None, do_load=True):
                 except IOError:
                     warnings.warn('Unable to load AWS_CREDENTIAL_FILE (%s)' % full_path)
 
+    def __setstate__(self, state):
+        # There's test that verify that (transitively) a Config
+        # object can be pickled.  Now that we're storing a _parser
+        # attribute and relying on __getattr__ to proxy requests,
+        # we need to implement setstate to ensure we don't get
+        # into recursive loops when looking up _parser when
+        # this object is unpickled.
+        self._parser = state['_parser']
+
+    def __getattr__(self, name):
+        return getattr(self._parser, name)
+
+    def has_option(self, *args, **kwargs):
+        return self._parser.has_option(*args, **kwargs)
+
     def load_credential_file(self, path):
         """Load a credential file as is setup like the Java utilities"""
         c_data = StringIO()
@@ -116,21 +129,21 @@ def save_system_option(self, section, option, value):
     def get_instance(self, name, default=None):
         try:
             val = self.get('Instance', name)
-        except:
+        except (NoOptionError, NoSectionError):
             val = default
         return val
 
     def get_user(self, name, default=None):
         try:
             val = self.get('User', name)
-        except:
+        except (NoOptionError, NoSectionError):
             val = default
         return val
 
     def getint_user(self, name, default=0):
         try:
             val = self.getint('User', name)
-        except:
+        except (NoOptionError, NoSectionError):
             val = default
         return val
 
@@ -139,24 +152,21 @@ def get_value(self, section, name, default=None):
 
     def get(self, section, name, default=None):
         try:
-            val = ConfigParser.get(self, section, name)
-        except:
-            val = default
-        return val
+            return self._parser.get(section, name)
+        except (NoOptionError, NoSectionError):
+            return default
 
     def getint(self, section, name, default=0):
         try:
-            val = ConfigParser.getint(self, section, name)
-        except:
-            val = int(default)
-        return val
+            return self._parser.getint(section, name)
+        except (NoOptionError, NoSectionError):
+            return int(default)
 
     def getfloat(self, section, name, default=0.0):
         try:
-            val = ConfigParser.getfloat(self, section, name)
-        except:
-            val = float(default)
-        return val
+            return self._parser.getfloat(section, name)
+        except (NoOptionError, NoSectionError):
+            return float(default)
 
     def getbool(self, section, name, default=False):
         if self.has_option(section, name):


Test patch:
diff --git a/tests/unit/pyami/__init__.py b/tests/unit/pyami/__init__.py
new file mode 100644
diff --git a/tests/unit/pyami/test_config.py b/tests/unit/pyami/test_config.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/pyami/test_config.py
@@ -0,0 +1,63 @@
+# Copyright (c) 2016 Amazon.com, Inc. or its affiliates.  All Rights Reserved
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish, dis-
+# tribute, sublicense, and/or sell copies of the Software, and to permit
+# persons to whom the Software is furnished to do so, subject to the fol-
+# lowing conditions:
+#
+# The above copyright notice and this permission notice shall be included
+# in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABIL-
+# ITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
+# SHALL THE AUTHOR BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+#
+from tests.compat import mock, unittest
+
+from boto.pyami import config
+from boto.compat import StringIO
+
+
+class TestCanLoadConfigFile(unittest.TestCase):
+    maxDiff = None
+
+    def setUp(self):
+        self.file_contents = StringIO()
+        file_contents = StringIO(
+            '[Boto]\n'
+            'https_validate_certificates = true\n'
+            'other = false\n'
+            'http_socket_timeout = 1\n'
+            '[Credentials]\n'
+            'aws_access_key_id=foo\n'
+            'aws_secret_access_key=bar\n'
+        )
+        self.config = config.Config(fp=file_contents)
+
+    def test_can_get_bool(self):
+        self.assertTrue(
+            self.config.getbool('Boto', 'https_validate_certificates'))
+        self.assertFalse(self.config.getbool('Boto', 'other'))
+        self.assertFalse(self.config.getbool('Boto', 'does-not-exist'))
+
+    def test_can_get_int(self):
+        self.assertEqual(self.config.getint('Boto', 'http_socket_timeout'), 1)
+        self.assertEqual(self.config.getint('Boto', 'does-not-exist'), 0)
+        self.assertEqual(
+            self.config.getint('Boto', 'does-not-exist', default=20), 20)
+
+    def test_can_get_strings(self):
+        self.assertEqual(
+            self.config.get('Credentials', 'aws_access_key_id'), 'foo')
+        self.assertIsNone(
+            self.config.get('Credentials', 'no-exist'))
+        self.assertEqual(
+            self.config.get('Credentials', 'no-exist', 'default-value'),
+            'default-value')