Repository: getmoto/moto

Issue:
Python 3.7 issue because of new re.Pattern
Hi,

after upgrading to Python 3.7, Moto raises the following exception when I run my tests:
`AttributeError: 're.Pattern' object has no attribute 'decode'`

Python Version: 3.7
Moto Version: 1.3.3 

I already looked through the code and I think I found the cause for this (see below).

But first, here is how to reproduce:

```
In [1]: import moto

In [2]: moto.mock_s3_deprecated().start()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-8b514e5bb847> in <module>()
----> 1 moto.mock_s3_deprecated().start()

.../lib/python3.7/site-packages/moto/core/models.py in start(self, reset)
     54                 backend.reset()
     55 
---> 56         self.enable_patching()
     57 
     58     def stop(self):

.../lib/python3.7/site-packages/moto/core/models.py in enable_patching(self)
    113                         method=method,
    114                         uri=re.compile(key),
--> 115                         body=convert_httpretty_response(value),
    116                     )
    117 

.../lib/python3.7/site-packages/moto/packages/httpretty/core.py in register_uri(cls, method, uri, body, adding_headers, forcing_headers, status, responses, match_querystring, **headers)
    962 
    963         matcher = URIMatcher(uri, entries_for_this_uri,
--> 964                              match_querystring)
    965         if matcher in cls._entries:
    966             matcher.entries.extend(cls._entries[matcher])

~/workspace/cnag/cms/cms/__init__.py in patched_init(self, uri, entries, match_querystring)
     13             POTENTIAL_HTTP_PORTS.add(int(result.port or 80))
     14     else:
---> 15         self.info = core.URIInfo.from_uri(uri, entries)
     16 
     17     self.entries = entries

.../lib/python3.7/site-packages/moto/packages/httpretty/core.py in from_uri(cls, uri, entry)
    751     @classmethod
    752     def from_uri(cls, uri, entry):
--> 753         result = urlsplit(uri)
    754         if result.scheme == 'https':
    755             POTENTIAL_HTTPS_PORTS.add(int(result.port or 443))

/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py in urlsplit(url, scheme, allow_fragments)
    398     Note that we don't break the components up in smaller bits
    399     (e.g. netloc is a single string) and we don't expand % escapes."""
--> 400     url, scheme, _coerce_result = _coerce_args(url, scheme)
    401     allow_fragments = bool(allow_fragments)
    402     key = url, scheme, allow_fragments, type(url), type(scheme)

/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py in _coerce_args(*args)
    121     if str_input:
    122         return args + (_noop,)
--> 123     return _decode_args(args) + (_encode_result,)
    124 
    125 # Result objects are more helpful than simple tuples

/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py in _decode_args(args, encoding, errors)
    105 def _decode_args(args, encoding=_implicit_encoding,
    106                        errors=_implicit_errors):
--> 107     return tuple(x.decode(encoding, errors) if x else '' for x in args)
    108 
    109 def _coerce_args(*args):

/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py in <genexpr>(.0)
    105 def _decode_args(args, encoding=_implicit_encoding,
    106                        errors=_implicit_errors):
--> 107     return tuple(x.decode(encoding, errors) if x else '' for x in args)
    108 
    109 def _coerce_args(*args):

AttributeError: 're.Pattern' object has no attribute 'decode'

```

If I'm not mistaken this happens because the vendored version of `httpretty` (0.8.10) in `moto.packages.httpretty` doesn't work with Python 3.7. It checks for `SRE_Pattern` only but since Python 3.7 introduced `re.Pattern` it doesn't recognize it is a regular expression pattern and treats it as a string. Thus the `decode()` call.

https://github.com/spulec/moto/blob/bb6da93891482ddcb87c23e6abddc0e451f72d30/moto/packages/httpretty/core.py#L775

The latest version of `httpretty` (0.9.5) doesn't have that issue and should work fine.

So I think updating the vendored version of `httpretty` should make the problem go away.

Moto is a great convenience when testing AWS interactions. Thanks for that!


Discussion:
I started work on upgrading this but don't have much to show for it so far. Two things worth noting:

1. `lxml` and `sure` need to be upgraded to run the tests against 3.7.
2. I attempted to upgrade the vendored HTTPretty and manually reapply the [changes](https://github.com/spulec/moto/commits/master/moto/packages/httpretty), but this breaks the tests, causing threads to hang and a bunch of `MotoNotImplementedError`s in the test output. It looks like the [changes in the 0.8.11 release](https://github.com/gabrielfalcao/HTTPretty/compare/0.8.10...0.8.11) are where these failures first occur (possibly threading-related).

I'll post updates here if I make any progress.
Since `httpretty` is only for the deprecated decorators, I'd be fine just backporting the one or two line change that needs to be made so this works.

Resolving patch:
diff --git a/moto/packages/httpretty/core.py b/moto/packages/httpretty/core.py
--- a/moto/packages/httpretty/core.py
+++ b/moto/packages/httpretty/core.py
@@ -85,6 +85,7 @@
 old_ssl_wrap_socket = None
 old_sslwrap_simple = None
 old_sslsocket = None
+old_sslcontext_wrap_socket = None
 
 if PY3:  # pragma: no cover
     basestring = (bytes, str)
@@ -100,6 +101,10 @@
     if not PY3:
         old_sslwrap_simple = ssl.sslwrap_simple
     old_sslsocket = ssl.SSLSocket
+    try:
+        old_sslcontext_wrap_socket = ssl.SSLContext.wrap_socket
+    except AttributeError:
+        pass
 except ImportError:  # pragma: no cover
     ssl = None
 
@@ -281,7 +286,7 @@ def getpeercert(self, *a, **kw):
             return {
                 'notAfter': shift.strftime('%b %d %H:%M:%S GMT'),
                 'subjectAltName': (
-                    ('DNS', '*%s' % self._host),
+                    ('DNS', '*.%s' % self._host),
                     ('DNS', self._host),
                     ('DNS', '*'),
                 ),
@@ -772,7 +777,7 @@ class URIMatcher(object):
 
     def __init__(self, uri, entries, match_querystring=False):
         self._match_querystring = match_querystring
-        if type(uri).__name__ == 'SRE_Pattern':
+        if type(uri).__name__ in ('SRE_Pattern', 'Pattern'):
             self.regex = uri
             result = urlsplit(uri.pattern)
             if result.scheme == 'https':
@@ -1012,6 +1017,10 @@ def disable(cls):
         if ssl:
             ssl.wrap_socket = old_ssl_wrap_socket
             ssl.SSLSocket = old_sslsocket
+            try:
+                ssl.SSLContext.wrap_socket = old_sslcontext_wrap_socket
+            except AttributeError:
+                pass
             ssl.__dict__['wrap_socket'] = old_ssl_wrap_socket
             ssl.__dict__['SSLSocket'] = old_sslsocket
 
@@ -1058,6 +1067,14 @@ def enable(cls):
             ssl.wrap_socket = fake_wrap_socket
             ssl.SSLSocket = FakeSSLSocket
 
+            try:
+                def fake_sslcontext_wrap_socket(cls, *args, **kwargs):
+                    return fake_wrap_socket(*args, **kwargs)
+
+                ssl.SSLContext.wrap_socket = fake_sslcontext_wrap_socket
+            except AttributeError:
+                pass
+
             ssl.__dict__['wrap_socket'] = fake_wrap_socket
             ssl.__dict__['SSLSocket'] = FakeSSLSocket
 


Test patch: