Repository: pypa/pipenv

Issue:
Incompatibility with pypy3


### Issue description

Recent versions of pipenv no longer work with pypy3.


### Expected result

Expected pipenv to work.

### Actual result

```
❯ PYENV_VERSION=pypy3.5-6.0.0 pyenv exec pipenv --verbose
Traceback (most recent call last):
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/bin/pipenv", line 7, in <module>
    from pipenv import cli
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/__init__.py", line 47, in <module>
    from .cli import cli
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/cli/__init__.py", line 3, in <module>
    from .command import cli
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/cli/command.py", line 19, in <module>
    from .options import (
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/cli/options.py", line 11, in <module>
    from .. import environments
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/environments.py", line 7, in <module>
    from ._compat import fix_utf8
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/_compat.py", line 98, in <module>
    OUT_ENCODING, ERR_ENCODING = force_encoding()
  File "/Users/ob/.pyenv/versions/pypy3.5-6.0.0/site-packages/pipenv/_compat.py", line 68, in force_encoding
    from ctypes import pythonapi, py_object, c_char_p
ImportError: cannot import name 'pythonapi'
```


### Steps to replicate

Simply install pipenv with PyPy3, then try to run any pipenv command.

-------------------------------------------------------------------------------

`pipenv --support` won't work either.

PyPy doesn't support `ctypes.pythonapi`, but the root cause of the issue is that it shouldn't even try to import this module in the first place. I've investigated a bit and it looks with PyPy3, `sys.stdout.encoding` returns `"utf8"` instead of `"utf-8"`. I'm unsure whether this is a PyPy bug or not, but since pipenv used to work fine with PyPy3 prior to (I think) #3096, can we relax the condition on [this line](https://github.com/pypa/pipenv/blob/b03983e61ef284157c725dfedef02fef5829341f/pipenv/_compat.py#L64) to allow for `"utf8"` as well as `"utf-8"`?




Discussion:
Pipenv is not guaranteed to work on PyPy, nor is PyPy included in our CI setup.

Well, PyPy appears in setup.py classifiers? That should be removed, sorry for the trouble.
Hey @frostming, thanks for the reply. It's a bit disappointing to hear that PyPy isn't officially supported, as it means that packages that _do_ support PyPy need to introduce additional complexity to use pipenv only on supported Python versions or not use pipenv at all.

Also, regarding the exact issue I described, I suspect this _is_ a bug in pipenv:
1. In `__init__.py`, `sys.stdout` and `sys.stderr` are replaced by `io.TextIOWrapper`s with `encoding` set to `'utf8'` (no dash): https://github.com/pypa/pipenv/blob/d9ebad335dc4cfff9cb223f7cc3141066044b65b/pipenv/__init__.py#L31-L36
2. In `_compat.py`, the encoding is forced to UTF-8 by checking if `sys.stdout/err.encoding.lower()` is equal to `'utf-8'` (with a dash): https://github.com/pypa/pipenv/blob/d9ebad335dc4cfff9cb223f7cc3141066044b65b/pipenv/_compat.py#L331

Good catch, would like to send a PR for it?
Certainly :) Will send one in a moment!

Resolving patch:
diff --git a/pipenv/_compat.py b/pipenv/_compat.py
--- a/pipenv/_compat.py
+++ b/pipenv/_compat.py
@@ -26,8 +26,8 @@
 
 __all__ = [
     "NamedTemporaryFile", "Path", "ResourceWarning", "TemporaryDirectory",
-    "get_terminal_size", "getpreferredencoding", "DEFAULT_ENCODING", "force_encoding",
-    "UNICODE_TO_ASCII_TRANSLATION_MAP", "decode_output", "fix_utf8"
+    "get_terminal_size", "getpreferredencoding", "DEFAULT_ENCODING", "canonical_encoding_name",
+    "force_encoding", "UNICODE_TO_ASCII_TRANSLATION_MAP", "decode_output", "fix_utf8"
 ]
 
 
@@ -46,6 +46,16 @@ def getpreferredencoding():
 DEFAULT_ENCODING = getpreferredencoding()
 
 
+def canonical_encoding_name(name):
+    import codecs
+    try:
+        codec = codecs.lookup(name)
+    except LookupError:
+        return name
+    else:
+        return codec.name
+
+
 # From https://github.com/CarlFK/veyepar/blob/5c5de47/dj/scripts/fixunicode.py
 # MIT LIcensed, thanks Carl!
 def force_encoding():
@@ -57,20 +67,23 @@ def force_encoding():
     else:
         if not (stdout_isatty() and stderr_isatty()):
             return DEFAULT_ENCODING, DEFAULT_ENCODING
-    stdout_encoding = sys.stdout.encoding
-    stderr_encoding = sys.stderr.encoding
+    stdout_encoding = canonical_encoding_name(sys.stdout.encoding)
+    stderr_encoding = canonical_encoding_name(sys.stderr.encoding)
     if sys.platform == "win32" and sys.version_info >= (3, 1):
         return DEFAULT_ENCODING, DEFAULT_ENCODING
-    if stdout_encoding.lower() != "utf-8" or stderr_encoding.lower() != "utf-8":
+    if stdout_encoding != "utf-8" or stderr_encoding != "utf-8":
 
-        from ctypes import pythonapi, py_object, c_char_p
+        try:
+            from ctypes import pythonapi, py_object, c_char_p
+        except ImportError:
+            return DEFAULT_ENCODING, DEFAULT_ENCODING
         try:
             PyFile_SetEncoding = pythonapi.PyFile_SetEncoding
         except AttributeError:
             return DEFAULT_ENCODING, DEFAULT_ENCODING
         else:
             PyFile_SetEncoding.argtypes = (py_object, c_char_p)
-            if stdout_encoding.lower() != "utf-8":
+            if stdout_encoding != "utf-8":
                 try:
                     was_set = PyFile_SetEncoding(sys.stdout, "utf-8")
                 except OSError:
@@ -80,7 +93,7 @@ def force_encoding():
                 else:
                     stdout_encoding = "utf-8"
 
-            if stderr_encoding.lower() != "utf-8":
+            if stderr_encoding != "utf-8":
                 try:
                     was_set = PyFile_SetEncoding(sys.stderr, "utf-8")
                 except OSError:


Test patch: