Repository: pypa/pipenv

Issue:
Can't lock package with `git` and `extras`
##### Issue description

A package spec with `git` and `extras` fails to lock. This happens regardless of the `editable` flag.

##### Expected result

The package should be installed with all extras.

##### Actual result

```
[I] ~/test $ pipenv --version
pipenv, version 2018.7.1
[I] ~/test $ ll
total 8
-rw-r--r--  1 jacobhayes    96B Jul  3 13:12 Pipfile
[I] ~/test $ cat Pipfile
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}
[I] ~/test $ pipenv --rm ; pipenv install --verbose
Removing virtualenv (/Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt)...
Creating a virtualenv for this project...
Pipfile: /Users/jacobhayes/test/Pipfile
Using /usr/local/opt/python/bin/python3.6 (3.6.5) to create virtualenv...
⠋Already using interpreter /usr/local/opt/python/bin/python3.6
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python3.6
Also creating executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python
Installing setuptools, pip, wheel...done.
Setting project for test-qFl43Lkt to /Users/jacobhayes/test

Virtualenv location: /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1943, in do_install
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1312, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1140, in do_lock
    project.write_lockfile(lockfile)
  File "/usr/local/lib/python3.6/site-packages/pipenv/project.py", line 648, in write_lockfile
    content, indent=4, separators=(u',', u': '), sort_keys=True,
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ArrayElement' is not JSON serializable
```

##### Steps to replicate

```toml
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}
```

`pipenv install`

-------------

<details><summary>$ pipenv --support</summary>

Pipenv version: `'2018.7.1'`

Pipenv location: `'/usr/local/lib/python3.6/site-packages/pipenv'`

Python location: `'/usr/local/opt/python/bin/python3.6'`

Other Python installations in `PATH`:

  - `2.7`: `/usr/local/bin/python2.7`
  - `2.7`: `/usr/local/bin/python2.7`
  - `2.7`: `/usr/bin/python2.7`
  - `3.6`: `/usr/local/bin/python3.6m`
  - `3.6`: `/usr/local/bin/python3.6`

  - `2.7.15`: `/usr/local/bin/python`
  - `2.7.10`: `/usr/bin/python`
  - `2.7.15`: `/usr/local/bin/python2`
  - `3.6.5`: `/usr/local/bin/python3`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT '
                     '2018; root:xnu-4570.61.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}
```

System environment variables:

  - `Apple_PubSub_Socket_Render`
  - `COLORFGBG`
  - `COLORTERM`
  - `COPYFILE_DISABLE`
  - `EDITOR`
  - `GOPATH`
  - `HOME`
  - `ITERM_PROFILE`
  - `ITERM_SESSION_ID`
  - `LANG`
  - `LOGNAME`
  - `PAGER`
  - `PATH`
  - `PWD`
  - `SECURITYSESSIONID`
  - `SHELL`
  - `SHLVL`
  - `SSH_AUTH_SOCK`
  - `TERM`
  - `TERM_PROGRAM`
  - `TERM_PROGRAM_VERSION`
  - `TERM_SESSION_ID`
  - `TMPDIR`
  - `TMUX`
  - `TMUX_PANE`
  - `TMUX_PLUGIN_MANAGER_PATH`
  - `USER`
  - `XPC_FLAGS`
  - `XPC_SERVICE_NAME`
  - `__CF_USER_TEXT_ENCODING`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`

Pipenv–specific environment variables:


Debug–specific environment variables:

  - `PATH`: `/Users/jacobhayes/bin:/Users/jacobhayes/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/jacobhayes/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin`
  - `SHELL`: `/usr/local/bin/fish`
  - `EDITOR`: `vim`
  - `LANG`: `en_US.UTF-8`
  - `PWD`: `/Users/jacobhayes/test`


---------------------------

Contents of `Pipfile` ('/Users/jacobhayes/test/Pipfile'):

```toml
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

```

</details>



Discussion:
This works with `v2018.05.18`, but fails in `v2018.6.25` and `v2018.7.1`
Interesting! I can definitely replicate this.  
`git bisect` tells me that https://github.com/pypa/pipenv/commit/18530dfb46bbe6aef9dc23ce8774ddca6990db65 is the culprit, if that helps.
I did some digging today and an easy fix for this is to make the following change in `Project.write_lockfile()`:

```python
    def write_lockfile(self, content):
        """Write out the lockfile.
        """
        newlines = self._lockfile_newlines

        def encode_objects(obj):
            if isinstance(obj, ArrayElement): # prettytoml.elements.array.ArrayElement
                return obj.primitive_value
            return obj

        s = simplejson.dumps(   # Send Unicode in to guarentee Unicode out.
            content, indent=4, separators=(u',', u': '), sort_keys=True,
            default=encode_objects
        )
        with atomic_open_for_write(self.lockfile_location, newline=newlines) as f:
            f.write(s)
            if not s.endswith(u'\n'):
                f.write(u'\n')  # Write newline at end of document. GH #319.
```
thanks for digging into this, I know very little about the toml parser -- do you have a sense of what's going on here?
When the [lockfile is read from the file](https://github.com/pypa/pipenv/blob/master/pipenv/project.py#L472), it just copies the structure over from the parsed TOML. I'm not sure why, but the TOML structure returns a specialized `ArrayElement` instead of a Python `list` (which is why `simplejson.dumps()` throws an error). Rather than iterate through the TOML coming out of the parser, it looked easier to have `simplejson.dumps()` convert the `ArrayElement`s to `list`s.
Development since v2018.5.18 completely killed our deployment. This bug and #2317 are easily worked around (patch and external variable substitution) ~~but #2552 is probably not~~. It would be great to have pipenv tested so that these issues don't find their way to the open :)
Otherwise great work, its much appreciated
This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1. I took the weekend off of working on this project, but we’d definitely merge a failing test / fix if we haven’t already (not totally caught up here)
> This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1.

This is definitely happening on 3.6.* for me.
@bryanforbes would you be willing to submit a PR with your changes or mind if I do?
@JacobHayes you're more than welcome to do the PR. I'm swamped this week.

Resolving patch:
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -13,7 +13,6 @@
 import pipfile.api
 import six
 import toml
-import json as simplejson
 
 from ._compat import Path
 
@@ -64,10 +63,36 @@ def _normalized(p):
 DEFAULT_NEWLINES = u"\n"
 
 
+class _LockFileEncoder(json.JSONEncoder):
+    """A specilized JSON encoder to convert loaded TOML data into a lock file.
+
+    This adds a few characteristics to the encoder:
+
+    * The JSON is always prettified with indents and spaces.
+    * PrettyTOML's container elements are seamlessly encodable.
+    * The output is always UTF-8-encoded text, never binary, even on Python 2.
+    """
+    def __init__(self):
+        super(_LockFileEncoder, self).__init__(
+            indent=4, separators=(",", ": "), sort_keys=True,
+        )
+
+    def default(self, obj):
+        from prettytoml.elements.common import ContainerElement, TokenElement
+        if isinstance(obj, (ContainerElement, TokenElement)):
+            return obj.primitive_value
+        return super(_LockFileEncoder, self).default(obj)
+
+    def encode(self, obj):
+        content = super(_LockFileEncoder, self).encode(obj)
+        if not isinstance(content, six.text_type):
+            content = content.decode("utf-8")
+        return content
+
+
 def preferred_newlines(f):
     if isinstance(f.newlines, six.text_type):
         return f.newlines
-
     return DEFAULT_NEWLINES
 
 
@@ -105,6 +130,8 @@ class SourceNotFound(KeyError):
 class Project(object):
     """docstring for Project"""
 
+    _lockfile_encoder = _LockFileEncoder()
+
     def __init__(self, which=None, python_version=None, chdir=True):
         super(Project, self).__init__()
         self._name = None
@@ -629,14 +656,17 @@ def write_toml(self, data, path=None):
     def write_lockfile(self, content):
         """Write out the lockfile.
         """
-        newlines = self._lockfile_newlines
-        s = simplejson.dumps(  # Send Unicode in to guarentee Unicode out.
-            content, indent=4, separators=(u",", u": "), sort_keys=True
-        )
-        with atomic_open_for_write(self.lockfile_location, newline=newlines) as f:
+        s = self._lockfile_encoder.encode(content)
+        open_kwargs = {
+            'newline': self._lockfile_newlines,
+            'encoding': 'utf-8',
+        }
+        with atomic_open_for_write(self.lockfile_location, **open_kwargs) as f:
             f.write(s)
+            # Write newline at end of document. GH-319.
+            # Only need '\n' here; the file object handles the rest.
             if not s.endswith(u"\n"):
-                f.write(u"\n")  # Write newline at end of document. GH #319.
+                f.write(u"\n")
 
     @property
     def pipfile_sources(self):
@@ -751,7 +781,7 @@ def recase_pipfile(self):
             self.write_toml(self.parsed_pipfile)
 
     def load_lockfile(self, expand_env_vars=True):
-        with io.open(self.lockfile_location) as lock:
+        with io.open(self.lockfile_location, encoding='utf-8') as lock:
             j = json.load(lock)
             self._lockfile_newlines = preferred_newlines(lock)
         # lockfile is just a string


Test patch:
diff --git a/tests/integration/test_lock.py b/tests/integration/test_lock.py
--- a/tests/integration/test_lock.py
+++ b/tests/integration/test_lock.py
@@ -1,6 +1,5 @@
 import pytest
 import os
-import six
 
 from pipenv.utils import temp_environ
 
@@ -348,6 +347,27 @@ def test_lock_editable_vcs_without_install(PipenvInstance, pypi):
         assert c.return_code == 0
 
 
+@pytest.mark.extras
+@pytest.mark.lock
+@pytest.mark.vcs
+@pytest.mark.needs_internet
+def test_lock_editable_vcs_with_extras_without_install(PipenvInstance, pypi):
+    with PipenvInstance(pypi=pypi, chdir=True) as p:
+        with open(p.pipfile_path, 'w') as f:
+            f.write("""
+[packages]
+requests = {git = "https://github.com/requests/requests.git", editable = true, extras = ["socks"]}
+            """.strip())
+        c = p.pipenv('lock')
+        assert c.return_code == 0
+        assert 'requests' in p.lockfile['default']
+        assert 'idna' in p.lockfile['default']
+        assert 'chardet' in p.lockfile['default']
+        assert "socks" in p.lockfile["default"]["requests"]["extras"]
+        c = p.pipenv('install')
+        assert c.return_code == 0
+
+
 @pytest.mark.lock
 @pytest.mark.skip(reason="This doesn't work for some reason.")
 def test_lock_respecting_python_version(PipenvInstance, pypi):