Repository: pypa/pipenv

Issue:
Do not destroy lock early
It was introduced in https://github.com/pypa/pipenv/commit/719656481064dfdc7fb4d9e77a6f23ac7709a087, but I don't see what it has to do with other parts.

Remove `destroy_lockfile`, let's see whether this revision fails testing.

Fixes #1888 
Delete Pipfile.lock as late as possible
When updating Pipfile.lock, the old version is deleted before anything happens.  If the update process fails (maybe because one Ctrl+C's it because of a mistake, or it's [taking too long][3], or [it crashes][4], etc.), then one is left with no lockfile.

Could the behaviour be changed so that the [lockfile deletion][1] is performed **immediately** before the [serialisation of the new content][2]?  (FWIW, I'd be happy to attempt a PR for this.)

I guess two rationales for the current behaviour:

1. Your `Pipfile.lock` should be in Git.  Yes, but it still sucks if you're doing a sequence of new package installs/etc. and haven't committed your work.
2. The update of the `Pipfile.lock` should be consistent with the current state of the virtualenv.  I'd buy that, but the current behaviour (no lockfile) doesn't strike me as better :wink:.

[1]: https://github.com/pypa/pipenv/blob/master/pipenv/core.py#L1026
[2]: https://github.com/pypa/pipenv/blob/master/pipenv/core.py#L1173
[3]: https://github.com/pypa/pipenv/issues/1785
[4]: https://github.com/pypa/pipenv/issues/1837

------------

##### Steps to replicate

1. Start with an existing Pipenv environment (i.e. you have a working `Pipfile`/`Pipfile.lock`).
2. Run `pipenv install <any_dep_that_takes_a_long_time>`.
3. Hit Ctrl+C before it's finished.



Discussion:

Pull requests are welcomed :)
@uranusjr - Good :)  I'll do so once maintainers agree the proposed change wouldn't break some intentional behaviour.
@tsiq-oliverc I would say that the lockfile should _never_ be updated with the current state of the environment. We never arbitrarily update lockfiles _ever_. If you activate your environment and pip install some stuff, you didn’t use pipenv dependency management and it could badly break resolution. We can’t be responsible for managing this

However, if you want to write a PR that renames the old lockfile at the start of the process and swaps it back on exceptions or deletes it on success that would make sense to me
@techalchemy - Sorry, to clarify, what I mean is this:

I could imagine that the current behaviour is intentional, as it would avoid situations where there's an outdated lockfile.  Imagine `pipenv` is installing/updating some package (`A`), but it crashes (or is killed) **after** updating `A` but **before** updating the lockfile.  Then the lockfile would say one thing (old or no version of `A`), but the actual contents of the environment would be different.

Nevertheless, if you guys are happy for me to go with the "atomic" replacement approach then I'll get to that this week, hopefully!


+1 from me on the atomic file replacing part. I’ve always disliked how Pipenv deletes the lockfile and then fails to lock, personally. Nothing wrong technically, but is a bit expectected from a UX perspective.
Yeah that is what I had in mind. 
This reminds me Armin has a [battle-tested `atomic_open` snippet](https://github.com/lektor/lektor/blob/05a541120e6f6add0d5d9d30d0585166f838594c/lektor/utils.py#L451) we might be able to steal.
This would be really helpful for us. I am currently in a location where the network is very slow, and I keep seeing developers hit a download timeout, not noticing Pipfile.lock disappeared, running` pipenv install` again and then noticing all kinds of unexpected upgrades happened.

Resolving patch:
diff --git a/pipenv/core.py b/pipenv/core.py
--- a/pipenv/core.py
+++ b/pipenv/core.py
@@ -25,6 +25,7 @@
 from .cmdparse import ScriptEmptyError
 from .project import Project, SourceNotFound
 from .utils import (
+    atomic_open_for_write,
     convert_deps_from_pip,
     convert_deps_to_pip,
     is_required_version,
@@ -1005,8 +1006,6 @@ def do_lock(
             )
             sys.exit(1)
         cached_lockfile = project.lockfile_content
-    if write:
-        project.destroy_lockfile()
     if write:
         # Alert the user of progress.
         click.echo(
@@ -1167,7 +1166,7 @@ def do_lock(
             ]
     if write:
         # Write out the lockfile.
-        with open(project.lockfile_location, 'w') as f:
+        with atomic_open_for_write(project.lockfile_location) as f:
             simplejson.dump(
                 lockfile, f, indent=4, separators=(',', ': '), sort_keys=True
             )
diff --git a/pipenv/project.py b/pipenv/project.py
--- a/pipenv/project.py
+++ b/pipenv/project.py
@@ -673,14 +673,6 @@ def find_source(sources, name=None, url=None):
             return found_source
         raise SourceNotFound(name or url)
 
-    def destroy_lockfile(self):
-        """Deletes the lockfile."""
-        try:
-            return os.remove(self.lockfile_location)
-
-        except OSError:
-            pass
-
     def get_package_name_in_pipfile(self, package_name, dev=False):
         """Get the equivalent package name in pipfile"""
         key = 'dev-packages' if dev else 'packages'
diff --git a/pipenv/utils.py b/pipenv/utils.py
--- a/pipenv/utils.py
+++ b/pipenv/utils.py
@@ -1323,3 +1323,44 @@ def split_argument(req, short=None, long_=None):
             index, more_req = remaining_line[0], ' '.join(remaining_line[1:])
             req = '{0} {1}'.format(req, more_req)
     return req, index
+
+
+@contextmanager
+def atomic_open_for_write(target, binary=False):
+    """Atomically open `target` for writing.
+
+    This is based on Lektor's `atomic_open()` utility, but simplified a lot
+    to handle only writing, and skip many multi-process/thread edge cases
+    handled by Werkzeug.
+
+    How this works:
+
+    * Create a temp file (in the same directory of the actual target), and
+      yield for surrounding code to write to it.
+    * If some thing goes wrong, try to remove the temp file. The actual target
+      is not touched whatsoever.
+    * If everything goes well, close the temp file, and replace the actual
+      target with this new file.
+    """
+    fd, tmp = tempfile.mkstemp(
+        dir=os.path.dirname(target),
+        prefix='.__atomic-write',
+    )
+    os.chmod(tmp, 0o644)
+    f = os.fdopen(fd, 'wb' if binary else 'w')
+    try:
+        yield f
+    except BaseException:
+        f.close()
+        try:
+            os.remove(tmp)
+        except OSError:
+            pass
+        raise
+    else:
+        f.close()
+        try:
+            os.remove(target)   # This is needed on Windows.
+        except OSError:
+            pass
+        os.rename(tmp, target)  # No os.replace() on Python 2.


Test patch: