Repository: pypa/pip

Issue:
Preserving timestamps on copy
We're using timestamps to determine whether the `.c` files generated with cython are up to date with the source `.pyx` files. However pip seems to modify timestamps (I suppose when copying the files from cache dir to build dir). Would it be possible to preserve the timestamps on copy?



Discussion:
@xavfernandez  I would like to take this up. Could you point me where this happens?

I'm not really sure what the use case of @akhmerov is.
But if it is for `pip install ./some_directory` then the copy happens here: https://github.com/pypa/pip/blob/30f4af0/pip/download.py#L671

We observed this behavior when using pypi install. I believe the timestamps get modified when files are copied from cache dir to build dir. The package suffering from the issue was `kwant==1.1.0`.

@akhmerov Correct me if I'm wrong.

For example, the `lapack.pyx` is used to produce the `lapack.c` file and since in the `.tar.gz` file the timestamp of `lapack.pyx` is smaller than the one of `lapack.c`, it means the `lapack.c` is up-to-date and does not need to be regenerated.

On PyPI, kwant is only available as a `.tar.gz` which would be extracted directly in the build directory.
And our `untar_file` function (https://github.com/pypa/pip/blob/98ee8761/pip/utils/__init__.py#L505-L577) does not update the modification time.

To reproduce the problem, execute something like

```
virtualenv foo
. foo/bin/activate
pip install --no-clean kwant==1.1.0
```

On my (and also @akhmerov's) machine, the build fails with a message like this:

```
************************************ Error ************************************
Some Cython source files are newer than files that should have
 been derived from them, but Cython is not installed.
```

The reason is that all the files have the same timestamp:

```
(foo)cwg@neron:/tmp/foo% ls -lt /tmp/pip-build-kYopVI/kwant/kwant
total 1376
-rw-rw-r-- 1 cwg cwg 1200152 Dec 18 18:06 _system.c
-rw-rw-r-- 1 cwg cwg   12192 Dec 18 18:06 _system.pyx
drwxrwxr-x 3 cwg cwg     360 Dec 18 18:06 linalg
-rw-rw-r-- 1 cwg cwg    3519 Dec 18 18:06 digest.py
drwxrwxr-x 3 cwg cwg     140 Dec 18 18:06 physics
-rw-rw-r-- 1 cwg cwg   67709 Dec 18 18:06 plotter.py
-rw-rw-r-- 1 cwg cwg    9836 Dec 18 18:06 system.py
-rw-rw-r-- 1 cwg cwg    3076 Dec 18 18:06 _common.py
-rw-rw-r-- 1 cwg cwg      60 Dec 18 18:06 _kwant_version.py
-rw-rw-r-- 1 cwg cwg   52592 Dec 18 18:06 builder.py
drwxrwxr-x 4 cwg cwg     380 Dec 18 18:06 graph
drwxrwxr-x 2 cwg cwg     180 Dec 18 18:06 tests
-rw-rw-r-- 1 cwg cwg    2019 Dec 18 18:06 __init__.py
-rw-rw-r-- 1 cwg cwg   27122 Dec 18 18:06 lattice.py
-rw-rw-r-- 1 cwg cwg    9454 Dec 18 18:06 rmt.py
drwxrwxr-x 3 cwg cwg     160 Dec 18 18:06 solvers
```

If you download and untar https://pypi.python.org/packages/source/k/kwant/kwant-1.1.0.tar.gz manually, the timestamps are of course preserved.

Note that we now work around this problem by disabling cythonization by default.  But I think it would be still nice if the problem was fixed in pip.


Resolving patch:
diff --git a/pip/utils/__init__.py b/pip/utils/__init__.py
--- a/pip/utils/__init__.py
+++ b/pip/utils/__init__.py
@@ -562,12 +562,10 @@ def untar_file(filename, location):
                     )
                     continue
                 ensure_dir(os.path.dirname(path))
-                destfp = open(path, 'wb')
-                try:
+                with open(path, 'wb') as destfp:
                     shutil.copyfileobj(fp, destfp)
-                finally:
-                    destfp.close()
-                fp.close()
+                # Update the timestamp (useful for cython compiled files)
+                tar.utime(member, path)
                 # member have any execute permissions for user/group/world?
                 if member.mode & 0o111:
                     # make dest file have execute for user/group/world


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -327,6 +327,10 @@ def test_unpack_tgz(self, data):
         test_file = data.packages.join("test_tar.tgz")
         untar_file(test_file, self.tempdir)
         self.confirm_files()
+        # Check the timestamp of an extracted file
+        file_txt_path = os.path.join(self.tempdir, 'file.txt')
+        mtime = time.gmtime(os.stat(file_txt_path).st_mtime)
+        assert mtime[0:6] == (2013, 8, 16, 5, 13, 37), mtime
 
     def test_unpack_zip(self, data):
         """