Repository: conda/conda

Issue:
Add repodata information to extracted package directory
Immediately following the extraction of a package tarball, add the repodata information for the package to the `info` directory within the extracted package directory.  Put this information in a new file `info/external.json`.  Or maybe `info/post_extract.json`.  Or come up with a better name.

The information included should be all the information for the package under the `packages` key in repodata.  For example

    $ curl -sS https://repo.continuum.io/pkgs/free/linux-64/repodata.json | jq .packages | jq '.["cryptography-1.5.3-py27_0.tar.bz2"]'

    {
      "build": "py27_0",
      "build_number": 0,
      "date": "2016-11-07",
      "depends": [
        "cffi >=1.4.1",
        "enum34",
        "idna >=2.0",
        "ipaddress",
        "openssl 1.0.2*",
        "pyasn1 >=0.1.8",
        "python 2.7*",
        "setuptools",
        "six >=1.4.1"
      ],
      "license": "Apache",
      "md5": "af03c1cde0584396f9422324701f6136",
      "name": "cryptography",
      "size": 829309,
      "version": "1.5.3"
    }

It should also include

* `fn` (which is the filename of the tarball)
* `url` (the exact, original url from where the tarball was downloaded)
* `arch` (which comes from the root-level `info` key in repodata)
* `platform` (which comes from the root-level `info` key in repodata)
* `priority` (the channel priority number at the time the package was downloaded)
* `channel` (which is right now equivalent to `Channel(full_url).base_url`
* `schannel` (equivalent to `Channel(full_url).canonical_name`)

I've probably missed some information, so feel free to add to the list.

The correct way to do this is probably to create a new PathAction in `conda/core/path_actions.py`, and then incorporate that action into the `ProgressiveFetchExtract.make_actions_for_dist` method in `conda/core/package_cache.py`.



Discussion:
Out of curiosity, is this required for making `conda info $package` faster?
I don't think it'll affect `conda info package`.  That requires downloading repodata, and the round trip http time there, even for 304 responses, is noticeable and what slows things down.  At some point we need to make those repodata fetch calls async.

The reason for this ticket is because the ONLY time we know this information 100% for sure is when we download and extract the tarball.  It'll come in handy in a lot of places where we sort of end up guessing at information.  It'll also come in very handy for `--offline` stuff.
The logic in the source code suggests that downloading the tarball and extracting it are distinct actions. Furthermore, it may be possible for a tarball to already be downloaded (to the package cache) without being extracted. In that case, how can I obtain all the required information ("the exact, original url from where the tarball was downloaded") ? It seems the information would need to be stored in the same transaction that does the downloading, no ?
Yes, they are separate actions.  At some point, we actually need to combine the actions at an individual package level (not for the full fetch/extract operation), and then make individual transactions out of them.  We also can then parallelize them.

I think the best approach here for this immediate PR is to change the return value of `make_actions_for_dist` from a two-value tuple to a three-value tuple, where the third value is a new PathAction.  And then, I think the only place where the new action is actually used is in that last return value, where we're actually downloading packages

```
        # if we got here, we couldn't find a matching package in the caches
        #   we'll have to download one; fetch and extract
        cache_axn = CacheUrlAction(
            url=record.get('url') or dist.to_url(),
            target_pkgs_dir=first_writable_cache.pkgs_dir,
            target_package_basename=dist.to_filename(),
            md5sum=md5,
        )
        extract_axn = ExtractPackageAction(
            source_full_path=cache_axn.target_full_path,
            target_pkgs_dir=first_writable_cache.pkgs_dir,
            target_extracted_dirname=dist.dist_name,
        )
        return cache_axn, extract_axn, new_axn
```

For the rest of the return values, that third position is just `None`.
Right, you are describing almost precisely what I have done already :-)
I'm right now merely wondering where to get the content from to fill the new json file. You indicated earlier that there was information that was available only during the download, which triggered my follow-up question, since at the point where the new action is placed in the pipeline that information may already be lost (unless we change the download action itself).
Ah.  All of the information we have right now available is in `record` from the function siganture.

    def make_actions_for_dist(dist, record):

I think that's an `IndexRecord` object right now.  It should have all of those fields listed above.


Resolving patch:
diff --git a/conda/core/package_cache.py b/conda/core/package_cache.py
--- a/conda/core/package_cache.py
+++ b/conda/core/package_cache.py
@@ -410,6 +410,7 @@ def make_actions_for_dist(dist, record):
                 source_full_path=pc_entry_writable_cache.package_tarball_full_path,
                 target_pkgs_dir=pc_entry_writable_cache.pkgs_dir,
                 target_extracted_dirname=pc_entry_writable_cache.dist.dist_name,
+                record=record,
             )
             return None, extract_axn
 
@@ -431,6 +432,7 @@ def make_actions_for_dist(dist, record):
                 source_full_path=cache_axn.target_full_path,
                 target_pkgs_dir=first_writable_cache.pkgs_dir,
                 target_extracted_dirname=dist.dist_name,
+                record=record,
             )
             return cache_axn, extract_axn
 
@@ -446,6 +448,7 @@ def make_actions_for_dist(dist, record):
             source_full_path=cache_axn.target_full_path,
             target_pkgs_dir=first_writable_cache.pkgs_dir,
             target_extracted_dirname=dist.dist_name,
+            record=record,
         )
         return cache_axn, extract_axn
 
diff --git a/conda/core/path_actions.py b/conda/core/path_actions.py
--- a/conda/core/path_actions.py
+++ b/conda/core/path_actions.py
@@ -26,7 +26,7 @@
 from ..gateways.disk.create import (compile_pyc, copy, create_application_entry_point,
                                     create_fake_executable_softlink, create_hard_link_or_copy,
                                     create_link, create_python_entry_point, extract_tarball,
-                                    make_menu, write_linked_package_record)
+                                    make_menu, write_linked_package_record, write_as_json_to_file)
 from ..gateways.disk.delete import rm_rf, try_rmdir_all_empty
 from ..gateways.disk.link import symlink
 from ..gateways.disk.read import compute_md5sum, isfile, islink, lexists
@@ -1092,11 +1092,13 @@ def __str__(self):
 
 class ExtractPackageAction(PathAction):
 
-    def __init__(self, source_full_path, target_pkgs_dir, target_extracted_dirname):
+    def __init__(self, source_full_path, target_pkgs_dir, target_extracted_dirname,
+                 record):
         self.source_full_path = source_full_path
         self.target_pkgs_dir = target_pkgs_dir
         self.target_extracted_dirname = target_extracted_dirname
         self.hold_path = self.target_full_path + '.c~'
+        self.record = record
 
     def verify(self):
         self._verified = True
@@ -1123,6 +1125,8 @@ def execute(self):
                 else:
                     raise
         extract_tarball(self.source_full_path, self.target_full_path)
+        meta = join(self.target_full_path, 'info', 'repodata_record.json')
+        write_as_json_to_file(meta, self.record)
 
         target_package_cache = PackageCache(self.target_pkgs_dir)
 


Test patch: