Repository: conda/conda

Issue:
Python missing when installing a noarch package from explicit environment file
Hello!

We're running into a really strange issue with `conda 4.4.6` (and previously with `conda 4.4.0`). 

We usually provide installation instructions for our software using `conda install --file` which works really well for our users. But for one particular package, `html5lib` on `conda-forge`, It appears that the `.pyc` compile step of installing a noarch package is happening before python is actually installed in the environment causing the entire transaction to fail. 

Here's a bit more info from my debugging:
This happens in Python 3.5 and 3.6
This does not occur with `conda 4.3.31`. 
This happens on both linux and osx as far as we can tell.

Our usual environment files are really large, but I was able to reproduce the problem with the following:

```
§ conda create -n html5 -c conda-forge html5lib
```

<details>
<summary>output</summary>

```
Solving environment: done

## Package Plan ##

  environment location: /home/evan/.conda/envs/html5

  added / updated specs: 
    - html5lib


The following NEW packages will be INSTALLED:

    ca-certificates: 2017.11.5-0      conda-forge
    certifi:         2017.11.5-py36_0 conda-forge
    html5lib:        1.0.1-py_0       conda-forge
    ncurses:         5.9-10           conda-forge
    openssl:         1.0.2n-0         conda-forge
    pip:             9.0.1-py36_1     conda-forge
    python:          3.6.4-0          conda-forge
    readline:        7.0-0            conda-forge
    setuptools:      38.2.4-py36_0    conda-forge
    six:             1.11.0-py36_1    conda-forge
    sqlite:          3.20.1-2         conda-forge
    tk:              8.6.7-0          conda-forge
    webencodings:    0.5-py36_0       conda-forge
    wheel:           0.30.0-py_1      conda-forge
    xz:              5.2.3-0          conda-forge
    zlib:            1.2.11-0         conda-forge

Proceed ([y]/n)? 

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate html5
#
# To deactivate an active environment, use:
# > source deactivate
#
```
</details>

&nbsp;

```
§ source activate html5
§ conda list --explicit --export > env.txt
§ source deactivate
```


<details>
<summary> env.txt </summary>

```
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
@EXPLICIT
https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2017.11.5-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/certifi-2017.11.5-py36_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/html5lib-1.0.1-py_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/ncurses-5.9-10.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/openssl-1.0.2n-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/pip-9.0.1-py36_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/python-3.6.4-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/readline-7.0-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/setuptools-38.2.4-py36_0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/six-1.11.0-py36_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.20.1-2.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.7-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/webencodings-0.5-py36_0.tar.bz2
https://conda.anaconda.org/conda-forge/noarch/wheel-0.30.0-py_1.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.3-0.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-0.tar.bz2
```
</details>

&nbsp;

```
§ conda create -n html5-copy --file env.txt -vv
```

<details>
<summary>verbose output</summary>

```
DEBUG conda.gateways.logging:set_verbosity(148): verbosity set to 2
DEBUG conda.base.context:root_writable(358): [Errno 13] Permission denied: '/opt/Miniconda3/conda-meta/history'
DEBUG conda.base.context:root_writable(358): [Errno 13] Permission denied: '/opt/Miniconda3/conda-meta/history'
DEBUG conda.core.package_cache:_check_writable(226): package cache directory '/home/evan/.conda/pkgs' writable: True
DEBUG conda.core.package_cache:__init__(528): instantiating ProgressiveFetchExtract with
  conda-forge/linux-64::ca-certificates==2017.11.5=0
  conda-forge/linux-64::certifi==2017.11.5=py36_0
  conda-forge/noarch::html5lib==1.0.1=py_0
  conda-forge/linux-64::ncurses==5.9=10
  conda-forge/linux-64::openssl==1.0.2n=0
  conda-forge/linux-64::pip==9.0.1=py36_1
  conda-forge/linux-64::python==3.6.4=0
  conda-forge/linux-64::readline==7.0=0
  conda-forge/linux-64::setuptools==38.2.4=py36_0
  conda-forge/linux-64::six==1.11.0=py36_1
  conda-forge/linux-64::sqlite==3.20.1=2
  conda-forge/linux-64::tk==8.6.7=0
  conda-forge/linux-64::webencodings==0.5=py36_0
  conda-forge/noarch::wheel==0.30.0=py_1
  conda-forge/linux-64::xz==5.2.3=0
  conda-forge/linux-64::zlib==1.2.11=0

DEBUG conda.gateways.disk.test:file_path_is_writable(29): [Errno 13] Permission denied: '/opt/Miniconda3/pkgs/urls.txt'
DEBUG conda.core.package_cache:_check_writable(226): package cache directory '/opt/Miniconda3/pkgs' writable: False
DEBUG conda.core.package_cache:_check_writable(226): package cache directory '/home/evan/.conda/pkgs' writable: True
DEBUG conda.core.link:__init__(157): instantiating UnlinkLinkTransaction with
  target_prefix: /home/evan/.conda/envs/html5-copy
  unlink_precs:
    
  link_precs:
    conda-forge::ca-certificates-2017.11.5-0
    conda-forge::certifi-2017.11.5-py36_0
    conda-forge::html5lib-1.0.1-py_0
    conda-forge::ncurses-5.9-10
    conda-forge::openssl-1.0.2n-0
    conda-forge::pip-9.0.1-py36_1
    conda-forge::python-3.6.4-0
    conda-forge::readline-7.0-0
    conda-forge::setuptools-38.2.4-py36_0
    conda-forge::six-1.11.0-py36_1
    conda-forge::sqlite-3.20.1-2
    conda-forge::tk-8.6.7-0
    conda-forge::webencodings-0.5-py36_0
    conda-forge::wheel-0.30.0-py_1
    conda-forge::xz-5.2.3-0
    conda-forge::zlib-1.2.11-0

Preparing transaction: ...working... DEBUG conda.core.link:get_python_version(598): found in current transaction python version 3.6.4
done
Verifying transaction: ...working... done
DEBUG conda.common.signals:signal_handler(43): registering handler for SIGABRT
DEBUG conda.common.signals:signal_handler(43): registering handler for SIGINT
DEBUG conda.common.signals:signal_handler(43): registering handler for SIGTERM
DEBUG conda.common.signals:signal_handler(43): registering handler for SIGQUIT
Executing transaction: ...working... INFO conda.core.link:_execute_actions(527): ===> LINKING PACKAGE: conda-forge::ca-certificates-2017.11.5-0 <===
  prefix=/home/evan/.conda/envs/html5-copy
  source=/home/evan/.conda/pkgs/ca-certificates-2017.11.5-0

INFO conda.core.link:_execute_actions(527): ===> LINKING PACKAGE: conda-forge::certifi-2017.11.5-py36_0 <===
  prefix=/home/evan/.conda/envs/html5-copy
  source=/home/evan/.conda/pkgs/certifi-2017.11.5-py36_0

INFO conda.core.link:_execute_actions(527): ===> LINKING PACKAGE: conda-forge::html5lib-1.0.1-py_0 <===
  prefix=/home/evan/.conda/envs/html5-copy
  source=/home/evan/.conda/pkgs/html5lib-1.0.1-py_0

DEBUG conda.gateways.subprocess:subprocess_call(46): executing>> "/home/evan/.conda/envs/html5-copy/bin/python3.6" -Wi -m py_compile "/home/evan/.conda/envs/html5-copy/lib/python3.6/site-packages/html5lib/__init__.py"
DEBUG conda.core.link:_execute_actions(541): Error in action #48 for pkg_idx #2 CompilePycAction(target_prefix='/home/evan/.conda/envs/html5-copy', target_short_path='lib/python3.6/site-packages/html5lib/__pycache__/__init__.cpython-36.pyc', source_prefix='/home/evan/.conda/envs/html5-copy', source_short_path='lib/python3.6/site-packages/html5lib/__init__.py', prefix_path_data=PathDataV1(_path='lib/python3.6/site-packages/html5lib/__pycache__/__init__.cpython-36.pyc', path_type='pyc_file'), _execute_successful=False, _verified=True)
Traceback (most recent call last):
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/link.py", line 535, in _execute_actions
    action.execute()
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/path_actions.py", line 523, in execute
    compile_pyc(python_full_path, self.source_full_path, self.target_full_path)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/gateways/disk/create.py", line 309, in compile_pyc
    result = subprocess_call(command, raise_on_error=False)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/gateways/subprocess.py", line 47, in subprocess_call
    p = Popen(command_arg, cwd=path, stdin=PIPE, stdout=PIPE, stderr=PIPE, env=env)
  File "/opt/Miniconda3/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/opt/Miniconda3/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/evan/.conda/envs/html5-copy/bin/python3.6'
INFO conda.core.link:_reverse_actions(569): ===> REVERSING PACKAGE LINK: conda-forge::html5lib-1.0.1-py_0 <===
  prefix=/home/evan/.conda/envs/html5-copy

DEBUG conda.core.link:_reverse_actions(571): reversing pkg_idx #2 from axn_idx #48
failed
ERROR conda.core.link:_execute(481): An error occurred while installing package 'conda-forge::html5lib-1.0.1-py_0'.
FileNotFoundError(2, "No such file or directory: '/home/evan/.conda/envs/html5-copy/bin/python3.6'")
Attempting to roll back.

Rolling back transaction: ...working... INFO conda.core.link:_reverse_actions(569): ===> REVERSING PACKAGE LINK: conda-forge::certifi-2017.11.5-py36_0 <===
  prefix=/home/evan/.conda/envs/html5-copy

DEBUG conda.core.link:_reverse_actions(571): reversing pkg_idx #1 from axn_idx #-1
INFO conda.core.link:_reverse_actions(569): ===> REVERSING PACKAGE LINK: conda-forge::ca-certificates-2017.11.5-0 <===
  prefix=/home/evan/.conda/envs/html5-copy

DEBUG conda.core.link:_reverse_actions(571): reversing pkg_idx #0 from axn_idx #-1
done
DEBUG conda.common.signals:signal_handler(56): de-registering handler for Signals.SIGABRT
DEBUG conda.common.signals:signal_handler(56): de-registering handler for Signals.SIGINT
DEBUG conda.common.signals:signal_handler(56): de-registering handler for Signals.SIGTERM
DEBUG conda.common.signals:signal_handler(56): de-registering handler for Signals.SIGQUIT
Traceback (most recent call last):
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/link.py", line 535, in _execute_actions
    action.execute()
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/path_actions.py", line 523, in execute
    compile_pyc(python_full_path, self.source_full_path, self.target_full_path)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/gateways/disk/create.py", line 309, in compile_pyc
    result = subprocess_call(command, raise_on_error=False)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/gateways/subprocess.py", line 47, in subprocess_call
    p = Popen(command_arg, cwd=path, stdin=PIPE, stdout=PIPE, stderr=PIPE, env=env)
  File "/opt/Miniconda3/lib/python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "/opt/Miniconda3/lib/python3.6/subprocess.py", line 1326, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/home/evan/.conda/envs/html5-copy/bin/python3.6'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/link.py", line 472, in _execute
    cls._execute_actions(pkg_idx, axngroup)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/link.py", line 553, in _execute_actions
    reverse_excs,
conda.CondaMultiError: [Errno 2] No such file or directory: '/home/evan/.conda/envs/html5-copy/bin/python3.6'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/exceptions.py", line 724, in __call__
    return func(*args, **kwargs)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
    exit_code = do_call(args, p)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 76, in do_call
    exit_code = getattr(module, func_name)(args, parser)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/cli/main_create.py", line 11, in execute
    install(args, parser, 'create')
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/cli/install.py", line 184, in install
    explicit(specs, prefix, verbose=not context.quiet, index_args=index_args)
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/misc.py", line 90, in explicit
    txn.execute()
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/link.py", line 225, in execute
    self._execute(tuple(concat(interleave(itervalues(self.prefix_action_groups)))))
  File "/opt/Miniconda3/lib/python3.6/site-packages/conda/core/link.py", line 500, in _execute
    rollback_excs,
conda.CondaMultiError: [Errno 2] No such file or directory: '/home/evan/.conda/envs/html5-copy/bin/python3.6'
```
</details>


&nbsp;

It kind of seems like `html5lib` is just broken on conda-forge, but the way `conda` is failing seems so bizarre I wanted to post this here in case there's something fundamentally wrong happening.

And here's the usual conda info:
<details><summary>conda info</summary>

```
§ conda info

     active environment : None
       user config file : /home/evan/.condarc
 populated config files : 
          conda version : 4.4.6
    conda-build version : 3.0.28
         python version : 3.6.1.final.0
       base environment : /opt/Miniconda3  (read only)
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
          package cache : /opt/Miniconda3/pkgs
                          /home/evan/.conda/pkgs
       envs directories : /home/evan/.conda/envs
                          /opt/Miniconda3/envs
               platform : linux-64
             user-agent : conda/4.4.6 requests/2.14.2 CPython/3.6.1 Linux/4.14.5-1-ARCH arch/ glibc/2.26
                UID:GID : 1000:100
             netrc file : None
           offline mode : False
```
</details>

&nbsp;

Let me know if I can provide further details! Thanks!


Discussion:
Thanks for the report.  I think `@EXPLICIT` files have always been installed in-order.  I don't think there's anything different here between 4.3 and 4.4.

I'm guessing it's more likely that conda 4.4 is giving the wrong order for `conda list --explicit --export`.  Could you do me a favor and verify the order you want is correct in 4.3 and not correct in 4.4?  That is, that html5lib is after python in the 4.3 case and before python in the 4.4 case?
Ok, got back to a computer now.  Actually both 4.3 and 4.4 list them in alphabetical order.  Definitely wrong.  I can fix that.  Especially in 4.4.

Resolving patch:
diff --git a/conda/cli/main_list.py b/conda/cli/main_list.py
--- a/conda/cli/main_list.py
+++ b/conda/cli/main_list.py
@@ -11,7 +11,7 @@
 from ..base.constants import DEFAULTS_CHANNEL_NAME, UNKNOWN_CHANNEL
 from ..base.context import context
 from ..common.compat import text_type
-from ..core.linked_data import is_linked, linked, linked_data
+from ..core.linked_data import PrefixData, is_linked, linked
 from ..egg_info import get_egg_info
 from ..gateways.disk.test import is_conda_environment
 from ..history import History
@@ -101,12 +101,12 @@ def print_explicit(prefix, add_md5=False):
         raise EnvironmentLocationNotFound(prefix)
     print_export_header(context.subdir)
     print("@EXPLICIT")
-    for meta in sorted(linked_data(prefix).values(), key=lambda x: x['name']):
-        url = meta.get('url')
+    for prefix_record in PrefixData(prefix).iter_records_sorted():
+        url = prefix_record.get('url')
         if not url or url.startswith(UNKNOWN_CHANNEL):
-            print('# no URL for: %s' % meta['fn'])
+            print('# no URL for: %s' % prefix_record['fn'])
             continue
-        md5 = meta.get('md5')
+        md5 = prefix_record.get('md5')
         print(url + ('#%s' % md5 if add_md5 and md5 else ''))
 
 
diff --git a/conda/core/linked_data.py b/conda/core/linked_data.py
--- a/conda/core/linked_data.py
+++ b/conda/core/linked_data.py
@@ -89,6 +89,13 @@ def get(self, package_name, default=NULL):
     def iter_records(self):
         return itervalues(self._prefix_records)
 
+    def iter_records_sorted(self):
+        from ..resolve import Resolve
+        index = {Dist(rec): rec for rec in self.iter_records()}
+        r = Resolve(index)
+        sorted_dists = r.dependency_sort({d.name: d for d in index})
+        return (index[d] for d in sorted_dists)
+
     def all_subdir_urls(self):
         subdir_urls = set()
         for prefix_record in itervalues(self._prefix_records):


Test patch: