Repository: conda/conda

Issue:
conda env export omits pip packages (again)
<!--
Hi!  Read this; it's important.

This is an issue tracker for conda -- the package manager.  File feature requests
for conda here, as well as bug reports about something conda has messed up.

If your issue is a bug report for:
  * a specific conda package from Anaconda ('defaults' channel):
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * a specific conda package from conda-forge:
      ==> file at the corresponding feedstock under https://github.com/conda-forge
  * repo.anaconda.com access and service:
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * anaconda.org access and service:
      ==> file at https://anaconda.org/contact/report
  * commands under 'conda build':
      ==> file at https://github.com/conda/conda-build
  * commands under 'conda env':
      ==> please file it here!
  * all other conda commands that start with 'conda':
      ==> please file it here!

If you continue on,

   **please include all requested information below.**

If a maintainer determines the information is required to understand 
your issue, and if it is not provided, your issue may be
closed automatically.

-->

## Current Behavior
<!-- What actually happens?
     If you want to include console output, please use "Steps to Reproduce" below. -->
conda env export omits pip install'ed packages. I can see this has come up before. This time it's on conda 4.6.0. I tried changing the pip version to 10.*, 18.1, 18.0, but this did not fix the issue.

Even when exporting with the -vv flag there are no informative outputs printed to the log. The failure to include pip appears to be completely silent.

### Steps to Reproduce
<!-- If the current behavior is a bug, please provide specific, minimal steps to independently reproduce.
     Include the exact conda commands that reproduce the issue and their output between the ticks below. -->
I test using conda-forge and and the base channels. First conda-forge:
```
(base) 09:36:54 login27:~/python/neuroconda$ conda create -n test-forge -c conda-forge python=3.6
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /imaging/local/software/centos7/anaconda3/envs/test-forge

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/linux-64::ca-certificates-2018.11.29-ha4d7672_0
  certifi            conda-forge/linux-64::certifi-2018.11.29-py36_1000
  libffi             conda-forge/linux-64::libffi-3.2.1-hf484d3e_1005
  libgcc-ng          conda-forge/linux-64::libgcc-ng-7.3.0-hdf63c60_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-7.3.0-hdf63c60_0
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  openssl            conda-forge/linux-64::openssl-1.0.2p-h14c3975_1002
  pip                conda-forge/linux-64::pip-18.1-py36_1000
  python             conda-forge/linux-64::python-3.6.7-hd21baee_1001
  readline           conda-forge/linux-64::readline-7.0-hf8c457e_1001
  setuptools         conda-forge/linux-64::setuptools-40.6.3-py36_0
  sqlite             conda-forge/linux-64::sqlite-3.26.0-h67949de_1000
  tk                 conda-forge/linux-64::tk-8.6.9-h84994c4_1000
  wheel              conda-forge/linux-64::wheel-0.32.3-py36_0
  xz                 conda-forge/linux-64::xz-5.2.4-h14c3975_1001
  zlib               conda-forge/linux-64::zlib-1.2.11-h14c3975_1004


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test-forge
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) 09:37:38 login27:~/python/neuroconda$ conda activate test-forge
(test-forge) 09:38:20 login27:~/python/neuroconda$ pip install --no-deps pybids
Collecting pybids
Installing collected packages: pybids
Successfully installed pybids-0.7.0
(test-forge) 09:38:50 login27:~/python/neuroconda$ conda env export -f test.yml
(test-forge) 09:39:22 login27:~/python/neuroconda$ cat test.yml
name: test-forge
channels:
  - conda-forge
  - defaults
  - r
dependencies:
  - ca-certificates=2018.11.29=ha4d7672_0
  - certifi=2018.11.29=py36_1000
  - libffi=3.2.1=hf484d3e_1005
  - libgcc-ng=7.3.0=hdf63c60_0
  - libstdcxx-ng=7.3.0=hdf63c60_0
  - ncurses=6.1=hf484d3e_1002
  - openssl=1.0.2p=h14c3975_1002
  - pip=18.1=py36_1000
  - python=3.6.7=hd21baee_1001
  - readline=7.0=hf8c457e_1001
  - setuptools=40.6.3=py36_0
  - sqlite=3.26.0=h67949de_1000
  - tk=8.6.9=h84994c4_1000
  - wheel=0.32.3=py36_0
  - xz=5.2.4=h14c3975_1001
  - zlib=1.2.11=h14c3975_1004
prefix: /imaging/local/software/centos7/anaconda3/envs/test-forge
(test-forge) 09:46:10 login27:~/python/neuroconda$ conda list pybids
# packages in environment at /imaging/local/software/centos7/anaconda3/envs/test-forge:
#
# Name                    Version                   Build  Channel
pybids                    0.7.0                    pypi_0    pypi
```
Then base:
```
(base) 09:49:44 login27:~/python/neuroconda$ conda create -n test-base -c base python=3.6
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /imaging/local/software/centos7/anaconda3/envs/test-base

  added / updated specs:
    - python=3.6


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2018.12.5  |                0         123 KB
    certifi-2018.11.29         |           py36_0         146 KB
    libedit-3.1.20181209       |       hc058e9b_0         188 KB
    ncurses-6.1                |       he6710b0_1         958 KB
    openssl-1.1.1a             |       h7b6447c_0         5.0 MB
    pip-18.1                   |           py36_0         1.8 MB
    python-3.6.8               |       h0371630_0        34.4 MB
    readline-7.0               |       h7b6447c_5         392 KB
    setuptools-40.6.3          |           py36_0         625 KB
    sqlite-3.26.0              |       h7b6447c_0         1.9 MB
    tk-8.6.8                   |       hbc83047_0         3.1 MB
    wheel-0.32.3               |           py36_0          35 KB
    zlib-1.2.11                |       h7b6447c_3         120 KB
    ------------------------------------------------------------
                                           Total:        48.8 MB

The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/linux-64::ca-certificates-2018.12.5-0
  certifi            pkgs/main/linux-64::certifi-2018.11.29-py36_0
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1a-h7b6447c_0
  pip                pkgs/main/linux-64::pip-18.1-py36_0
  python             pkgs/main/linux-64::python-3.6.8-h0371630_0
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-64::setuptools-40.6.3-py36_0
  sqlite             pkgs/main/linux-64::sqlite-3.26.0-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.32.3-py36_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Proceed ([y]/n)? y


Downloading and Extracting Packages
openssl-1.1.1a       | 5.0 MB    | ######################################################################################################################################################################## | 100%
zlib-1.2.11          | 120 KB    | ######################################################################################################################################################################## | 100%
certifi-2018.11.29   | 146 KB    | ######################################################################################################################################################################## | 100%
setuptools-40.6.3    | 625 KB    | ######################################################################################################################################################################## | 100%
ca-certificates-2018 | 123 KB    | ######################################################################################################################################################################## | 100%
libedit-3.1.20181209 | 188 KB    | ######################################################################################################################################################################## | 100%
sqlite-3.26.0        | 1.9 MB    | ######################################################################################################################################################################## | 100%
ncurses-6.1          | 958 KB    | ######################################################################################################################################################################## | 100%
tk-8.6.8             | 3.1 MB    | ######################################################################################################################################################################## | 100%
python-3.6.8         | 34.4 MB   | ######################################################################################################################################################################## | 100%
readline-7.0         | 392 KB    | ######################################################################################################################################################################## | 100%
wheel-0.32.3         | 35 KB     | ######################################################################################################################################################################## | 100%
pip-18.1             | 1.8 MB    | ######################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test-base
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) 09:50:45 login27:~/python/neuroconda$ conda activate test-base
(test-base) 09:51:13 login27:~/python/neuroconda$ pip install --no-deps pybids
Collecting pybids
Installing collected packages: pybids
Successfully installed pybids-0.7.0
(test-base) 09:51:32 login27:~/python/neuroconda$ conda env export -f test.yml
(test-base) 09:51:57 login27:~/python/neuroconda$ cat test.yml
name: test-base
channels:
  - defaults
  - conda-forge
  - r
dependencies:
  - ca-certificates=2018.12.5=0
  - certifi=2018.11.29=py36_0
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - ncurses=6.1=he6710b0_1
  - openssl=1.1.1a=h7b6447c_0
  - pip=18.1=py36_0
  - python=3.6.8=h0371630_0
  - readline=7.0=h7b6447c_5
  - setuptools=40.6.3=py36_0
  - sqlite=3.26.0=h7b6447c_0
  - tk=8.6.8=hbc83047_0
  - wheel=0.32.3=py36_0
  - xz=5.2.4=h14c3975_4
  - zlib=1.2.11=h7b6447c_3
prefix: /imaging/local/software/centos7/anaconda3/envs/test-base
(test-base) 09:52:00 login27:~/python/neuroconda$ conda list pybids
# packages in environment at /imaging/local/software/centos7/anaconda3/envs/test-base:
#
# Name                    Version                   Build  Channel
pybids                    0.7.0                    pypi_0    pypi
```

## Expected Behavior
<!-- What do you think should happen? -->
pip installed packages should appear in '-pip:' sub category. If this functionality is broken for whatever reason conda should complain loudly about this when exporting.

## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

```
(test-base) 09:54:19 login27:~/python/neuroconda$ conda info

     active environment : test-base
    active env location : /imaging/local/software/centos7/anaconda3/envs/test-base
            shell level : 3
       user config file : /home/jc01/.condarc
 populated config files : /home/jc01/.condarc
          conda version : 4.6.0
    conda-build version : 3.10.5
         python version : 3.6.5.final.0
       base environment : /imaging/local/software/centos7/anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/r/linux-64
                          https://conda.anaconda.org/r/noarch
          package cache : /imaging/local/software/centos7/anaconda3/pkgs
                          /home/jc01/.conda/pkgs
       envs directories : /imaging/local/software/centos7/anaconda3/envs
                          /home/jc01/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.0 requests/2.18.4 CPython/3.6.5 Linux/3.10.0-862.el7.x86_64 centos/7.5.1804 glibc/2.17
                UID:GID : 6303:50
             netrc file : None
           offline mode : False
```
</p></details>


<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

```
==> /home/jc01/.condarc <==
ssl_verify: False
channels:
  - conda-forge
  - defaults
  - r
```
</p></details>


<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda list --show-channel-urls' -->

```
# packages in environment at /imaging/local/software/centos7/anaconda3/envs/test-base:
#
# Name                    Version                   Build  Channel
ca-certificates           2018.12.5                     0    defaults
certifi                   2018.11.29               py36_0    defaults
libedit                   3.1.20181209         hc058e9b_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 8.2.0                hdf63c60_1    defaults
libstdcxx-ng              8.2.0                hdf63c60_1    defaults
ncurses                   6.1                  he6710b0_1    defaults
openssl                   1.1.1a               h7b6447c_0    defaults
pip                       18.1                     py36_0    defaults
pybids                    0.7.0                    pypi_0    pypi
python                    3.6.8                h0371630_0    defaults
readline                  7.0                  h7b6447c_5    defaults
setuptools                40.6.3                   py36_0    defaults
sqlite                    3.26.0               h7b6447c_0    defaults
tk                        8.6.8                hbc83047_0    defaults
wheel                     0.32.3                   py36_0    defaults
xz                        5.2.4                h14c3975_4    defaults
zlib                      1.2.11               h7b6447c_3    defaults
```
</p></details>



Discussion:
... However, downgrading to conda 4.5.9 restores normal pip export function. But broke my installation as detailed in #7788. I repaired it using the instructions there (removing `# conda version` lines from conda-meta/history). I then was able to `conda update conda`... to 4.5.12, which is now the latest available version on base and conda-forge. Anyway, exporting on 4.5.12 also works fine.

Which raises the question of how I ended up on 4.6.0 in the first place. I looked in the history file, and it seems `conda update -n base conda` put me on 4.6.0 on 16 January. So it looks like there was a conda-forge 4.6.0 release that got pulled. And that's where my problems started.
... And now that the 4.6.1 release is official, I tried the update again (from conda-forge). Same problem. However, it turns out that you can get pip export to work again if you turn on the new pip_interop_enabled flag:

```
(cbu_nipy_1_03dev) 02:34:09 login27:~/python/neuroconda$ conda config --set pip_interop_enabled False
(cbu_nipy_1_03dev) 02:34:15 login27:~/python/neuroconda$ conda env export -f test.yml
(cbu_nipy_1_03dev) 02:34:23 login27:~/python/neuroconda$ cat test.yml | grep pip
  - apipkg=1.5=py_0
  - pip=10.0.1=py36_0
(cbu_nipy_1_03dev) 02:34:39 login27:~/python/neuroconda$ conda config --set pip_interop_enabled True
(cbu_nipy_1_03dev) 02:34:44 login27:~/python/neuroconda$ conda env export -f test.yml
(cbu_nipy_1_03dev) 02:34:56 login27:~/python/neuroconda$ cat test.yml | grep pip
  - apipkg=1.5=py_0
  - pip=10.0.1=py36_0
  - pip:
```
Hope this is useful for someone.
thanks! was trolling thru the source code trying to figure out how to re-enable this

Resolving patch:
diff --git a/conda/history.py b/conda/history.py
--- a/conda/history.py
+++ b/conda/history.py
@@ -260,7 +260,7 @@ def get_user_requests(self):
                 'allow_conda_downgrades' to False at their own risk.
                 """)
 
-                # we need to rethink this.  It's fine as a warning to try to get users
+                # TODO: we need to rethink this.  It's fine as a warning to try to get users
                 #    to avoid breaking their system.  However, right now it is preventing
                 #    normal conda operation after downgrading conda.
                 # raise CondaUpgradeError(message)
diff --git a/conda_env/cli/main_update.py b/conda_env/cli/main_update.py
--- a/conda_env/cli/main_update.py
+++ b/conda_env/cli/main_update.py
@@ -107,7 +107,7 @@ def execute(args, parser):
     for installer_type in env.dependencies:
         try:
             installers[installer_type] = get_installer(installer_type)
-        except InvalidInstaller as e:
+        except InvalidInstaller:
             sys.stderr.write(textwrap.dedent("""
                 Unable to install package for {0}.
 
diff --git a/conda_env/env.py b/conda_env/env.py
--- a/conda_env/env.py
+++ b/conda_env/env.py
@@ -78,7 +78,9 @@ def from_environment(name, prefix, no_builds=False, ignore_channels=False):
     Returns:     Environment object
     """
     # requested_specs_map = History(prefix).get_requested_specs_map()
-    precs = tuple(PrefixGraph(PrefixData(prefix).iter_records()).graph)
+    pd = PrefixData(prefix, pip_interop_enabled=True)
+
+    precs = tuple(PrefixGraph(pd.iter_records()).graph)
     grouped_precs = groupby(lambda x: x.package_type, precs)
     conda_precs = sorted(concatv(
         grouped_precs.get(None, ()),


Test patch:
diff --git a/tests/conda_env/support/pip_argh.yml b/tests/conda_env/support/pip_argh.yml
new file mode 100644
--- /dev/null
+++ b/tests/conda_env/support/pip_argh.yml
@@ -0,0 +1,6 @@
+name: pip_argh
+
+dependencies:
+  - pip
+  - pip:
+      - argh==0.26.2
diff --git a/tests/conda_env/test_create.py b/tests/conda_env/test_create.py
--- a/tests/conda_env/test_create.py
+++ b/tests/conda_env/test_create.py
@@ -1,13 +1,8 @@
 # -*- coding: utf-8 -*-
 from __future__ import absolute_import, division, print_function
 
-from argparse import ArgumentParser
-from contextlib import contextmanager
 from logging import Handler, getLogger
 from os.path import exists, join
-from shlex import split
-from shutil import rmtree
-from tempfile import mkdtemp
 from unittest import TestCase
 from uuid import uuid4
 
@@ -19,18 +14,12 @@
 from conda.install import on_win
 from conda.models.enums import PackageType
 from conda.models.match_spec import MatchSpec
-from conda_env.cli.main import do_call as do_call_conda_env
-from conda_env.cli.main_create import configure_parser as create_configure_parser
-from conda_env.cli.main_update import configure_parser as update_configure_parser
 from . import support_file
+from .utils import make_temp_envs_dir, Commands, run_command
 
 PYTHON_BINARY = 'python.exe' if on_win else 'bin/python'
 
 
-def escape_for_winpath(p):
-    return p.replace('\\', '\\\\')
-
-
 def disable_dotlog():
     class NullHandler(Handler):
         def emit(self, record):
@@ -47,38 +36,6 @@ def reenable_dotlog(handlers):
     dotlogger.handlers = handlers
 
 
-class Commands:
-    CREATE = "create"
-    UPDATE = "update"
-
-
-parser_config = {
-    Commands.CREATE: create_configure_parser,
-    Commands.UPDATE: update_configure_parser,
-}
-
-
-def run_command(command, env_name, *arguments):
-    p = ArgumentParser()
-    sub_parsers = p.add_subparsers(metavar='command', dest='cmd')
-    parser_config[command](sub_parsers)
-
-    arguments = list(map(escape_for_winpath, arguments))
-    command_line = "{0} -n {1} -f {2}".format(command, env_name, " ".join(arguments))
-
-    args = p.parse_args(split(command_line))
-    do_call_conda_env(args, p)
-
-
-@contextmanager
-def make_temp_envs_dir():
-    envs_dir = mkdtemp()
-    try:
-        yield envs_dir
-    finally:
-        rmtree(envs_dir, ignore_errors=True)
-
-
 def package_is_installed(prefix, spec, pip=None):
     spec = MatchSpec(spec)
     prefix_recs = tuple(PrefixData(prefix).query(spec))
diff --git a/tests/conda_env/test_env.py b/tests/conda_env/test_env.py
--- a/tests/conda_env/test_env.py
+++ b/tests/conda_env/test_env.py
@@ -1,9 +1,21 @@
 from collections import OrderedDict
 import os
+from os.path import join
 import random
 import unittest
+from uuid import uuid4
 
+from conda.core.prefix_data import PrefixData
+from conda.base.context import reset_context
+from conda.common.io import env_vars
 from conda.common.serialize import yaml_load
+from conda.install import on_win
+import ruamel_yaml
+
+from . import support_file
+from .utils import make_temp_envs_dir, Commands, run_command
+
+PYTHON_BINARY = 'python.exe' if on_win else 'bin/python'
 
 
 try:
@@ -14,8 +26,6 @@
 from conda_env import env
 from conda_env import exceptions
 
-from . import support_file
-
 
 class FakeStream(object):
     def __init__(self):
@@ -340,3 +350,31 @@ def _test_saves_yaml_representation_of_file(self):
 
         self.assert_(len(actual) > 0, msg='sanity check')
         self.assertEqual(e.to_yaml(), actual)
+
+
+class SaveExistingEnvTestCase(unittest.TestCase):
+    def test_create_advanced_pip(self):
+        with make_temp_envs_dir() as envs_dir:
+            with env_vars({
+                'CONDA_ENVS_DIRS': envs_dir,
+                'CONDA_PIP_INTEROP_ENABLED': 'true',
+            }, reset_context):
+                env_name = str(uuid4())[:8]
+                prefix = join(envs_dir, env_name)
+                python_path = join(prefix, PYTHON_BINARY)
+
+                run_command(Commands.CREATE, env_name,
+                            support_file('pip_argh.yml'))
+                out_file = join(envs_dir, 'test_env.yaml')
+
+            # make sure that the export reconsiders the presence of pip interop being enabled
+            PrefixData._cache_.clear()
+
+            with env_vars({
+                'CONDA_ENVS_DIRS': envs_dir,
+            }, reset_context):
+                # note: out of scope of pip interop var.  Should be enabling conda pip interop itself.
+                run_command(Commands.EXPORT, env_name, out_file)
+                with open(out_file) as f:
+                    d = ruamel_yaml.load(f)
+                assert {'pip': ['argh==0.26.2']} in d['dependencies']
diff --git a/tests/conda_env/utils.py b/tests/conda_env/utils.py
new file mode 100644
--- /dev/null
+++ b/tests/conda_env/utils.py
@@ -0,0 +1,49 @@
+from argparse import ArgumentParser
+from contextlib import contextmanager
+from tempfile import mkdtemp
+from shlex import split
+
+from conda.gateways.disk.delete import rm_rf
+
+from conda_env.cli.main import do_call as do_call_conda_env
+from conda_env.cli.main_create import configure_parser as create_configure_parser
+from conda_env.cli.main_update import configure_parser as update_configure_parser
+from conda_env.cli.main_export import configure_parser as export_configure_parser
+
+
+class Commands:
+    CREATE = "create"
+    UPDATE = "update"
+    EXPORT = "export"
+
+
+parser_config = {
+    Commands.CREATE: create_configure_parser,
+    Commands.UPDATE: update_configure_parser,
+    Commands.EXPORT: export_configure_parser,
+}
+
+
+def escape_for_winpath(p):
+    return p.replace('\\', '\\\\')
+
+
+@contextmanager
+def make_temp_envs_dir():
+    envs_dir = mkdtemp()
+    try:
+        yield envs_dir
+    finally:
+        rm_rf(envs_dir)
+
+
+def run_command(command, env_name, *arguments):
+    p = ArgumentParser()
+    sub_parsers = p.add_subparsers(metavar='command', dest='cmd')
+    parser_config[command](sub_parsers)
+
+    arguments = list(map(escape_for_winpath, arguments))
+    command_line = "{0} -n {1} -f {2}".format(command, env_name, " ".join(arguments))
+
+    args = p.parse_args(split(command_line))
+    do_call_conda_env(args, p)