Question: There are many posts on this site which reference, typically in passing, the idea of setting pip_interop_enabled=True within some environment. This makes conda and pip3 somehow interact better, I am told. To be precise, people say conda will search PyPI for packages that don't exist in the main channels if this is true. They also say it's "experimental." Here is conda's documentation about this. It notes that much of conda's behavior in recent versions has also improved even with pip_interop_enabled=False, leading to questions about what this setting even does. Here is my question: in real terms, what does all of this mean?  Is the only difference that conda will search PyPI if this is True and not if it's False? Are there other things that it does? For instance, if I need to install some package from pip, will conda know better not to clobber it if this setting is True? What, to be precise, goes wrong if I set this to True? Are there known edge cases that somehow break things if this "experimental" setting is set to True? Why would I ever not want to set this? 

Answer: Not a PyPI Searching Feature First, let's clarify: Conda will not "search PyPI" - that is not what the pip_interop_enabled configuration option adds. Rather, it enables the solver to allow a package already installed with pip to satisfy a dependency requirement of a Conda package. Note that the option is about Pip interoperability (as distinct from PyPI) and it doesn't matter whether the package was sourced from PyPI, GitHub, local, etc.. Example: scipy -> numpy Let's consider a simple example to illustrate the behavior. Start with the following environment that has Python 3.10 and numpy installed from PyPI. pip_interop.yaml name: pip_interop channels: - conda-forge dependencies: - python=3.10 - pip ## PyPI packages - pip: - numpy  which we can create with conda env create -n pip_interop -f pip_interop.yaml  and verify that the numpy is from PyPI: $ conda list -n pip_interop numpy # packages in environment at /Users/user/mambaforge/envs/pip_interop: # # Name Version Build Channel numpy 1.24.2 pypi_0  Let's see what would happen installing scipy and in particular, how it satisfies its numpy dependency. Installing without Pip interoperability In default mode, we see the following behavior $ conda install -n pip_interop scipy Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /Users/user/mambaforge/envs/pip_interop added / updated specs: - scipy The following packages will be downloaded: package | build ---------------------------|----------------- cryptography-39.0.1 | py310hdd0c95c_0 1.1 MB numpy-1.24.2 | py310h788a5b3_0 6.1 MB scipy-1.10.0 | py310h240c617_2 20.2 MB ------------------------------------------------------------ Total: 27.4 MB The following NEW packages will be INSTALLED: appdirs conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0 brotlipy conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 certifi conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 cffi conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 cryptography conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 idna conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 libblas conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas libcblas conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas libcxx conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 libgfortran conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 libgfortran5 conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 liblapack conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas libopenblas conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 numpy conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0 packaging conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 pooch conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0 pycparser conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 pyopenssl conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 pysocks conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 python_abi conda-forge/osx-64::python_abi-3.10-3_cp310 requests conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 scipy conda-forge/osx-64::scipy-1.10.0-py310h240c617_2 urllib3 conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 Proceed ([y]/n)?  Observe that despite numpy already being installed in the environment, Conda is proposing to replace it with a Conda version. That is, only considers the information in conda-meta/ to determine whether a package is installed and won't check the environment's lib/python3.10/site-packages/. Installing with Pip interoperability Now we try it with the pip_interop_enabled turned on: $ CONDA_PIP_INTEROP_ENABLED=1 conda install -n foo scipy Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /Users/user/mambaforge/envs/pip_interop added / updated specs: - scipy The following packages will be downloaded: package | build ---------------------------|----------------- cryptography-39.0.1 | py310hdd0c95c_0 1.1 MB scipy-1.10.0 | py310h240c617_2 20.2 MB ------------------------------------------------------------ Total: 21.3 MB The following NEW packages will be INSTALLED: appdirs conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0 brotlipy conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 certifi conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 cffi conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 cryptography conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 idna conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 libblas conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas libcblas conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas libcxx conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 libgfortran conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 libgfortran5 conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 liblapack conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas libopenblas conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 packaging conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 pooch conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0 pycparser conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 pyopenssl conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 pysocks conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 python_abi conda-forge/osx-64::python_abi-3.10-3_cp310 requests conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 scipy conda-forge/osx-64::scipy-1.10.0-py310h240c617_2 urllib3 conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 Proceed ([y]/n)?  Note that now the numpy is not proposed to be replaced and this is because the existing pip-installed version is consider able to satisfy the dependency. Why is this experimental? There may be multiple reasons why this remains experimental after several years. One important reason is that Conda only tests its package builds against Conda builds of the dependencies. So, it cannot guarantee that the packages are functionally exchangeable. Furthermore, Conda packages often bring in non-Python dependencies. There has been a rise in wheel deployments, which is the PyPI approach to this, but isn't ubiquitous. There are still many "wrapper" packages out there where the PyPI version assumes some binary is on PATH, whereas the installation of the Conda package guarantees the binary is also installed. Another important issue is that the PyPI-Conda name mapping is not well-defined. That is, the name of a package in PyPI may not correspond to its Conda package name. This can directly lead to cryptic issues when the names diverge. Specifically, Conda will not correctly recognize that a pip-installed package satisfies the requirement when the names don't match. Hence, the is some unexpected heterogeneity in how the interoperability applies. Example: torch vs pytorch In the Python ecosystem, the torch module is provided by the PyPI package torch. However, the package torch in PyPI goes by pytorch on Conda channels. Here's how this can lead to inconsistent behavior. Let's begin with torch installed from PyPI: pip_interop.yaml name: pip_interop channels: - conda-forge dependencies: - python=3.10 - pip ## PyPI packages - pip: - torch  Creating with: conda env create -n pip_interop -f pip_interop.yaml  Now if we install torchvision from Conda, even with the pip_interop_enabled on, we get: $ CONDA_PIP_INTEROP_ENABLED=1 conda install -n pip_interop torchvision Collecting package metadata (current_repodata.json): done Solving environment: done ## Package Plan ## environment location: /Users/user/mambaforge/envs/pip_interop added / updated specs: - torchvision The following packages will be downloaded: package | build ---------------------------|----------------- cryptography-39.0.1 | py310hdd0c95c_0 1.1 MB jpeg-9e | hb7f2c08_3 226 KB libprotobuf-3.21.12 | hbc0c0cd_0 1.8 MB mkl-2022.2.1 | h44ed08c_16952 113.1 MB numpy-1.24.2 | py310h788a5b3_0 6.1 MB pillow-9.4.0 | py310h306a057_1 44.1 MB pytorch-1.13.1 |cpu_py310h2bbf33f_1 56.9 MB sleef-3.5.1 | h6db0672_2 1.0 MB torchvision-0.14.1 |cpu_py310hd5ee960_0 5.9 MB ------------------------------------------------------------ Total: 230.1 MB The following NEW packages will be INSTALLED: brotlipy conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005 certifi conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 cffi conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3 charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0 cryptography conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0 freetype conda-forge/osx-64::freetype-2.12.1-h3f81eb7_1 idna conda-forge/noarch::idna-3.4-pyhd8ed1ab_0 jpeg conda-forge/osx-64::jpeg-9e-hb7f2c08_3 lcms2 conda-forge/osx-64::lcms2-2.14-h29502cd_1 lerc conda-forge/osx-64::lerc-4.0.0-hb486fe8_0 libblas conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas libcblas conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas libcxx conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0 libdeflate conda-forge/osx-64::libdeflate-1.17-hac1461d_0 libgfortran conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27 libgfortran5 conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27 liblapack conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas libopenblas conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3 libpng conda-forge/osx-64::libpng-1.6.39-ha978bb4_0 libprotobuf conda-forge/osx-64::libprotobuf-3.21.12-hbc0c0cd_0 libtiff conda-forge/osx-64::libtiff-4.5.0-hee9004a_2 libwebp-base conda-forge/osx-64::libwebp-base-1.2.4-h775f41a_0 libxcb conda-forge/osx-64::libxcb-1.13-h0d85af4_1004 llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0 mkl conda-forge/osx-64::mkl-2022.2.1-h44ed08c_16952 numpy conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0 openjpeg conda-forge/osx-64::openjpeg-2.5.0-h13ac156_2 pillow conda-forge/osx-64::pillow-9.4.0-py310h306a057_1 pthread-stubs conda-forge/osx-64::pthread-stubs-0.4-hc929b4f_1001 pycparser conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0 pyopenssl conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0 pysocks conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6 python_abi conda-forge/osx-64::python_abi-3.10-3_cp310 pytorch conda-forge/osx-64::pytorch-1.13.1-cpu_py310h2bbf33f_1 requests conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0 sleef conda-forge/osx-64::sleef-3.5.1-h6db0672_2 tbb conda-forge/osx-64::tbb-2021.7.0-hb8565cd_1 torchvision conda-forge/osx-64::torchvision-0.14.1-cpu_py310hd5ee960_0 typing_extensions conda-forge/noarch::typing_extensions-4.4.0-pyha770c72_0 urllib3 conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0 xorg-libxau conda-forge/osx-64::xorg-libxau-1.0.9-h35c211d_0 xorg-libxdmcp conda-forge/osx-64::xorg-libxdmcp-1.1.3-h35c211d_0 zstd conda-forge/osx-64::zstd-1.5.2-hbc0c0cd_6 Proceed ([y]/n)?  That is, Conda still tries to install pytorch and this means that it will lead to clobbering of the existing torch package installed from PyPI. This has the potential to having residual files from the clobbered version of the package intermixed with the clobbering version. Basically, this is undefined behavior and the Conda software may not give you any warning about potential problems.