Repository: conda/conda

Issue:
Conda (4.4): 'No such file or directory' in a new environment using non-shell process
<!--
Hi!

This is an issue tracker for conda -- the package manager.  File feature requests
for conda here, as well as bug reports about something conda has messed up.

If your issue is a bug report or feature request for:
  * a specific conda package from Anaconda ('defaults' channel):
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * a specific conda package from conda-forge:
      ==> file at the corresponding feedstock under https://github.com/conda-forge
  * repo.continuum.io access and service:
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * anaconda.org access and service:
      ==> file at https://anaconda.org/contact/report
  * commands under 'conda build':
      ==> file at https://github.com/conda/conda-build
  * commands under 'conda env':
      ==> please file it here!
  * all other conda commands that start with 'conda':
      ==> please file it here!

-->

**I'm submitting a...** (I don't know whether it is a bug or feature)
  - [x] bug report
  - [x] feature request


## Current Behavior
<!-- What actually happens?
     If you want to include console output, please use "Steps to Reproduce" below. -->


### Steps to Reproduce
<!-- If the current behavior is a bug, please provide specific, minimal steps to independently reproduce.
     Include the exact conda commands that reproduce the issue and their output between the ticks below. -->
Create an environment, 'test' using Conda 4.4, open a new tab and directly activate that environment. Now try to invoke conda commands using any non shell program (lua/python/java/....) it will error out saying `conda: command not found` or `No such file or directory` since they can't `exec()` that.

## Expected Behavior
<!-- What do you think should happen? -->
The program invoking conda should not error out, or should have a way to figure out where the conda  program resides, so that it can exec the full `/path/to/conda`. Currently the only thing close to this is `$_CONDA_EXE`, but it is not documented, and also not exported, so not available to child processes and no guarantee whether it will be available in future releases.

The current work around is to first activate the base environment and then activate the non-base environment in the same shell session, which for whatever reason, retains the root environment's bin directory in the PATH.


Discussion:
In conda 4.4, posix shells should be initialized via `. <CONDA_BASE_PREFIX>/etc/profile.d/conda.sh` in a `~/.bashrc` file, etc.
One way to do this right now, is to exec():

```
os.environ['CONDA_PYTHON_EXE'] -m conda <conda arguments>
```
I think we just need to add a `CONDA_EXE` environment variable alongside the `CONDA_PYTHON_EXE` environment variable management in `conda/activate.py`.
I didn't get at first, what was requested, but essentially it's the following difference:
```bash
$ (. etc/profile.d/conda.sh && conda activate base && conda --version)
conda 4.4.9
$ (. etc/profile.d/conda.sh && conda activate env && conda --version)
conda 4.4.9
$ (. etc/profile.d/conda.sh && conda activate base && bash -c 'conda --version')
conda 4.4.9
$ (. etc/profile.d/conda.sh && conda activate env && bash -c 'conda --version')
bash: conda: command not found
```
In the last case the `base` environment (and thus `conda`) isn't on `PATH` (which is a nice `4.4` feature, IMHO!) and we're not trying to run `conda` in a `bash` instance that has the `conda` shell function available; i.e. that `bash -c` represents one arbitrary program that tries to run `conda`.

To me it would make sense to have a `$PREFIX/bin/conda` script in each environment which would `exec` the `base` `conda`. Those scripts could be made into a package, e.g., called something like `conda-launcher`/`conda-scripts`/???. The problem with that would be that, as of know, non-`base` environment have (on their own) no knowledge of the `base` prefix. So you'd either have to use that `CONDA_PYTHON_EXE` workaround with something like:
```bash
#!/bin/bash
exec "$CONDA_PYTHON_EXE" -m conda "${@}"
```
or hardcode the base prefix at install time. As for the former, I don't think there are guarantees that `CONDA_PYTHON_EXE` must be available (and I would rather have `conda` be called instead of that "Python implementation detail"). For the latter, I don't think the base prefix is exposed to post-link scripts or the like.
A third possibility would to always put some directory from `base` on `PATH` that contains some `conda` even if a non-`base` environment is activated. Preferably (to retain that `4.4` feature) this wouldn't be `$CONDA_ROOT/bin` so that only `conda` but nothing else pollutes `PATH`.
In any case (and if I'm not mistaken -- which could very well be) I think a change `conda` would be needed for this feature.
\[EDIT: Just to clarify, I'm with @mingwandroid regarding that third possibility; it's a possibility, but definitely not a good one!\]
> I think we just need to add a `CONDA_EXE` environment variable 

That would be much preferred in lieu of `CONDA_PYTHON_EXE`, yes!
> To me it would make sense to have a $PREFIX/bin/conda script in each environment which would exec the base conda.

I hadn't thought of that option.  I'm going to think more about it.

> A third possibility would to always put some directory from base on PATH that contains some conda even if a non-base environment is activated.

I *had* thought of that option.  And was thinking about implementing it as part of https://github.com/conda/conda/pull/6518.  It would be similar to what the GitBash installer does on Windows.  It's been a while, and I don't remember the exact details, but there's an option in the installer that ends up putting the GitBash executables in a separate `cmd` directory, and then adds that directory to the system-configured `PATH`.

So, when a shell is at `CONDA_SHLVL` `0`, e.g. right after `conda.sh` is sourced, we'd at that point modify PATH with only that special directory.
> A third possibility would to always put some directory from base on PATH that contains some conda even if a non-base environment is activated

Please, no.
I think having a smart conda executable in every conda environment probably makes the most sense.  I think we're going to run into unanticipated problems with that though too.

For example, I had a problem with `os.execv()` on Windows.  See https://github.com/conda/conda/issues/6607
I guess one edge case would be users expecting to have `conda activate` available when hitting this executable, which it wouldn't be, because we're hitting the executable rather than the shell function.  We probably just need to have clear messaging there explaining the problem.
> because we're hitting the executable rather than the shell function

At least in `bash` I think the shell function takes precedence.
The way to implement this is to create a new `PathAction` that creates this stub executable if it doesn't exist.  We need to watch out for a gotcha with the `reverse()` method, because something that triggers `reverse()` on `conda create` should remove the file, while something that triggers `reverse()` on `conda install` probably shouldn't.

Basically, we need a flag in the `execute()` method to indicate whether the file was created, or whether it already existed.  If it was newly created, `reverse()` should just remove it.
> At least in bash I think the shell function takes precedence.

I think it always does for all shells I know of that support functions and aliases.  The problem will be encountered when you're not interacting with conda through a shell though.
If all we're doing is adding a new environment variable to be managed alongside `CONDA_PYTHON_EXE`, I consider that an enhancement, which can go into any patch release.  If we're adding a new `PathAction`, I consider that a feature which will need to wait for a feature release.

> The problem will be encountered when you're not interacting with conda through a shell though.

Since `conda activate`'s job is to modify the shell's environment in currently runs in, a user should *not* \[/EDIT\] expect anything if they run it in a non-shell context 😉 -- but outputting an appropriate error message never hurts.
> If we're adding a new `PathAction`, I consider that a feature which will need to wait for a feature release.

Yes, please. If such a change will be added at some time, I'll have to remember to ping @bgruening about this; in Bioconda, for each package a Docker [Biocontainer](https://biocontainers.pro/) is created -- so this would be a use case where we would *not* \[/EDIT\] want such a launcher.
Well, I think I'm already too much in the `4.4.x` mindset and only now realize/remember that with `4.3.x` we have
```bash
$ . bin/activate 
(root) $ conda --version
conda 4.3.34
(root) $ conda create -qynempty
 

Package plan for installation in environment /home/maba/code/conda/conda-forge/envs/empty:


(root) $ ls -la envs/empty/bin/
total 8
drwx------ 2 maba maba 4096 Feb 14 20:00 .
drwx------ 4 maba maba 4096 Feb 14 20:00 ..
lrwxrwxrwx 1 maba maba   46 Feb 14 20:00 activate -> /home/maba/code/conda/conda-forge/bin/activate
lrwxrwxrwx 1 maba maba   43 Feb 14 20:00 conda -> /home/maba/code/conda/conda-forge/bin/conda
lrwxrwxrwx 1 maba maba   48 Feb 14 20:00 deactivate -> /home/maba/code/conda/conda-forge/bin/deactivate
```
. So bringing back the `conda` symlink wouldn't be such an unexpected thing. Some could even categorize its absence as a regression in 4.4, I guess.
I think I favor a specific directory containing only a conda executable that we add to PATH when we first initialize the shell (e.g. source `conda.sh`), rather than making some type of brittle pointer to where conda is supposed to live in the environment at environment creation time.  However I know Ray prefers the latter.
Another thing to be aware of here... Conda 4.4 lifted the restriction that conda could only be installed in the base environment.  That restriction came about because of the symlinking behavior on activate, and now that we're not doing that, we also don't need the restriction.  Since there's been a strong driver for conda to be both an *application* and a *library* (with the push to have stable internal APIs), lifting the restriction makes sense.

As a future improvement, it might make sense for conda *not* to include the `bin/conda` entry point when conda isn't installed into the base environment, to further enforce it's use as a library in non-base envs.
The only solution I like so far:
>I think we just need to add a CONDA_EXE environment variable

I can't think of any any downsides with it. It's agnostic of:
- Whether it is going to be used by a shell or a non-shell program
- Whether conda is implemented in python or rust or go or whatever
- What platform is going to use it

Plus, it is probably easier to maintain across future releases and we'll also need to guarantee that we don't break it.

Resolving patch:
diff --git a/conda/activate.py b/conda/activate.py
--- a/conda/activate.py
+++ b/conda/activate.py
@@ -244,6 +244,7 @@ def build_activate(self, env_name_or_prefix):
             new_path = self.pathsep_join(self._add_prefix_to_path(prefix))
             export_vars = {
                 'CONDA_PYTHON_EXE': self.path_conversion(sys.executable),
+                'CONDA_EXE': self.path_conversion(context.conda_exe),
                 'PATH': new_path,
                 'CONDA_PREFIX': prefix,
                 'CONDA_SHLVL': old_conda_shlvl + 1,
@@ -313,6 +314,7 @@ def build_deactivate(self):
                 'CONDA_PREFIX',
                 'CONDA_DEFAULT_ENV',
                 'CONDA_PYTHON_EXE',
+                'CONDA_EXE',
                 'CONDA_PROMPT_MODIFIER',
             )
             export_vars = {
diff --git a/conda/base/context.py b/conda/base/context.py
--- a/conda/base/context.py
+++ b/conda/base/context.py
@@ -479,6 +479,12 @@ def root_prefix(self):
     def conda_prefix(self):
         return normpath(sys.prefix)
 
+    @property
+    def conda_exe(self):
+        bin_dir = 'Scripts' if on_win else 'bin'
+        exe = 'conda.exe' if on_win else 'conda'
+        return join(self.conda_prefix, bin_dir, exe)
+
     @memoizedproperty
     def channel_alias(self):
         from ..models.channel import Channel


Test patch:
diff --git a/tests/test_activate.py b/tests/test_activate.py
--- a/tests/test_activate.py
+++ b/tests/test_activate.py
@@ -189,6 +189,7 @@ def test_build_activate_shlvl_0(self):
                     }
                     export_vars = {
                         'CONDA_PYTHON_EXE': activator.path_conversion(sys.executable),
+                        'CONDA_EXE': activator.path_conversion(context.conda_exe),
                         'PATH': new_path,
                         'CONDA_PREFIX': td,
                         'CONDA_SHLVL': 1,
@@ -380,6 +381,7 @@ def test_build_deactivate_shlvl_1(self):
                         'CONDA_PREFIX',
                         'CONDA_DEFAULT_ENV',
                         'CONDA_PYTHON_EXE',
+                        'CONDA_EXE',
                         'CONDA_PROMPT_MODIFIER',
                     )
 
@@ -460,6 +462,7 @@ def test_posix_basic(self):
         assert activate_data == dals("""
         PS1='%(ps1)s'
         \\export CONDA_DEFAULT_ENV='%(native_prefix)s'
+        \\export CONDA_EXE='%(conda_exe)s'
         \\export CONDA_PREFIX='%(native_prefix)s'
         \\export CONDA_PROMPT_MODIFIER='(%(native_prefix)s) '
         \\export CONDA_PYTHON_EXE='%(sys_executable)s'
@@ -472,7 +475,8 @@ def test_posix_basic(self):
             'new_path': activator.pathsep_join(new_path_parts),
             'sys_executable': activator.path_conversion(sys.executable),
             'activate1': activator.path_conversion(join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.sh')),
-            'ps1': '(%s) ' % self.prefix + os.environ.get('PS1', '')
+            'ps1': '(%s) ' % self.prefix + os.environ.get('PS1', ''),
+            'conda_exe': activator.path_conversion(context.conda_exe),
         }
 
         with env_vars({
@@ -513,6 +517,7 @@ def test_posix_basic(self):
             assert deactivate_data == dals("""
             \\. "%(deactivate1)s"
             \\unset CONDA_DEFAULT_ENV
+            \\unset CONDA_EXE
             \\unset CONDA_PREFIX
             \\unset CONDA_PROMPT_MODIFIER
             \\unset CONDA_PYTHON_EXE
@@ -543,6 +548,7 @@ def test_cmd_exe_basic(self):
         new_path_parts = activator._add_prefix_to_path(self.prefix)
         assert activate_data == dals("""
         @SET "CONDA_DEFAULT_ENV=%(native_prefix)s"
+        @SET "CONDA_EXE=%(conda_exe)s"
         @SET "CONDA_PREFIX=%(converted_prefix)s"
         @SET "CONDA_PROMPT_MODIFIER=(%(native_prefix)s) "
         @SET "CONDA_PYTHON_EXE=%(sys_executable)s"
@@ -557,6 +563,7 @@ def test_cmd_exe_basic(self):
             'sys_executable': activator.path_conversion(sys.executable),
             'activate1': activator.path_conversion(join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.bat')),
             'PYTHONIOENCODING': PYTHONIOENCODING,
+            'conda_exe': activator.path_conversion(context.conda_exe),
         }
 
         with env_vars({
@@ -601,6 +608,7 @@ def test_cmd_exe_basic(self):
             assert deactivate_data == dals("""
             @CALL "%(deactivate1)s"
             @SET CONDA_DEFAULT_ENV=
+            @SET CONDA_EXE=
             @SET CONDA_PREFIX=
             @SET CONDA_PROMPT_MODIFIER=
             @SET CONDA_PYTHON_EXE=
@@ -625,6 +633,7 @@ def test_csh_basic(self):
         assert activate_data == dals("""
         set prompt='%(prompt)s';
         setenv CONDA_DEFAULT_ENV "%(native_prefix)s";
+        setenv CONDA_EXE "%(conda_exe)s";
         setenv CONDA_PREFIX "%(native_prefix)s";
         setenv CONDA_PROMPT_MODIFIER "(%(native_prefix)s) ";
         setenv CONDA_PYTHON_EXE "%(sys_executable)s";
@@ -637,7 +646,8 @@ def test_csh_basic(self):
             'new_path': activator.pathsep_join(new_path_parts),
             'sys_executable': activator.path_conversion(sys.executable),
             'activate1': activator.path_conversion(join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.csh')),
-            'prompt': '(%s) ' % self.prefix + os.environ.get('prompt', '')
+            'prompt': '(%s) ' % self.prefix + os.environ.get('prompt', ''),
+            'conda_exe': activator.path_conversion(context.conda_exe),
         }
 
         with env_vars({
@@ -678,6 +688,7 @@ def test_csh_basic(self):
             assert deactivate_data == dals("""
             source "%(deactivate1)s";
             unset CONDA_DEFAULT_ENV;
+            unset CONDA_EXE;
             unset CONDA_PREFIX;
             unset CONDA_PROMPT_MODIFIER;
             unset CONDA_PYTHON_EXE;
@@ -707,6 +718,7 @@ def test_xonsh_basic(self):
         new_path_parts = activator._add_prefix_to_path(self.prefix)
         assert activate_data == dals("""
         $CONDA_DEFAULT_ENV = '%(native_prefix)s'
+        $CONDA_EXE = '%(conda_exe)s'
         $CONDA_PREFIX = '%(native_prefix)s'
         $CONDA_PROMPT_MODIFIER = '(%(native_prefix)s) '
         $CONDA_PYTHON_EXE = '%(sys_executable)s'
@@ -719,6 +731,7 @@ def test_xonsh_basic(self):
             'new_path': activator.pathsep_join(new_path_parts),
             'sys_executable': activator.path_conversion(sys.executable),
             'activate1': activator.path_conversion(join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.xsh')),
+            'conda_exe': activator.path_conversion(context.conda_exe),
         }
 
         with env_vars({
@@ -763,6 +776,7 @@ def test_xonsh_basic(self):
             assert deactivate_data == dals("""
             source "%(deactivate1)s"
             del $CONDA_DEFAULT_ENV
+            del $CONDA_EXE
             del $CONDA_PREFIX
             del $CONDA_PROMPT_MODIFIER
             del $CONDA_PYTHON_EXE
@@ -786,6 +800,7 @@ def test_fish_basic(self):
         new_path_parts = activator._add_prefix_to_path(self.prefix)
         assert activate_data == dals("""
         set -gx CONDA_DEFAULT_ENV "%(native_prefix)s";
+        set -gx CONDA_EXE "%(conda_exe)s";
         set -gx CONDA_PREFIX "%(native_prefix)s";
         set -gx CONDA_PROMPT_MODIFIER "(%(native_prefix)s) ";
         set -gx CONDA_PYTHON_EXE "%(sys_executable)s";
@@ -798,6 +813,7 @@ def test_fish_basic(self):
             'new_path': activator.pathsep_join(new_path_parts),
             'sys_executable': activator.path_conversion(sys.executable),
             'activate1': activator.path_conversion(join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.fish')),
+            'conda_exe': activator.path_conversion(context.conda_exe),
         }
 
         with env_vars({
@@ -836,6 +852,7 @@ def test_fish_basic(self):
             assert deactivate_data == dals("""
             source "%(deactivate1)s";
             set -e CONDA_DEFAULT_ENV;
+            set -e CONDA_EXE;
             set -e CONDA_PREFIX;
             set -e CONDA_PROMPT_MODIFIER;
             set -e CONDA_PYTHON_EXE;
@@ -860,6 +877,7 @@ def test_powershell_basic(self):
         new_path_parts = activator._add_prefix_to_path(self.prefix)
         assert activate_data == dals("""
         $env:CONDA_DEFAULT_ENV = "%(prefix)s"
+        $env:CONDA_EXE = "%(conda_exe)s"
         $env:CONDA_PREFIX = "%(prefix)s"
         $env:CONDA_PROMPT_MODIFIER = "(%(prefix)s) "
         $env:CONDA_PYTHON_EXE = "%(sys_executable)s"
@@ -870,7 +888,8 @@ def test_powershell_basic(self):
             'prefix': self.prefix,
             'new_path': activator.pathsep_join(new_path_parts),
             'sys_executable': sys.executable,
-            'activate1': join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.ps1')
+            'activate1': join(self.prefix, 'etc', 'conda', 'activate.d', 'activate1.ps1'),
+            'conda_exe': context.conda_exe,
         }
 
         with env_vars({
@@ -909,6 +928,7 @@ def test_powershell_basic(self):
             assert deactivate_data == dals("""
             . "%(deactivate1)s"
             Remove-Variable CONDA_DEFAULT_ENV
+            Remove-Variable CONDA_EXE
             Remove-Variable CONDA_PREFIX
             Remove-Variable CONDA_PROMPT_MODIFIER
             Remove-Variable CONDA_PYTHON_EXE