Repository: conda/conda

Issue:
Allow activating multiple environments
Hi,
I'm trying to configure conda to use it on a computing cluster with command line access for multiple users (~300 accounts).
The idea is that we will provide to users some preinstalled conda environments that they can use as they need.

Some people will want to activate several of these environments to perform their compute task. The problem is that the `activate` script calls `deactivate` before doing the activation job, so users can only have 1 environment loaded.

I know that activating multiple environments could be a problem as some incompatibility could arise depending on the packages installed in each env. But in our case, in many situations, it would work perfectly well (and if it's not, users would be able to take care of it).

So I'm wondering if you think this feature could be added to conda somehow? Maybe with a special option for the activate script, and a modification to deactivate to allow it to deactivate _all_ the loaded env.
I can work on a PR if you are ok with the idea.

The other option would be to force them to adopt the proper conda way (they create their own environments), but it would make the transition too hard for most of them...



Discussion:
So, if I understand correctly, you're proposing that it be possible to activate more than one environment at a time by removing the deactivate step from the activate script.  Then you would add an option to deactivate that would allow it to deactivate all the previously activated environments.  Is that right?

Activating an environment prepends that environment's directories to PATH.  Thus activating two environments at the same time in the same terminal session is somewhat ambigous.  Which environment should have precedence?  Since we can't be ambigous about what environment is active, one environment can be active in a session at a time.

Each user should be able to activate a conda environment for each terminal session they have open.  Each terminal session has its own shell environment. For example, on my linux system, if I want to activate two conda environments, I need to open two terminal sessions.  Since each terminal session has its own shell environment, I can activate a different conda environment in each session without any problems.  There are no conda environments the need to be deactivated.

When using preinstalled environments, it might be a good idea to ensure somehow that users don't change the preinstalled conda environment.  One problem that can arise is when a user upgrades packages that breaks another users's code.  This is one of the several problems that conda environments solve very well.

I would, of course, highly encourage you to educate your users about conda environments.  Environments are pretty powerful and a little knowledge goes a long way.

If I've misunderstood your post, please let me know.  I think the above information should help you.

This issue relates to #2502, and I'd encourage you to weigh in there. @pzwang has proposed the ability to "stack" envs, pretty much as you've described. I consider #2502 a first step toward that.

Also, implementing #2502 would resolve https://github.com/conda/conda/issues/3411, which often happens in cluster environments.

> So, if I understand correctly, you're proposing that it be possible to activate more than one environment at a time by removing the deactivate step from the activate script. Then you would add an option to deactivate that would allow it to deactivate all the previously activated environments. Is that right?

Yep, that's it

> Activating an environment prepends that environment's directories to PATH. Thus activating two environments at the same time in the same terminal session is somewhat ambigous. Which environment should have precedence? Since we can't be ambigous about what environment is active, one environment can be active in a session at a time.

Precedence would depend on the order or activation, ie each new env gets its path prepended to PATH. That's how it works currently on our cluster with simple bash scripts, and it works not so bad for most of the softs we use.

> Each user should be able to activate a conda environment for each terminal session they have open. Each terminal session has its own shell environment. For example, on my linux system, if I want to activate two conda environments, I need to open two terminal sessions. Since each terminal session has its own shell environment, I can activate a different conda environment in each session without any problems. There are no conda environments the need to be deactivated.

Hum, yes, no problem in having multiple shells with specific activated env. Our use case is that user want to run as a job a big pipeline that uses many different software, that are womewhat unrelated. A script like this:

```

#!/bin/bash
source activate softA
source activate softB
source activate softC
source activate softD

my_huge_pipeline.sh|py|pl|... --that uses --internally the -4 softs
```

> When using preinstalled environments, it might be a good idea to ensure somehow that users don't change the preinstalled conda environment. One problem that can arise is when a user upgrades packages that breaks another users's code. This is one of the several problems that conda environments solve very well.

I _think_ it's not a big problem if we don't give write permissions to users for the system-wide conda, and let them create their own environments (if they need to) in their /home:

```
conda create -p /home/my_env softA softB
source activate /home/my_env
```

> I would, of course, highly encourage you to educate your users about conda environments. Environments are pretty powerful and a little knowledge goes a long way.

Yes, I agree, but that will take a looong time I think... Most of our users are not power users.

@kalefranz right, #2502 looks like a good first step, but still a little bit different as here I'd like to stack multiple envs.

@abretaud I think I understand your use case a little better now.  AFAIK, what you are proposing should technically work.  

My opinion is: Having a few envs for certain specific purposes is fine. Those purposes are:

```
1. To store only conda in order to simplify conda's code.
2. To store the root/Administrator installed, read only packages that everyone can use.
3. To store the user's customizations of that env.
```

Making this arbitrarily cascadable would be really confusing and difficult to manage and communicate. We'd also run out of PATH on Windows quite quickly.

Actually, a change like this would start morphing Conda into something more akin to lmod.

An interesting concept with potential.

I would personally want a solution like this to have a broader understanding of what kinds of environments are being activated. That way if say the first/base environment activated is Python 3.5, any subsequent activations will ensure that the Python versions match (to an extent). This should eliminate the most obvious dangers.

In conjunction with the suggested boolean variable `stack_envs` from #2502, I would suggest something like a boolean variable `deactivate_on_activate`. I'm sure these variables are self explanatory in the scope of this discussion.

Basically, in peoples mind, there are 3 types of environments:
- the main python environment
- additional packages environment
- additional binary/lib environment

So each environment can be treated as source for python, source for packages (using site packages environment), source for bin/libs.

Let's assume we have
- EnvA
  - binA
  - PythonA
  - PkgA1
  - PkgA2
- EnvB
  - binB
- EnvC
  - binC
  - PythonC
  - PkgC1
  - PkgC2

So I would suggest extending the activation logic to something like this:

``` bash
activate EnvA
activate -p EnvB
activate -b EnvC
```

Would setup something like this:
- Env
  - binC:binA
  - PythonA
  - PkgA1
  - PkgA2

In contrast:

``` bash
activate EnvA
activate -b EnvB
activate -p EnvC
```

Would setup something like this:
- Env
  - binB:binA
  - PythonA
  - PkgA1
  - PkgA2
  - PkgC1
  - PkgC2

Or even weird as:

``` bash
activate EnvC
activate -b EnvA
activate -b EnvC
```

Would setup something like this (to make sure that the order of search for packages would be preserved as needed):
- Env
  - binC:binA:binC
  - PythonC
  - PkgC1
  - PkgC2

Regarding deactivate - by default it will deactivate the last activate.
There will be also additional switch to make a specific deactivation of some environment.
And additionally a way to deactivate all environments.

This will allow to write strange scripts like this one:

``` bash
activate EnvA
   do_only for_A
   activate -b EnvB
      do_for_A_and_B
   deactivate
   activate -p EnvC
      do_for_A_and_C

deactivate -all
do_something_else
```

I'm just thinking that without decent definition of atomic operations within activate - there is no way to stack the environments properly.

I contend there's little value to this. Conda envs are cheap to create and destroy and we shouldn't complicate things with this stuff. I stand by my 3 and 3 only.

Again, just like @njalerikson said [here](https://github.com/conda/conda/issues/3580#issuecomment-252082291): introducing stacking might move activate/deactivate closer to lmod behavior and, who knows, may be there is a need to think about integration some of the conda stuff with lmod... 

@mingwandroid is right in my opinion.  Conda environments, in almost all cases, are incredibly cheap in terms of overhead and weight.  The size limit on PATH is also another factor.  Having separate environments means now you have to maintain many more conda environments.  Packages in every environment now have to be maintained and updated.  Environments are sure to overlap and shadow each other.  You now need to track binaries and libraries in each environment. On top of that, order of activation would be incredibly important.

Not only would you have to understand environments, but you would have to understand how environments interact with each other.

I've had to read about `lmod` just now, and I don't think being like `lmod` is a design goal for `conda` nor do I think `lmod` has the same constraints as `conda`.

Something that hasn't been addressed (if it has I missed it). Why can't you just install all the packages you need into a single environment? Installing the same package into multiple environments is extremely cheap, since conda uses hard linking. 

I agree, creating environments is cheap and is the _right_ way.

But we have _many_ users, and some software already available in the described way (and not enough time to port them to conda in a reasonable time frame).
So if we begin to use conda the default way, people will have to deal with 2 logics: some softs can be stacked, others not... I expect a deluge of support requests if we do this (and we're thinking about using conda to save us time, so...).

On the other hand, currently we use the described behavior with our local tools, and we get very few user requests of people having problems.

Again I perfectly understand that this should not be the default behavior, and that it should only be activated by people who are ready to assume the consequences (ie, teach impacted users how to create their own env).

I don't know if you have all seen it: I opened the #3924 PR for this, your feedback is welcome!
What do you mean by "some softs can be stacked, others not"?

Resolving patch:
diff --git a/conda/activate.py b/conda/activate.py
--- a/conda/activate.py
+++ b/conda/activate.py
@@ -110,10 +110,13 @@ def build_activate(self, name_or_prefix):
         # query environment
         old_conda_shlvl = int(os.getenv('CONDA_SHLVL', 0))
         old_conda_prefix = os.getenv('CONDA_PREFIX')
+        max_shlvl = self.context.max_shlvl
 
         if old_conda_prefix == prefix:
             return self.build_reactivate()
-        elif old_conda_shlvl == 2 and os.getenv('CONDA_PREFIX_1') == prefix:
+        elif os.getenv('CONDA_PREFIX_%s' % (old_conda_shlvl-1)) == prefix:
+            # in this case, user is attenmpting to activate the previous environment,
+            #  i.e. step back down
             return self.build_deactivate()
 
         activate_scripts = glob(join(
@@ -122,6 +125,7 @@ def build_activate(self, name_or_prefix):
         conda_default_env = self._default_env(prefix)
         conda_prompt_modifier = self._prompt_modifier(conda_default_env)
 
+        assert 0 <= old_conda_shlvl <= max_shlvl
         if old_conda_shlvl == 0:
             new_path = self.pathsep_join(self._add_prefix_to_path(prefix))
             set_vars = {
@@ -133,30 +137,28 @@ def build_activate(self, name_or_prefix):
                 'CONDA_PROMPT_MODIFIER': conda_prompt_modifier,
             }
             deactivate_scripts = ()
-        elif old_conda_shlvl == 1:
-            new_path = self.pathsep_join(self._add_prefix_to_path(prefix))
+        elif old_conda_shlvl == max_shlvl:
+            new_path = self.pathsep_join(self._replace_prefix_in_path(old_conda_prefix, prefix))
             set_vars = {
                 'PATH': new_path,
                 'CONDA_PREFIX': prefix,
-                'CONDA_PREFIX_%d' % old_conda_shlvl: old_conda_prefix,
-                'CONDA_SHLVL': old_conda_shlvl + 1,
                 'CONDA_DEFAULT_ENV': conda_default_env,
                 'CONDA_PROMPT_MODIFIER': conda_prompt_modifier,
             }
-            deactivate_scripts = ()
-        elif old_conda_shlvl == 2:
-            new_path = self.pathsep_join(self._replace_prefix_in_path(old_conda_prefix, prefix))
+            deactivate_scripts = glob(join(
+                old_conda_prefix, 'etc', 'conda', 'deactivate.d', '*' + self.script_extension
+            ))
+        else:
+            new_path = self.pathsep_join(self._add_prefix_to_path(prefix))
             set_vars = {
                 'PATH': new_path,
                 'CONDA_PREFIX': prefix,
+                'CONDA_PREFIX_%d' % old_conda_shlvl: old_conda_prefix,
+                'CONDA_SHLVL': old_conda_shlvl + 1,
                 'CONDA_DEFAULT_ENV': conda_default_env,
                 'CONDA_PROMPT_MODIFIER': conda_prompt_modifier,
             }
-            deactivate_scripts = glob(join(
-                old_conda_prefix, 'etc', 'conda', 'deactivate.d', '*' + self.script_extension
-            ))
-        else:
-            raise NotImplementedError()
+            deactivate_scripts = ()
 
         return {
             'unset_vars': (),
@@ -174,6 +176,7 @@ def build_deactivate(self):
         new_conda_shlvl = old_conda_shlvl - 1
         new_path = self.pathsep_join(self._remove_prefix_from_path(old_conda_prefix))
 
+        assert old_conda_shlvl > 0
         if old_conda_shlvl == 1:
             # TODO: warn conda floor
             unset_vars = (
@@ -187,7 +190,7 @@ def build_deactivate(self):
                 'CONDA_SHLVL': new_conda_shlvl,
             }
             activate_scripts = ()
-        elif old_conda_shlvl == 2:
+        else:
             new_prefix = os.getenv('CONDA_PREFIX_%d' % new_conda_shlvl)
             conda_default_env = self._default_env(new_prefix)
             conda_prompt_modifier = self._prompt_modifier(conda_default_env)
@@ -203,8 +206,6 @@ def build_deactivate(self):
                 'CONDA_PROMPT_MODIFIER': conda_prompt_modifier,
             }
             activate_scripts = self._get_activate_scripts(new_prefix)
-        else:
-            raise NotImplementedError()
 
         return {
             'unset_vars': unset_vars,
diff --git a/conda/base/context.py b/conda/base/context.py
--- a/conda/base/context.py
+++ b/conda/base/context.py
@@ -91,8 +91,9 @@ class Context(Configuration):
     default_python = PrimitiveParameter('%d.%d' % sys.version_info[:2],
                                         element_type=string_types + (NoneType,))
     disallow = SequenceParameter(string_types)
-    force_32bit = PrimitiveParameter(False)
     enable_private_envs = PrimitiveParameter(False)
+    force_32bit = PrimitiveParameter(False)
+    max_shlvl = PrimitiveParameter(2)
     path_conflict = PrimitiveParameter(PathConflict.clobber)
     pinned_packages = SequenceParameter(string_types, string_delimiter='/')  # TODO: consider a different string delimiter  # NOQA
     rollback_enabled = PrimitiveParameter(True)
@@ -471,6 +472,7 @@ def list_parameters(self):
             'default_python',
             'enable_private_envs',
             'force_32bit',
+            'max_shlvl',
             'migrated_custom_channels',
             'prune',
             'respect_pinned',


Test patch: