Repository: jupyter/jupyter_core

Issue:
Make  JUPYTER_PREFER_ENV_PATH=1 default
I propose that `JUPYTER_PREFER_ENV_PATH=1` be made the default behavior.

I cannot imagine a case where someone would set up a virtual environment and not expect the python based software such as Jupyter lab to use what is installed in the virtual environment first. Thus, I believe Jupyter lab should never use versions of extensions, etc outside of the virtual environment in preference to those inside. I can imagine people wanting to set up some utilities that work in all environments. So, can see there might be issues surrounding that. I suggest the following logic:

1. If a package is installed in the virtual environment use that in preference to any other available versions.
2. If a package does not exist in the virtual environment but is available at the user level use it in preference to versions available at the system level.
3. Only use versions available at the system level if they have not been overridden at the previous two levels.
4. There may need to be a switch to not use packages from the user or system level.

I encountered an unexpected issue with ipywidgets because of this (see https://github.com/jupyter-widgets/ipywidgets/issues/3559).

The original implementation was discussed in https://github.com/jupyter/jupyter_core/pull/199

Thanks for a great tool.

Jonathan

Discussion:
jasongrout: Of course, several tests fail. @gutow (or anyone else), feel free to take over this if you can.
gutow: I am trying to pick this up. But am having trouble setting up an environment where I can run tests locally.

It looks to me like line 29 is causing the error in the CI testing:
```
def envset(name, default=False):
```
Should be: 
```
def envset(name, default="no"):
```
To be consistent with the vocabulary used previously.

I get errors on running the tests that appear to be caused by isolating the tests in a virtual environment. They are not the same as occur in the github CI testing.

So, I think I would have to set up a virtual machine with no virtual environments to do the tests on my local machine. I have lots of different projects on my machine, so have them all isolated in virtual environments. I definitely do not want to set this one up outside of a virtual environment. Am I on the right track here?
jasongrout: > It looks to me like line 29 of paths.py is causing the error in the CI testing:

Actually, most of the errors came from an actual bug in my code. Doh! https://github.com/jupyter/jupyter_core/pull/286/commits/8b5f1c5598950ad4890f0daca3deebdeeac9ea08

Patch:
diff --git a/jupyter_core/command.py b/jupyter_core/command.py
index 43698c0..33804c9 100644
--- a/jupyter_core/command.py
+++ b/jupyter_core/command.py
@@ -231,13 +231,13 @@ def main():
                 if args.debug:
                     env = os.environ
 
-                    if paths.envset("JUPYTER_PREFER_ENV_PATH"):
+                    if paths.prefer_environment_over_user():
                         print(
-                            "JUPYTER_PREFER_ENV_PATH is set, making the environment-level path preferred over the user-level path for data and config"
+                            "JUPYTER_PREFER_ENV_PATH is set to a true value, or JUPYTER_PREFER_ENV_PATH is not set and we detected a virtual environment, making the environment-level path preferred over the user-level path for data and config"
                         )
                     else:
                         print(
-                            "JUPYTER_PREFER_ENV_PATH is not set, making the user-level path preferred over the environment-level path for data and config"
+                            "JUPYTER_PREFER_ENV_PATH is set to a false value, or JUPYTER_PREFER_ENV_PATH is not set and we did not detect a virtual environment, making the user-level path preferred over the environment-level path for data and config"
                         )
 
                     # config path list
diff --git a/jupyter_core/paths.py b/jupyter_core/paths.py
index f2716a8..7dafd91 100644
--- a/jupyter_core/paths.py
+++ b/jupyter_core/paths.py
@@ -26,13 +26,18 @@ pjoin = os.path.join
 UF_HIDDEN = getattr(stat, "UF_HIDDEN", 32768)
 
 
-def envset(name):
-    """Return True if the given environment variable is set
+def envset(name, default=False):
+    """Return the boolean value of a given environment variable.
 
     An environment variable is considered set if it is assigned to a value
     other than 'no', 'n', 'false', 'off', '0', or '0.0' (case insensitive)
+
+    If the environment variable is not defined, the default value is returned.
     """
-    return os.environ.get(name, "no").lower() not in ["no", "n", "false", "off", "0", "0.0"]
+    if name not in os.environ:
+        return default
+
+    return os.environ[name].lower() not in ["no", "n", "false", "off", "0", "0.0"]
 
 
 def get_home_dir():
@@ -47,6 +52,23 @@ def get_home_dir():
 _dtemps: dict = {}
 
 
+def prefer_environment_over_user():
+    """Determine if environment-level paths should take precedence over user-level paths."""
+    # If JUPYTER_PREFER_ENV_PATH is defined, that signals user intent, so return its value
+    if "JUPYTER_PREFER_ENV_PATH" in os.environ:
+        return envset("JUPYTER_PREFER_ENV_PATH")
+
+    # If we are in a Python virtualenv, default to True (see https://docs.python.org/3/library/venv.html#venv-def)
+    if sys.prefix != sys.base_prefix:
+        return True
+
+    # If sys.prefix indicates Python comes from a conda/mamba environment, default to True
+    if "CONDA_PREFIX" in os.environ and sys.prefix.startswith(os.environ["CONDA_PREFIX"]):
+        return True
+
+    return False
+
+
 def _mkdtemp_once(name):
     """Make or reuse a temporary directory.
 
@@ -184,7 +206,7 @@ def jupyter_path(*subdirs):
 
     env = [p for p in ENV_JUPYTER_PATH if p not in SYSTEM_JUPYTER_PATH]
 
-    if envset("JUPYTER_PREFER_ENV_PATH"):
+    if prefer_environment_over_user():
         paths.extend(env)
         paths.extend(user)
     else:
@@ -253,7 +275,7 @@ def jupyter_config_path():
 
     env = [p for p in ENV_CONFIG_PATH if p not in SYSTEM_CONFIG_PATH]
 
-    if envset("JUPYTER_PREFER_ENV_PATH"):
+    if prefer_environment_over_user():
         paths.extend(env)
         paths.extend(user)
     else:


Test patch:
diff --git a/jupyter_core/tests/test_command.py b/jupyter_core/tests/test_command.py
index 4ef38cd..fb54005 100644
--- a/jupyter_core/tests/test_command.py
+++ b/jupyter_core/tests/test_command.py
@@ -23,7 +23,6 @@ resetenv = patch.dict(os.environ)
 
 def setup_module():
     resetenv.start()
-    os.environ.pop("JUPYTER_PREFER_ENV_PATH", None)
 
 
 def teardown_module():
diff --git a/jupyter_core/tests/test_paths.py b/jupyter_core/tests/test_paths.py
index 36811f6..0d16358 100644
--- a/jupyter_core/tests/test_paths.py
+++ b/jupyter_core/tests/test_paths.py
@@ -25,6 +25,7 @@ from jupyter_core.paths import (
     jupyter_data_dir,
     jupyter_path,
     jupyter_runtime_dir,
+    prefer_environment_over_user,
     secure_write,
 )
 
@@ -61,13 +62,16 @@ no_config_env = patch.dict(
 jupyter_config_env = "/jupyter-cfg"
 config_env = patch.dict("os.environ", {"JUPYTER_CONFIG_DIR": jupyter_config_env})
 prefer_env = patch.dict("os.environ", {"JUPYTER_PREFER_ENV_PATH": "True"})
+prefer_user = patch.dict("os.environ", {"JUPYTER_PREFER_ENV_PATH": "False"})
 
 resetenv = patch.dict(os.environ)
 
 
 def setup_module():
     resetenv.start()
-    os.environ.pop("JUPYTER_PREFER_ENV_PATH", None)
+    # For these tests, default to preferring the user-level over environment-level paths
+    # Tests can override this preference using the prefer_env decorator/context manager
+    os.environ["JUPYTER_PREFER_ENV_PATH"] = "no"
 
 
 def teardown_module():
@@ -89,7 +93,10 @@ def test_envset():
             assert paths.envset(f"FOO_{v}")
         for v in false_values:
             assert not paths.envset(f"FOO_{v}")
-        assert not paths.envset("THIS_VARIABLE_SHOULD_NOT_BE_SET")
+        # Test default value is False
+        assert paths.envset("THIS_VARIABLE_SHOULD_NOT_BE_SET") is False
+        # Test envset returns the given default if supplied
+        assert paths.envset("THIS_VARIABLE_SHOULD_NOT_BE_SET", None) is None
 
 
 @pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
@@ -328,6 +335,21 @@ def test_jupyter_config_path_env():
     assert path[:2] == [pjoin("foo", "bar"), pjoin("bar", "baz")]
 
 
+def test_prefer_environment_over_user():
+    with prefer_env:
+        assert prefer_environment_over_user() is True
+
+    with prefer_user:
+        assert prefer_environment_over_user() is False
+
+    # Test default if environment variable is not set, and try to determine if we are in a virtual environment
+    os.environ.pop("JUPYTER_PREFER_ENV_PATH", None)
+    in_venv = sys.prefix != sys.base_prefix or (
+        "CONDA_PREFIX" in os.environ and sys.prefix.startswith(os.environ["CONDA_PREFIX"])
+    )
+    assert prefer_environment_over_user() is in_venv
+
+
 def test_is_hidden():
     with tempfile.TemporaryDirectory() as root:
         subdir1 = os.path.join(root, "subdir")