Repository: pypa/pip

Issue:
Unexpected --user behavior since version 23.1
### Description

Hey everyone,
I'm using pip typically in virtual environments (managed by PyCharm). Since version 23.1 I get an error message when installing packages.

```
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
```

A little research told me, that a --user flag within a venv does not really make sense. However, it seems that some kind of default --user is set since 23.1.

A workaround I found is to add the --no-user flag.   

```
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy --no-user
Collecting numpy
   Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2
```  

For completeness, downgrading to 23.0.1 with the --no-user flag gives the me the behavior I'm used to.
 
```
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip --version
pip 23.1 from c:\users\florian.wiese\pycharmprojects\guimanager\venv\lib\site-packages\pip (python 3.9)

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>python.exe -m pip install --upgrade pip==23.0.1 --no-user
Collecting pip==23.0.1
  Using cached pip-23.0.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.1
    Uninstalling pip-23.1:
      Successfully uninstalled pip-23.1
Successfully installed pip-23.0.1

(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip install numpy
Collecting numpy
  Using cached numpy-1.24.2-cp39-cp39-win_amd64.whl (14.9 MB)
Installing collected packages: numpy
Successfully installed numpy-1.24.2

[notice] A new release of pip is available: 23.0.1 -> 23.1
[notice] To update, run: python.exe -m pip install --upgrade pip
```
  
Not sure if this is a bug or a feature. Maybe I'm also missing something very obvious. But at least it is documented now for anyone experiencing the same issue :)

### Expected behavior

_No response_

### pip version

23.1

### Python version

3.9

### OS

Windows 10

### How to Reproduce

create a venv with pip 23.1
try to install a package with `pip install <any-package-name>`

### Output

`ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.`

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
Pip 23.1 does not have a default to `--user`. However it's possible you are picking up a user config of some sort. Please can you provide the output of `pip config debug` in both the failing and the successful cases?
Sure, thanks in advance.

Pip 23.0.1 (successful case):
```
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip config debug
env_var:
env:
global:
  C:\ProgramData\pip\pip.ini, exists: False
site:
  C:\Users\florian.wiese\PycharmProjects\GUIManager\venv\pip.ini, exists: False
user:
  C:\Users\florian.wiese\pip\pip.ini, exists: False
  C:\Users\florian.wiese\AppData\Roaming\pip\pip.ini, exists: False
```

Pip 23.1 (Failing case):
```
(venv) C:\Users\florian.wiese\PycharmProjects\GUIManager>pip config debug
env_var:
base:
  C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\pip.ini, exists: True
    install.user: yes
    uninstall.user: yes
    freeze.user: yes
    list.user: yes
env:
global:
  C:\ProgramData\pip\pip.ini, exists: False
site:
  C:\Users\florian.wiese\PycharmProjects\GUIManager\venv\pip.ini, exists: False
user:
  C:\Users\florian.wiese\pip\pip.ini, exists: False
  C:\Users\florian.wiese\AppData\Roaming\pip\pip.ini, exists: False
```

So that's your issue. In the failing case, you have a `base` config that sets `user` to true. This looks like the Windows Store installation of Python. I'm not sure why they set a pip config (I think it's a workaround for a permission issue with Windows Store apps) but IMO setting `user` globally like this is a bad idea. But it's what we have to deal with, unfortunately.

This was caused by

> In the case of virtual environments, configuration files are now also included from the base installation. ([#9752](https://github.com/pypa/pip/issues/9752))

That issue was extensively discussed, but it appears that the case of the Windows Store distribution wasn't known (or wasn't considered). IMO this is a sufficiently serious problem that it requires a bugfix. I'm not sure it's possible to simply revert the change (reverting a merge commit is a bit beyond my git skills!) so we'll need someone to work on a fix ASAP. @pelson as the author of the original change, are you able to look at this? Alternatively, if anyone can tell me the right git incantation to revert #11487, I can do that and we can revisit the feature for a future pip version.

Thanks for reporting this @flowmeadow!
As a quick proof of concept, I've created #11983 to patch out the BASE config. This might form the basis of a minimal fix if we can't find a better solution.
I also opened https://github.com/python/cpython/issues/103646 to discuss with CPython whether there's a better way than a pre-installed `pip.ini` to achieve what they are trying to do with the Store installation.
Thanks a lot for the analysis. I can confirm that its the Windows Store installation. I remember having a very bad feeling using it. Well, at least this was the first issue that arose in almost two years.
Generally, I think the Store install should be fine. There's some quirks like this, but they mostly seem to work. And having people use the distribution and report issues like you have is the only way we'll know what needs improving, so your help is appreciated.
jFYI, this is also impacting `proxy.py` workflows e.g. https://github.com/abhinavsingh/proxy.py/actions/runs/4717368971/jobs/8450593742
@abhinavsingh that's nothing to do with this issue. That's the deprecation of `pkg_resources`, which is #11975 (and which is now fixed in main, and awaiting a bugfix release which will come once we've established what fixes we need to include).
By the way @flowmeadow, as a temporary workaround, you should be able to set the `user` option to `false` in a config file that's higher priority than the "base" one (or just set `$PIP_USER=0` in your environment).
> So that's your issue. In the failing case, you have a `base` config that sets `user` to true. This looks like the Windows Store installation of Python. I'm not sure why they set a pip config (I think it's a workaround for a permission issue with Windows Store apps) but IMO setting `user` globally like this is a bad idea

My understanding is that this is the model of Windows Store installs, the base install is in a global space that any specific user may not have permissions to modify, therefore the application must store user specific data in a user space.

But unlike Linux distros which manage the whole Python pipeline for the system install such as installing Python packages like Pandas using their own distribution method outside Pip there is no such infrastructure on the Windows Store and users are expected to use Pip install.

Maybe a Windows specific exception will need to be made here as it's model of users is not a 1-1 mapping to Unix-like model of users and root anyway.

I'm going to ping @zooba  just in case he wants to comment here (and maybe correct any misunderstandings).

I do wishfully also wonder if the default behavior of installing Python on Windows could in the future switch from the the Windows Store to `winget` and behave more like Linux distros? But I'm throwing that idea out with no particular understanding of the problems, so it may be a faulty idea.
> switch from the the Windows Store to `winget` and behave more like Linux distros?

Show me a Linux distro that downloads our installer and runs it with unreliable command line options and I'll consider it (spoiler: no I won't 😉 `winget` is not a good way to manage Python installs, I'm afraid).

>  just in case he wants to comment here (and maybe correct any misunderstandings)

I've put plenty of explanation on the CPython bug already. My fundamental disagreement is with configurations located in `sys.base_prefix` being used when `sys.base_prefix != sys.prefix`, as it makes it impossible to use a `sys.prefix` configuration in something that may be used as a base environment.

Perhaps `sys.base_prefix` configuration files should use a different filename, or different section names or something. Or some extra setting to say "please consider the settings in this file for venvs created from this environment".

I can imagine useful scenarios for files in both `sys.base_prefix` and also `sys.prefix when ==sys.base_prefix`, but they all assume the distributor knows what you should use by default (e.g. platform name, index-url, etc.). If pip wants to say that distributors don't know the right install locations, that's fine, we distributors will just patch source files instead of adding overridable config settings.
> My fundamental disagreement is with configurations located in sys.base_prefix being used when sys.base_prefix != sys.prefix, as it makes it impossible to use a sys.prefix configuration in something that may be used as a base environment.

To be clear, there's no such thing as a "`sys.prefix` configuration" as you describe it. If you look in [the documentation](https://pip.pypa.io/en/stable/topics/configuration/), the location `{sys.prefix}/pip.ini` isn't mentioned as a source of config data, although in the code, it's used as the location of the "site" data when you're not in a virtual environment. I think that's probably just a problem with the docs, though, and what you *mean* here is the "site" location.

I do agree that it's wrong to have the "base" config come from the same place as the "site" config of the system Python, precisely because as you say it means that the system site config "leaks" into a virtual environment. I think that's a flaw in the design of #9752 which we should fix[^1].

[^1]: As a minor point, I'm not keen on having a different basename for the file - knowing that pip's config file is always called `pip.ini` is a useful consistency for people trying to work out what's going on.

As a general point, I personally don't consider pip's configuration files as intended for distributors - they are for users and site admins to manage. Distributors shouldn't be using them, and very definitely shouldn't be relying on what's in them. If distributors need configuration options, they should discuss that with the pip maintainers and we can see how best to address such a need. But that's just my personal view - I'd be interested if the other @pypa/pip-committers agree with me on that.

> If pip wants to say that distributors don't know the right install locations, that's fine, we distributors will just patch source files instead of adding overridable config settings.

I think that's an interesting use case that we should discuss further. It *may* be a reasonable use for certain types of config. One key difference if a distributor patches source files is that the support burden is squarely on the distributor in that case. One problem with diagnosing *this* issue was that it took a while to realise that the offending user config wasn't actually set by the user, and the user didn't know about it.
Same issue here, for now I just reverted back to pip 21.3 and everything worked again. Although, in my case both pip 23.0.1 and pip 23.1 are failing with the same error
> By the way @flowmeadow, as a temporary workaround, you should be able to set the `user` option to `false` in a config file that's higher priority than the "base" one (or just set `$PIP_USER=0` in your environment).

For solution 1 how do u find a file that is higher priority than the base file? I was wondering if its one of these files I will have to change? 
![image](https://user-images.githubusercontent.com/61637151/233489926-148a2140-3c3a-432d-903e-bc35958bbe54.png)

If it is one of the user or global files, I have tried adding this to the pip.ini, but it did not work:
```
[global]
user=false
```

For solution 2, I tried using `$PIP_USER=0` while in the venv it gives an error saying it is not a recognized internal or external command. Is there a specific way to set or run this? 



@pfmoore How do you feel about reverting and reopening https://github.com/pypa/pip/issues/9752, and we can come back around to re-adding that in a follow up change for 23.2?

@pradyunsg As I said [above](https://github.com/pypa/pip/issues/11982#issuecomment-1516015833) I'd be happy to do that but I don't know how (in git) to revert a merge commit. If you can confirm how I do that I'd be far happier with that approach.

On the longer term resolution of #9752, I'd point out that this problem has reinforced my dislike of having the "base" environment leak into virtual environments like this, so I'd actually oppose #9752 in its stated form. I'd like a much better justification of why the global setting isn't sufficient for the original use case, and assuming the change *can* be justified, I'd want the "base" config to be very explicitly *not* the same as the "site" config of the base environment. So I think there's more than just "re-adding" the feature involved here, more like "re-evaluating" and "re-designing" it in a way that accounts for the way the Windows Store distribution uses the existing config mechanism.
Whoop, I missed that while skimming the issue! Sorry about that!

`git revert -m 1 <hash>`

(at PyCon US, so my replies might be slow!)

Resolving patch:
diff --git a/src/pip/_internal/configuration.py b/src/pip/_internal/configuration.py
--- a/src/pip/_internal/configuration.py
+++ b/src/pip/_internal/configuration.py
@@ -36,20 +36,12 @@
 kinds = enum(
     USER="user",  # User Specific
     GLOBAL="global",  # System Wide
-    BASE="base",  # Base environment specific (e.g. for all venvs with the same base)
-    SITE="site",  # Environment Specific (e.g. per venv)
+    SITE="site",  # [Virtual] Environment Specific
     ENV="env",  # from PIP_CONFIG_FILE
     ENV_VAR="env-var",  # from Environment Variables
 )
-OVERRIDE_ORDER = (
-    kinds.GLOBAL,
-    kinds.USER,
-    kinds.BASE,
-    kinds.SITE,
-    kinds.ENV,
-    kinds.ENV_VAR,
-)
-VALID_LOAD_ONLY = kinds.USER, kinds.GLOBAL, kinds.BASE, kinds.SITE
+OVERRIDE_ORDER = kinds.GLOBAL, kinds.USER, kinds.SITE, kinds.ENV, kinds.ENV_VAR
+VALID_LOAD_ONLY = kinds.USER, kinds.GLOBAL, kinds.SITE
 
 logger = getLogger(__name__)
 
@@ -78,7 +70,6 @@ def get_configuration_files() -> Dict[Kind, List[str]]:
         os.path.join(path, CONFIG_BASENAME) for path in appdirs.site_config_dirs("pip")
     ]
 
-    base_config_file = os.path.join(sys.base_prefix, CONFIG_BASENAME)
     site_config_file = os.path.join(sys.prefix, CONFIG_BASENAME)
     legacy_config_file = os.path.join(
         os.path.expanduser("~"),
@@ -87,7 +78,6 @@ def get_configuration_files() -> Dict[Kind, List[str]]:
     )
     new_config_file = os.path.join(appdirs.user_config_dir("pip"), CONFIG_BASENAME)
     return {
-        kinds.BASE: [base_config_file],
         kinds.GLOBAL: global_config_files,
         kinds.SITE: [site_config_file],
         kinds.USER: [legacy_config_file, new_config_file],
@@ -354,8 +344,6 @@ def iter_config_files(self) -> Iterable[Tuple[Kind, List[str]]]:
             # The legacy config file is overridden by the new config file
             yield kinds.USER, config_files[kinds.USER]
 
-        yield kinds.BASE, config_files[kinds.BASE]
-
         # finally virtualenv configuration first trumping others
         yield kinds.SITE, config_files[kinds.SITE]
 


Test patch:
diff --git a/tests/unit/test_configuration.py b/tests/unit/test_configuration.py
--- a/tests/unit/test_configuration.py
+++ b/tests/unit/test_configuration.py
@@ -24,17 +24,11 @@ def test_user_loading(self) -> None:
         self.configuration.load()
         assert self.configuration.get_value("test.hello") == "2"
 
-    def test_base_loading(self) -> None:
-        self.patch_configuration(kinds.BASE, {"test.hello": "3"})
-
-        self.configuration.load()
-        assert self.configuration.get_value("test.hello") == "3"
-
     def test_site_loading(self) -> None:
-        self.patch_configuration(kinds.SITE, {"test.hello": "4"})
+        self.patch_configuration(kinds.SITE, {"test.hello": "3"})
 
         self.configuration.load()
-        assert self.configuration.get_value("test.hello") == "4"
+        assert self.configuration.get_value("test.hello") == "3"
 
     def test_environment_config_loading(self, monkeypatch: pytest.MonkeyPatch) -> None:
         contents = """
@@ -113,15 +107,6 @@ def test_no_such_key_error_message_missing_option(self) -> None:
         with pytest.raises(ConfigurationError, match=pat):
             self.configuration.get_value("global.index-url")
 
-    def test_overrides_normalization(self) -> None:
-        # Check that normalized names are used in precedence calculations.
-        # Reminder: USER has higher precedence than GLOBAL.
-        self.patch_configuration(kinds.USER, {"test.hello-world": "1"})
-        self.patch_configuration(kinds.GLOBAL, {"test.hello_world": "0"})
-        self.configuration.load()
-
-        assert self.configuration.get_value("test.hello_world") == "1"
-
 
 class TestConfigurationPrecedence(ConfigurationMixin):
     # Tests for methods to that determine the order of precedence of
@@ -148,13 +133,6 @@ def test_env_overides_global(self) -> None:
 
         assert self.configuration.get_value("test.hello") == "0"
 
-    def test_site_overides_base(self) -> None:
-        self.patch_configuration(kinds.BASE, {"test.hello": "2"})
-        self.patch_configuration(kinds.SITE, {"test.hello": "1"})
-        self.configuration.load()
-
-        assert self.configuration.get_value("test.hello") == "1"
-
     def test_site_overides_user(self) -> None:
         self.patch_configuration(kinds.USER, {"test.hello": "2"})
         self.patch_configuration(kinds.SITE, {"test.hello": "1"})
@@ -169,13 +147,6 @@ def test_site_overides_global(self) -> None:
 
         assert self.configuration.get_value("test.hello") == "1"
 
-    def test_base_overides_user(self) -> None:
-        self.patch_configuration(kinds.USER, {"test.hello": "2"})
-        self.patch_configuration(kinds.BASE, {"test.hello": "1"})
-        self.configuration.load()
-
-        assert self.configuration.get_value("test.hello") == "1"
-
     def test_user_overides_global(self) -> None:
         self.patch_configuration(kinds.GLOBAL, {"test.hello": "3"})
         self.patch_configuration(kinds.USER, {"test.hello": "2"})
diff --git a/tests/unit/test_options.py b/tests/unit/test_options.py
--- a/tests/unit/test_options.py
+++ b/tests/unit/test_options.py
@@ -587,7 +587,7 @@ def test_venv_config_file_found(self, monkeypatch: pytest.MonkeyPatch) -> None:
         for _, val in cp.iter_config_files():
             files.extend(val)
 
-        assert len(files) == 5
+        assert len(files) == 4
 
     @pytest.mark.parametrize(
         "args, expect",