Repository: pypa/pip

Issue:
Provide a global (system-wide) config file.
I have an internal PyPI implementation, and I'd like to configure pip system-wide to try getting packages from that internal location  before heading out to PyPI. Right now, the only ways I can think to do this involve distributing default config files for every user, or setting environment variables in all of the global shell configuration files. Either method is, at best, messy. A global config file should allow an admin to specify default configuration options for every user on the system, and if the user needs to override these options, they can point pip to their personal config file, or (for more stubborn users) set their own env variables in their personal shell init files.

If there is another recommended way to do this, please let me know! 



Discussion:
I think this is pretty reasonable, and makes a lot of sense from the use case of a systems managed installer. 

For *nix systems it's fairly obvious where the file should go, I'm a little less certain about windows.

If you can define "a global config file" in code, for all supported platforms, I'd be happy to add that to the list of files pip reads during startup.

I'm not sure what this means. All I'm after is getting pip to look for a pip.conf or pip.ini file in a standard location for system-wide defaults before referring to $HOME/.pip/pip.conf or %HOME\pip\pip.ini, which would have the ability to override the global settings, not unlike other tools that employ the same pattern. I'm not talking about a new file format or anything. I'm also not talking about requiring that this global file exist, just like pip doesn't require you to have a personal config file in $HOME. I just want the option of putting a pip.conf file in a known location and having pip use it. 

Given that, it's unclear what is meant by defining a global config file 'in code'. Can you clarify that? 

@bkjones I'm going to try and come up with a patch to address this issue, taking into account @jezdez's concerns. I think I understand what he's asking and can discuss on IRC with him further.

@bkjones I simply don't know what the canonical file system locations are for "global config file". It's easy enough for Linux/Unix. But this won't work for Mac OS and Windows (and its various versions). I'm not saying this isn't a good idea, just that implementing this half-assed is a bad idea.

@jezdez - agreed -- it should be done following best practices where they exist. I can't profess that knowledge either. I can probably help in doing the research to get us there. For Mac OS, is there some reason we can't use /etc/pip/pip.conf for Linux/UNIX and Mac OS? Seems like most things I install treat OS X like Linux in terms of config locations. Only building from source with the proper ./configure flags would change the config file locations.

Still, there's convention, and there's "what they'd have you do", so I'll try to find some authoritative source of information on that for each platform. 

@bkjones On Mac OS the `/etc` dir is strictly for the OS (actually is a symlink to `/private/etc`), AFAIK. Users usually store their personally configuration files in `~/Library/Preferences`. Since there is also a `/Library/Preferences` directory I assume this is supposed to be use for "global preferences". But it seems weird to recommend `~/.pip/pip.conf` for user files, and `/Library` for global ones. So what if we'd put it in `/usr/local/etc/pip.conf` on Mac OS?

Note that on a brand hew OS X installation there is no /usr/local (though I haven't checked this in some time), and apps like AirFoil and other things installed via .pkg can munge up the permissions on /usr/local such that parts of it are owned by root.

So be aware that using `/usr/local` can have support issues on OS X.

`/etc/pip.conf` seems like a good spot for OSX to me.

@kennethreitz You mean `/private/etc/pip.conf`? -1

@adamv Yeah, that's true. But on the other hand it's also a fairly small use case to add a global config in the first place. So having an admin create `/usr/local/etc` doesn't sound like too much to ask for.

`$ man hier` says `/etc/` is for "system configuration files and scripts", which this would be, no?

@kennethreitz Excellent, this is what I needed, thanks! Let's go with `/etc/pip.conf`.

:sparkles: :cake: :sparkles:

I'd personally rather $SYSCONFDIR/pip/pip.conf in case we want to add multiple files. I've been poking at the win32 side of things to see if we can get something standardized easily.

As I said I'm happy to put a patch together, with tests for the current config stuff, which when I took a look didn't have coverage.

I'd agree w/ @pnasrat - all software w/ a file in /etc eventually evolves to the point of moving that file to a /etc/something directory. Just do it up front.

I don't have a Windows box, however something like this seems the Redmond way:

### System-wide

HKEY_LOCAL_MACHINE\SOFTWARE\Python\PyPI\pip 

### Per-user

HKEY_CURRENT_USER\SOFTWARE\Python\PyPI\pip

Possibly useful starting-point:
http://code.activestate.com/recipes/66011-reading-from-and-writing-to-the-windows-registry/

@steakknife I would propose to keep the configuration in a file. It would require additional code in pip to read/write the configuration if it would be stored inside the Windows registry. Is there any other benefits to this, that you see?

I propose to put it in %ALLUSERSPROFILE%\Application Data\pip\pip.ini.

See http://stackoverflow.com/questions/750698/where-to-put-global-application-data-in-vista for some discussion on this path.

@holmboe: Wow, I didn't even remember writing that.  Clearly less effort is always superior, and I could give a trollbait about Microshit either way.

Go for it.

any update? I would rather see pip read /usr/lib/python/distutils/distutils.cfg just like everything else, but any global config file would be nice.

FWIW - I'd love to see a /etc/pip/pip.conf (better than teaching puppet to place a ~/.pip/pip.conf for all users)

+1 for this; this is a blocker for Puppet configurations that need configuration for pip (since pip doesn't follow http_proxy/https_proxy environment variables, for instance). If OSX and Win32 have definitions too vague to agree on for config paths, can the clear-cut Linux OSes be covered at the least? (ie the suggested /etc/pip(.conf|/pip.conf) path).

I'm pretty sure pip works with http_proxy other than for eg subversion, etc http://www.laurii.info/2010/11/using-pip-behind-proxy/

testing http_proxy=http://localhost:3128 pip install ipython with no proxy errors out which I expect.

Also just assuming /etc doesn't work with eg virtualenvs, etc. We'll probably get to something but not before next release

Any update on this / is anything holding things back to get a solution merged ?

Currently PIP_CONFIG_FILE is a possible work around for doing configuration management.

This feature would be very helpful to create base images pre-configured to use your mirror instead of main one in case you have a provider or something like that. 

:+1: 

i would also like to see this tested/merged. 

I was looking for something like this - is it done yet, I can't tell? Otherwise +1 , even though its nearly 3 years old O.o

Maybe I got lost at some point, but I see a comment in this thread from three years ago

> Excellent, this is what I needed, thanks! Let's go with  /etc/pip.conf

But in my rhel6.2 it does not work. For the impatient, search on pip/baseparser.py and on ConfigOptionParser.get_config_files and modify the return output to fit your needs.

For me

```
return ["/etc/pip.ini",default_config_file]
```

works without any problem. I'll maybe prepare a PR for this, although it will probably take longer fork and submit than finding this hack.

@pnasrat Has there been any further review on this, or anything similar cropped up in the meantime?
This is a very useful feature, especially in the days of configuration management, allowing a system-level default prevents users from ever needing to think about anything other than using the `pip` interface.

would really like to see this too.

For now i just made my `/etc/pip.conf` and I globally set `export PIP_CONFIG_FILE=/etc/pip.conf
` in `/etc/profile`. This overrides user based conf files though. Which is fine with my scenario since none of the users have their own `~/.pip/pip.conf` file.

A quick and nice fix for this would be to allow multiple values on this variable (like in PIP_FIND_LINKS for example). Then one could set the variable to something like this `export PIP_CONFIG_FILE="/etc/pip.conf $HOME/.pip/pip.conf"`

@slafs we're working on this in https://github.com/pypa/pip/issues/1733


Resolving patch:
diff --git a/pip/appdirs.py b/pip/appdirs.py
--- a/pip/appdirs.py
+++ b/pip/appdirs.py
@@ -51,6 +51,46 @@ def user_cache_dir(appname):
     return path
 
 
+# for the discussion regarding site_config_dirs locations
+# see <https://github.com/pypa/pip/issues/1733>
+def site_config_dirs(appname):
+    """Return a list of potential user-shared config dirs for this application.
+
+        "appname" is the name of application.
+
+    Typical user config directories are:
+        Mac OS X:   /Library/Application Support/<AppName>/
+        Unix:       /etc or $XDG_CONFIG_DIRS[i]/<AppName>/ for each value in
+                    $XDG_CONFIG_DIRS
+        Win XP:     C:\Documents and Settings\All Users\Application ...
+                    ...Data\<AppName>\
+        Vista:      (Fail! "C:\ProgramData" is a hidden *system* directory
+                    on Vista.)
+        Win 7:      Hidden, but writeable on Win 7:
+                    C:\ProgramData\<AppName>\
+    """
+    if sys.platform == "win32":
+        path = os.path.normpath(_get_win_folder("CSIDL_COMMON_APPDATA"))
+        pathlist = [os.path.join(path, appname)]
+    elif sys.platform == 'darwin':
+        pathlist = [os.path.join('/Library/Application Support', appname)]
+    else:
+        # try looking in $XDG_CONFIG_DIRS
+        xdg_config_dirs = os.getenv('XDG_CONFIG_DIRS', '/etc/xdg')
+        if xdg_config_dirs:
+            pathlist = [
+                os.sep.join([os.path.expanduser(x), appname])
+                for x in xdg_config_dirs.split(os.pathsep)
+            ]
+        else:
+            pathlist = []
+
+        # always look in /etc directly as well
+        pathlist.append('/etc')
+
+    return pathlist
+
+
 # -- Windows support functions --
 
 def _get_win_folder_from_registry(csidl_name):
diff --git a/pip/baseparser.py b/pip/baseparser.py
--- a/pip/baseparser.py
+++ b/pip/baseparser.py
@@ -10,6 +10,7 @@
 from pip._vendor.six.moves import configparser
 from pip.locations import (
     default_config_file, default_config_basename, running_under_virtualenv,
+    site_config_files
 )
 from pip.util import get_terminal_size
 
@@ -138,13 +139,23 @@ def __init__(self, *args, **kwargs):
         optparse.OptionParser.__init__(self, *args, **kwargs)
 
     def get_config_files(self):
+        # the files returned by this method will be parsed in order with the
+        # first files listed being overridden by later files in standard
+        # ConfigParser fashion
         config_file = os.environ.get('PIP_CONFIG_FILE', False)
         if config_file == os.devnull:
             return []
+
+        # at the base we have any site-wide configuration
+        files = list(site_config_files)
+
+        # per-user configuration next
         if config_file and os.path.exists(config_file):
-            files = [config_file]
+            files.append(config_file)
         else:
-            files = [default_config_file]
+            files.append(default_config_file)
+
+        # finally virtualenv configuration first trumping others
         if running_under_virtualenv():
             venv_config_file = os.path.join(
                 sys.prefix,
@@ -152,6 +163,7 @@ def get_config_files(self):
             )
             if os.path.exists(venv_config_file):
                 files.append(venv_config_file)
+
         return files
 
     def check_default(self, option, key, val):
diff --git a/pip/locations.py b/pip/locations.py
--- a/pip/locations.py
+++ b/pip/locations.py
@@ -202,6 +202,11 @@ def _get_build_prefix():
         bin_py = '/usr/local/bin'
         default_log_file = os.path.join(user_dir, 'Library/Logs/pip.log')
 
+site_config_files = [
+    os.path.join(path, default_config_basename)
+    for path in appdirs.site_config_dirs('pip')
+]
+
 
 def distutils_scheme(dist_name, user=False, home=None, root=None):
     """


Test patch:
diff --git a/tests/unit/test_appdirs.py b/tests/unit/test_appdirs.py
--- a/tests/unit/test_appdirs.py
+++ b/tests/unit/test_appdirs.py
@@ -43,3 +43,50 @@ def test_user_cache_dir_linux_override(self, monkeypatch):
         monkeypatch.setattr(sys, "platform", "linux2")
 
         assert appdirs.user_cache_dir("pip") == "/home/test/.other-cache/pip"
+
+    def test_site_config_dirs_win(self, monkeypatch):
+        @pretend.call_recorder
+        def _get_win_folder(base):
+            return "C:\\ProgramData"
+
+        monkeypatch.setattr(
+            appdirs,
+            "_get_win_folder",
+            _get_win_folder,
+            raising=False,
+        )
+        monkeypatch.setattr(sys, "platform", "win32")
+
+        result = [
+            e.replace("/", "\\")
+            for e in appdirs.site_config_dirs("pip")
+        ]
+        assert result == ["C:\\ProgramData\\pip"]
+        assert _get_win_folder.calls == [pretend.call("CSIDL_COMMON_APPDATA")]
+
+    def test_site_config_dirs_osx(self, monkeypatch):
+        monkeypatch.setenv("HOME", "/home/test")
+        monkeypatch.setattr(sys, "platform", "darwin")
+
+        assert appdirs.site_config_dirs("pip") == \
+            ["/Library/Application Support/pip"]
+
+    def test_site_config_dirs_linux(self, monkeypatch):
+        monkeypatch.delenv("XDG_CONFIG_DIRS")
+        monkeypatch.setattr(sys, "platform", "linux2")
+
+        assert appdirs.site_config_dirs("pip") == [
+            '/etc/xdg/pip',
+            '/etc'
+        ]
+
+    def test_site_config_dirs_linux_override(self, monkeypatch):
+        monkeypatch.setenv("XDG_CONFIG_DIRS", "/spam:/etc:/etc/xdg")
+        monkeypatch.setattr(sys, "platform", "linux2")
+
+        assert appdirs.site_config_dirs("pip") == [
+            '/spam/pip',
+            '/etc/pip',
+            '/etc/xdg/pip',
+            '/etc'
+        ]
diff --git a/tests/unit/test_options.py b/tests/unit/test_options.py
--- a/tests/unit/test_options.py
+++ b/tests/unit/test_options.py
@@ -275,6 +275,9 @@ def test_venv_config_file_found(self, monkeypatch):
             lambda self: None,
         )
 
+        # strict limit on the site_config_files list
+        monkeypatch.setattr(pip.baseparser, 'site_config_files', ['/a/place'])
+
         # If we are running in a virtualenv and all files appear to exist,
         # we should see two config files.
         monkeypatch.setattr(
@@ -284,4 +287,4 @@ def test_venv_config_file_found(self, monkeypatch):
         )
         monkeypatch.setattr(os.path, 'exists', lambda filename: True)
         cp = pip.baseparser.ConfigOptionParser()
-        assert len(cp.get_config_files()) == 2
+        assert len(cp.get_config_files()) == 3