Repository: pypa/pip

Issue:
Do not exclude legitimately installed packages from pip freeze
Silently excluding legitimately installed packages like setuptools, distribute and pip is not ok. For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available. Also, @qwcode gave rationale for not excluding setuptools in issue #1606:

> with pip 1.5 able to install from wheel without setuptools, I can see a real case where it's meaningful for setuptools to be listed, i.e. when one of the packages needs setuptools due to an import of pkg_resources, but the install is just using pip (and no setuptools) to install from wheel.

Reasons for having setuptools as a requirement were given in answers to my recent question [Is there any sense to put setuptools as a requirement?](https://mail.python.org/pipermail/distutils-sig/2014-February/023936.html) on distutils-sig mailing list.

There should be an option so that users could decide themselves what packages if any they want to exclude. In any case there should be information about excluded packages presented in the output of every command affected.

By _legitimately installed_ I mean all outside of stdlib as excluding stdlib packages with metadata is another subject – see issue #1570.



Discussion:
Putting pip inside of a requirements.txt file is almost never what you want. For instance, you stated:

> For instance, as requirements file can contain options for pip, someone might want to ensure that the proper version of pip (able to understand these options) would be available.

However the problem is, if you have a requirements file like that, then an incompatible version of pip will bomb out trying to read that requirements.txt file before it even has a chance to upgrade pip. Even if everything goes without a hitch and you successfully upgrade/downgrade pip to a different version via the requirements.txt file it won't actually take any effect until the _next_ invocation of pip (because the current version will have already been imported). If we want to handle the case where you modify the version of pip in a single invocation then we're going to need additional code to handle that.

In the past there was the same sort of problem with setuptools. You couldn't import pip without setuptools installed and you couldn't upgrade setuptools in the same process and have it take effect in the installation of the rest of the packages in that installation run. Now a days this is no longer the case, however I would still argue that it should be excluded by default because rarely does someone actually want to assert a particular version of setuptools (and in fact it can be harmful, especially on platforms like Heroku).

I _do_ not believe we should be putting pip into the output of pip freeze by default, and I feel pretty strongly about that. setuptools/distribute being included by default I still don't think is generally applicable and can be harmful, but I could probably be convinced otherwise. One option is to include setuptools/distribute by default, but have it commented out so that it's obvious that it's there and what version was installed, but that it doesn't affect anything without end user work. Another option is to make the list of what to ignore configurable. If the user hasn't specified any ignores it uses our default list. If they configure any at all they are responsible for configuring them all.

I don't have a strong opinion about setuptools, but I agree with @dstufft that including pip is wrong. And I do not want to see distribute included by default.

People can manually edit requirements files however they like - as far as I can see this is purely about using freeze and _not reviewing or amending its output in any way_. I think I'd need to see a specific, clear use case where the current behaviour causes an error and where manual editing of the requirements file is not practical, before I'd think this was worth it.

> One option is to include setuptools/distribute by default, 
> but have it commented out so that it's obvious that it's there and what version was installed
> Another option is to make the list of what to ignore configurable. 
> If the user hasn't specified any ignores it uses our default list.

I'd be ok with either or both of these.

I think part of the issue is that `pip` and `setuptools` are pretty much "build environment" requirements, not regular "install" requirements; but `requirements.txt`' use case was only ever consciously designed for the latter.

If you separate these two concepts, you come up with a modicum solution:

```
pip install -r build_requirements.txt
pip install -r requirements.txt
```

Where `build_requirements.txt` had something like:

```
setuptools==<version>
pip==<version>
```

As a reference point for how other package managers handle this, NPM explicitly splits out:
- Dependencies: https://docs.npmjs.com/files/package.json#dependencies
- Development dependencies: https://docs.npmjs.com/files/package.json#devdependencies
- A "shrinkwrap" that locks down all dependency and sub-dependency versions: https://docs.npmjs.com/cli/shrinkwrap

I think in general something like `pip freeze` that locks down the full set of dependencies and sub-dependencies is not the right way to handle development dependencies anyway.


Resolving patch:
diff --git a/pip/commands/freeze.py b/pip/commands/freeze.py
--- a/pip/commands/freeze.py
+++ b/pip/commands/freeze.py
@@ -3,11 +3,15 @@
 import sys
 
 import pip
+from pip.compat import stdlib_pkgs
 from pip.basecommand import Command
 from pip.operations.freeze import freeze
 from pip.wheel import WheelCache
 
 
+DEV_PKGS = ('pip', 'setuptools', 'distribute', 'wheel')
+
+
 class FreezeCommand(Command):
     """
     Output installed packages in requirements format.
@@ -52,12 +56,22 @@ def __init__(self, *args, **kw):
             action='store_true',
             default=False,
             help='Only output packages installed in user-site.')
+        self.cmd_opts.add_option(
+            '--all',
+            dest='freeze_all',
+            action='store_true',
+            help='Do not skip these packages in the output:'
+                 ' %s' % ', '.join(DEV_PKGS))
 
         self.parser.insert_option_group(0, self.cmd_opts)
 
     def run(self, options, args):
         format_control = pip.index.FormatControl(set(), set())
         wheel_cache = WheelCache(options.cache_dir, format_control)
+        skip = set(stdlib_pkgs)
+        if not options.freeze_all:
+            skip.update(DEV_PKGS)
+
         freeze_kwargs = dict(
             requirement=options.requirement,
             find_links=options.find_links,
@@ -65,7 +79,8 @@ def run(self, options, args):
             user_only=options.user,
             skip_regex=options.skip_requirements_regex,
             isolated=options.isolated_mode,
-            wheel_cache=wheel_cache)
+            wheel_cache=wheel_cache,
+            skip=skip)
 
         for line in freeze(**freeze_kwargs):
             sys.stdout.write(line + '\n')
diff --git a/pip/compat/__init__.py b/pip/compat/__init__.py
--- a/pip/compat/__init__.py
+++ b/pip/compat/__init__.py
@@ -138,9 +138,9 @@ def expanduser(path):
 # dist.location (py27:`sysconfig.get_paths()['stdlib']`,
 # py26:sysconfig.get_config_vars('LIBDEST')), but fear platform variation may
 # make this ineffective, so hard-coding
-stdlib_pkgs = ['python', 'wsgiref']
+stdlib_pkgs = ('python', 'wsgiref')
 if sys.version_info >= (2, 7):
-    stdlib_pkgs.extend(['argparse'])
+    stdlib_pkgs += ('argparse',)
 
 
 # windows detection, covers cpython and ironpython
diff --git a/pip/operations/freeze.py b/pip/operations/freeze.py
--- a/pip/operations/freeze.py
+++ b/pip/operations/freeze.py
@@ -4,29 +4,26 @@
 import re
 
 import pip
-from pip.compat import stdlib_pkgs
 from pip.req import InstallRequirement
-from pip.utils import get_installed_distributions
+from pip.utils import canonicalize_name, get_installed_distributions
 from pip._vendor import pkg_resources
 
 
 logger = logging.getLogger(__name__)
 
-# packages to exclude from freeze output
-freeze_excludes = stdlib_pkgs + ['setuptools', 'pip', 'distribute', 'wheel']
-
 
 def freeze(
         requirement=None,
         find_links=None, local_only=None, user_only=None, skip_regex=None,
         default_vcs=None,
         isolated=False,
-        wheel_cache=None):
+        wheel_cache=None,
+        skip=()):
     find_links = find_links or []
     skip_match = None
 
     if skip_regex:
-        skip_match = re.compile(skip_regex)
+        skip_match = re.compile(skip_regex).search
 
     dependency_links = []
 
@@ -42,7 +39,7 @@ def freeze(
         yield '-f %s' % link
     installations = {}
     for dist in get_installed_distributions(local_only=local_only,
-                                            skip=freeze_excludes,
+                                            skip=(),
                                             user_only=user_only):
         req = pip.FrozenRequirement.from_dist(
             dist,
@@ -55,7 +52,7 @@ def freeze(
             for line in req_file:
                 if (not line.strip() or
                         line.strip().startswith('#') or
-                        (skip_match and skip_match.search(line)) or
+                        (skip_match and skip_match(line)) or
                         line.startswith((
                             '-r', '--requirement',
                             '-Z', '--always-unzip',
@@ -112,4 +109,5 @@ def freeze(
         )
     for installation in sorted(
             installations.values(), key=lambda x: x.name.lower()):
-        yield str(installation).rstrip()
+        if canonicalize_name(installation.name) not in skip:
+            yield str(installation).rstrip()


Test patch:
diff --git a/tests/functional/test_freeze.py b/tests/functional/test_freeze.py
--- a/tests/functional/test_freeze.py
+++ b/tests/functional/test_freeze.py
@@ -63,6 +63,12 @@ def test_freeze_basic(script):
     _check_output(result.stdout, expected)
 
 
+def test_freeze_with_pip(script):
+    """Test pip shows itself"""
+    result = script.pip('freeze', '--all')
+    assert 'pip==' in result.stdout
+
+
 @pytest.mark.svn
 def test_freeze_svn(script, tmpdir):
     """Test freezing a svn checkout"""
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -16,7 +16,6 @@
 from pip.utils import (egg_link_path, get_installed_distributions,
                        untar_file, unzip_file, rmtree, normalize_path)
 from pip.utils.hashes import Hashes, MissingHashes
-from pip.operations.freeze import freeze_excludes
 from pip._vendor.six import BytesIO
 
 
@@ -261,7 +260,8 @@ def test_freeze_excludes(self, mock_dist_is_editable,
         mock_dist_is_editable.side_effect = self.dist_is_editable
         mock_dist_is_local.side_effect = self.dist_is_local
         mock_dist_in_usersite.side_effect = self.dist_in_usersite
-        dists = get_installed_distributions(skip=freeze_excludes)
+        dists = get_installed_distributions(
+            skip=('setuptools', 'pip', 'distribute'))
         assert len(dists) == 0