Repository: pypa/pip

Issue:
pip freeze does case sensitive comparsion
**Environment**

* pip version: 18.0
* Python version: 3.6
* OS: Mac OS X

<!-- Feel free to add more information about your environment here -->

**Description**
<!-- A clear and concise description of what the bug is. -->
Running `pip freeze -l -r requirements.txt` does a case sensitive comparison with dependencies mentioned in `requirements.txt`.

**Expected behavior**
Running `pip freeze -l -r requirements.txt` should do case insensitive comparison with dependencies mentioned in `requirements.txt` or at least allow that via a command line option.

**How to Reproduce**
<!-- Describe the steps to reproduce this bug. -->

1. Create a virtualenv.
2. Create a [requirements.txt](https://gist.github.com/abs51295/4222a40d6c69f80adc7b138f4c8765b3) file.
3. Run `pip install -r requirements.txt`.
4. Run `pip freeze -l -r requirements.txt`.

**Output**

```
#
# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --output-file requirements.txt requirements.in
asn1crypto==0.24.0
attrs==18.1.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
colorama==0.3.9
coverage==4.5.1
cryptography==2.2.2
flake8-polyfill==1.0.2
flake8==3.5.0
Requirement file [requirements.txt] contains flask-cors==3.0.4, but that package is not installed
Requirement file [requirements.txt] contains flask==1.0.2, but that package is not installed
gevent==1.3.1
greenlet==0.4.13
gunicorn==19.8.1
idna==2.6
itsdangerous==0.24
Requirement file [requirements.txt] contains jinja2==2.10, but that package is not installed
mando==0.6.4
Requirement file [requirements.txt] contains markupsafe==1.0, but that package is not installed
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psycopg2==2.7.4
py==1.5.3
pycodestyle==2.3.1
pycparser==2.18
pycrypto==2.6.1
pyflakes==1.6.0
Requirement file [requirements.txt] contains pyjwt==1.6.3, but that package is not installed
pytest-cov==2.5.1
pytest-mock==1.10.0
pytest==3.5.1
radon==2.2.0
requests-futures==0.9.7
requests==2.18.4
semantic-version==2.6.0
six==1.11.0
Requirement file [requirements.txt] contains sqlalchemy==1.2.7, but that package is not installed
urllib3==1.22
Requirement file [requirements.txt] contains werkzeug==0.14.1, but that package is not installed
## The following requirements were added by pip freeze:
Flask==1.0.2
Flask-Cors==3.0.4
Jinja2==2.10
MarkupSafe==1.0
PyJWT==1.6.3
SQLAlchemy==1.2.7
Werkzeug==0.14.1
```



Discussion:
@cjerdonek I will like to work on this issue. Let me know, when we have classified this as a bug or a feature request.
Hey @abs51295!

I'd consider this a bug since this is essentially a case where pip is not normalizing the given package names correctly.
Thanks @pradyunsg I will send a fix soon! 😄 
@pradyunsg how should the fix look like? Should we change the dependency names to lower case explicitly?
There's a normalize_name or a canonical_name (or similarly named) function in pip's codebase, which should be used to normalize names.

(sorry I missed you on IRC - I slept off with the client open) 
Thanks @pradyunsg will look into this.

( I don't see you on #pypa-dev anymore on IRC)
@pradyunsg I found this: https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89. Looks like this requires a discussion.
Hey @abs51295! I was away from a PC for most of the time over the weekend.

I've not looked carefully but you'd want to start looking here: https://github.com/pypa/pip/blob/b13cee6388adea5ddaf1ddb982beeec37d07ffda/src/pip/_internal/commands/freeze.py

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using `pip._vendor.packaging.utils.canonicalize_name`.

> I found this: /src/pip/_vendor/packaging/requirements.py@master#L86-L89.

That's a vendored library (packaging) and while I do think _that_ change needs discussion, the change in pip doesn't and they're fairly separate.
> Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name

@pradyunsg I did that, but why I was referring to the https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89, is because of the following:

1. Canonicalizing the name when pip's getting the information, which can be achieved by canonicalizing the name [here](https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/freeze.py#L162), solves only half the problem.

2. The other half of the problem lies when the requirement line is [parsed](https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/freeze.py#L110-L114) from `requirements.txt` file and [compared with existing installations](https://github.com/pypa/pip/blob/master/src/pip/_internal/operations/freeze.py#L126-L135) as obtained in step 1.

3. During the above comparsion, if we don't canonicalize the name obtained from the `requirements.txt` file, then we will again run into the same problem of case sensitive comparison. To mitigate that, we need to canonicalize the name of this [requirement](https://github.com/pypa/pip/blob/master/src/pip/_internal/req/constructors.py#L253) which leads to [this](https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L99) and hence the comment. 
@pradyunsg any updates on this?
Sorry, I haven't taken a deeper look yet. Real life has kept me more than busy lately.

I'll try to take a look today or this weekend. Feel free to ping me again in that time.
Thanks. Will wait for your reply 🙂 
@pradyunsg any updates?
any updates? hope it badly to [regenerate](https://github.com/MarketLeader/Toko-Chetabahana/releases/tag/v1.1) requirements.txt
all the warned packages were actually been added by pip already

- pip version: 19.1 (python 3.7)
- existing packages: [this requirements.txt](https://github.com/mirumee/saleor/blob/master/requirements.txt)
- new package added: [django-storages[google]](https://django-storages.readthedocs.io/en/latest/backends/gcloud.html)
```
(virtual-env)$ pip --version
pip 19.1 from /env/lib/python3.7/site-packages/pip (python 3.7)

(virtual-env)$ pip install -r requirements.txt
... done
Successfully installed all

(virtual-env)$ pip install django-storages[google]
... done
Successfully installed 
cachetools-3.1.0 google-api-core-1.9.0 google-auth-1.6.3 google-cloud-core-0.29.1 
google-cloud-storage-1.15.0 google-resumable-media-0.3.2 googleapis-common-protos-1.5.9 
protobuf-3.7.1 pyasn1-0.4.5 pyasn1-modules-0.2.5 rsa-4.0

(virtual-env)$ pip freeze -r requirements.txt
-i https://pypi.python.org/simple
amqp==2.4.2
aniso8601==3.0.2
autopep8==1.4.4
WARNING: Requirement file [requirements.txt] contains babel==2.6.0, but package 'babel' is not installed
billiard==3.6.0.0
bleach==2.1.4
boto3==1.9.138
botocore==1.12.138
braintree==3.49.0
cairocffi==1.0.2
WARNING: Requirement file [requirements.txt] contains cairosvg==2.3.1, but package 'cairosvg' is not installed
celery==4.3.0
certifi==2019.3.9
cffi==1.12.3
chardet==3.0.4
cssselect2==0.2.1
defusedxml==0.6.0
dj-database-url==0.5.0
dj-email-url==0.2.0
django-appconf==1.0.3
django-babel==0.6.2
django-bootstrap4==0.0.8
django-cache-url==3.0.0
django-celery-results==1.0.4
django-countries==5.3.3
django-debug-toolbar-request-history==0.0.10
django-debug-toolbar==1.11
django-elasticsearch-dsl==0.5.1
django-filter==2.1.0
django-graphql-jwt==0.2.1
django-impersonate==1.4
django-js-asset==1.2.2
django-measurement==3.0.0
django-mptt==0.10.0
django-phonenumber-field==2.3.1
django-prices-openexchangerates==1.0.1
django-prices-vatlayer==1.0.2
django-prices==1.0.4
django-recaptcha==1.5.0
django-redis==4.10.0
django-render-block==0.5
django-silk==2.0.0
django-storages==1.7.1
django-templated-email==2.3.0
django-versatileimagefield==1.10
django-webpack-loader==0.6.0
WARNING: Requirement file [requirements.txt] contains django==2.2, but package 'django' is not installed
docutils==0.14
elasticsearch-dsl==6.0.1
elasticsearch==6.3.1
WARNING: Requirement file [requirements.txt] contains faker==1.0.5, but package 'faker' is not installed
freezegun==0.3.11
google-i18n-address==2.3.4
google-measurement-protocol==1.0.0
gprof2dot==2016.10.13
graphene-django-optimizer==0.4.0
graphene-django==2.2.0
graphene==2.1.3
graphql-core==2.1
graphql-relay==0.4.5
gunicorn==19.9.0
html5lib==1.0.1
idna==2.8
ipaddress==1.0.22
WARNING: Requirement file [requirements.txt] contains jinja2==2.10.1, but package 'jinja2' is not installed
jmespath==0.9.4
jsonfield==2.0.2
kombu==4.5.0
WARNING: Requirement file [requirements.txt] contains markdown==3.1, but package 'markdown' is not installed
WARNING: Requirement file [requirements.txt] contains markupsafe==1.1.1, but package 'markupsafe' is not installed
maxminddb-geolite2==2018.703
maxminddb==1.4.1
measurement==2.0.1
mpmath==1.1.0
oauthlib==3.0.1
phonenumberslite==8.10.10
WARNING: Requirement file [requirements.txt] contains pillow==5.4.1, but package 'pillow' is not installed
prices==1.0.0
promise==2.2.1
psycopg2-binary==2.8.2
purl==1.5
pycodestyle==2.5.0
pycparser==2.19
WARNING: Requirement file [requirements.txt] contains pygments==2.3.1, but package 'pygments' is not installed
WARNING: Requirement file [requirements.txt] contains pyjwt==1.7.1, but package 'pyjwt' is not installed
WARNING: Requirement file [requirements.txt] contains pyphen==0.9.5, but package 'pyphen' is not installed
python-dateutil==2.8.0
python3-openid==3.1.0
pytz==2019.1
raven==6.9.0
razorpay==1.2.0
redis==3.2.1
requests-oauthlib==1.2.0
requests==2.21.0
WARNING: Requirement file [requirements.txt] contains rx==1.6.1, but package 'rx' is not installed
s3transfer==0.2.0
singledispatch==3.4.0.3
six==1.12.0
social-auth-app-django==3.1.0
social-auth-core==3.1.0
sqlparse==0.3.0
stripe==2.27.0
sympy==1.4
text-unidecode==1.2
tinycss2==1.0.2
typing==3.6.6
urllib3==1.24.2
WARNING: Requirement file [requirements.txt] contains uwsgi==2.0.18 ; platform_system != 'Windows', but package 'uwsgi' is not installed
vine==1.3.0
WARNING: Requirement file [requirements.txt] contains weasyprint==47, but package 'weasyprint' is not installed
webencodings==0.5.1
```
## The following requirements were added by pip freeze:
```
astroid==2.2.5
Babel==2.6.0
cachetools==3.1.0
CairoSVG==2.3.1
configparser==3.7.4
Django==2.2
django-fsm==2.6.0
Faker==1.0.5
google-api-core==1.9.0
google-auth==1.6.3
google-cloud-core==0.29.1
google-cloud-storage==1.15.0
google-resumable-media==0.3.2
googleapis-common-protos==1.5.9
isort==4.3.17
Jinja2==2.10.1
lazy-object-proxy==1.3.1
Markdown==3.1
MarkupSafe==1.1.1
mccabe==0.6.1
Pillow==5.4.1
protobuf==3.7.1
pyasn1==0.4.5
pyasn1-modules==0.2.5
pyflakes==2.1.1
Pygments==2.3.1
PyJWT==1.7.1
pylint==2.3.1
Pyphen==0.9.5
rsa==4.0
Rx==1.6.1
typed-ast==1.3.5
uWSGI==2.0.18
WeasyPrint==47
wrapt==1.11.1
```

Resolving patch:
diff --git a/src/pip/_internal/operations/freeze.py b/src/pip/_internal/operations/freeze.py
--- a/src/pip/_internal/operations/freeze.py
+++ b/src/pip/_internal/operations/freeze.py
@@ -79,7 +79,7 @@ def freeze(
             continue
         if exclude_editable and req.editable:
             continue
-        installations[req.name] = req
+        installations[req.canonical_name] = req
 
     if requirement:
         # the options that don't get turned into an InstallRequirement
@@ -138,22 +138,27 @@ def freeze(
                             "  (add #egg=PackageName to the URL to avoid"
                             " this warning)"
                         )
-                    elif line_req.name not in installations:
-                        # either it's not installed, or it is installed
-                        # but has been processed already
-                        if not req_files[line_req.name]:
-                            logger.warning(
-                                "Requirement file [%s] contains %s, but "
-                                "package %r is not installed",
-                                req_file_path,
-                                COMMENT_RE.sub('', line).strip(), line_req.name
-                            )
+                    else:
+                        line_req_canonical_name = canonicalize_name(
+                            line_req.name)
+                        if line_req_canonical_name not in installations:
+                            # either it's not installed, or it is installed
+                            # but has been processed already
+                            if not req_files[line_req.name]:
+                                logger.warning(
+                                    "Requirement file [%s] contains %s, but "
+                                    "package %r is not installed",
+                                    req_file_path,
+                                    COMMENT_RE.sub('', line).strip(),
+                                    line_req.name
+                                )
+                            else:
+                                req_files[line_req.name].append(req_file_path)
                         else:
+                            yield str(installations[
+                                line_req_canonical_name]).rstrip()
+                            del installations[line_req_canonical_name]
                             req_files[line_req.name].append(req_file_path)
-                    else:
-                        yield str(installations[line_req.name]).rstrip()
-                        del installations[line_req.name]
-                        req_files[line_req.name].append(req_file_path)
 
         # Warn about requirements that were included multiple times (in a
         # single requirements file or in different requirements files).
@@ -168,7 +173,7 @@ def freeze(
         )
     for installation in sorted(
             installations.values(), key=lambda x: x.name.lower()):
-        if canonicalize_name(installation.name) not in skip:
+        if installation.canonical_name not in skip:
             yield str(installation).rstrip()
 
 
@@ -238,6 +243,7 @@ class FrozenRequirement(object):
     def __init__(self, name, req, editable, comments=()):
         # type: (str, Union[str, Requirement], bool, Iterable[str]) -> None
         self.name = name
+        self.canonical_name = canonicalize_name(name)
         self.req = req
         self.editable = editable
         self.comments = comments


Test patch:
diff --git a/tests/functional/test_freeze.py b/tests/functional/test_freeze.py
--- a/tests/functional/test_freeze.py
+++ b/tests/functional/test_freeze.py
@@ -502,15 +502,20 @@ def test_freeze_with_requirement_option(script):
 
     """
 
-    script.scratch_path.joinpath("hint.txt").write_text(textwrap.dedent("""\
+    script.scratch_path.joinpath("hint1.txt").write_text(textwrap.dedent("""\
         INITools==0.1
         NoExist==4.2  # A comment that ensures end of line comments work.
         simple==3.0; python_version > '1.0'
         """) + _freeze_req_opts)
+    script.scratch_path.joinpath("hint2.txt").write_text(textwrap.dedent("""\
+        iniTools==0.1
+        Noexist==4.2  # A comment that ensures end of line comments work.
+        Simple==3.0; python_version > '1.0'
+        """) + _freeze_req_opts)
     result = script.pip_install_local('initools==0.2')
     result = script.pip_install_local('simple')
     result = script.pip(
-        'freeze', '--requirement', 'hint.txt',
+        'freeze', '--requirement', 'hint1.txt',
         expect_stderr=True,
     )
     expected = textwrap.dedent("""\
@@ -521,9 +526,18 @@ def test_freeze_with_requirement_option(script):
     expected += "## The following requirements were added by pip freeze:..."
     _check_output(result.stdout, expected)
     assert (
-        "Requirement file [hint.txt] contains NoExist==4.2, but package "
+        "Requirement file [hint1.txt] contains NoExist==4.2, but package "
         "'NoExist' is not installed"
     ) in result.stderr
+    result = script.pip(
+        'freeze', '--requirement', 'hint2.txt',
+        expect_stderr=True,
+    )
+    _check_output(result.stdout, expected)
+    assert (
+        "Requirement file [hint2.txt] contains Noexist==4.2, but package "
+        "'Noexist' is not installed"
+    ) in result.stderr
 
 
 def test_freeze_with_requirement_option_multiple(script):