Repository: pypa/pip

Issue:
23.2: `install` log output is much noisier due to metadata file fetching
### Description

In Pip 23.2, there are quite a few new log lines at the default logging level that make the logs more noisy (due to the new `.metadata` fetching behaviour now working correctly). In particular, the full URL of the `.metadata` files are output, which seems like it should perhaps be only output at debug level instead?

### Expected behavior

For the pip output at the standard logging level to be not much more verbose than it was in previous pip versions.

For example, by either:
1. (Preferred) Making the "Obtaining dependency information" line be debug only ([here](https://github.com/pypa/pip/blob/b252ad819bc7b998508a7ed8789b60dceddfd603/src/pip/_internal/operations/prepare.py#L389)).
2. Or at least, for the full URL of the metadata file not to be output on that line, and instead the line be shortened from:
   `Obtaining dependency information for django<5.0,>=4.2 from https://files.pythonhosted.org/packages/d4/83/227ebf197e413f3599cea96dddc7d6b8ff220310cc5b40dd0f1a15e5a9d1/Django-4.2.3-py3-none-any.whl.metadata`
   ...to something like:
   `Obtaining dependency information for django<5.0,>=4.2 from wheel metadata file`

However, given the full URL would be useful for debugging, then option (2) seems suboptimal, and instead option (1) seems like it might offer the best balance?

### pip version

23.2

### Python version

Python 3.11.4

### OS

Linux

### How to Reproduce

1. Create a `requirements.txt` file containing:
   ```
   django>=4.2,<5.0
   gunicorn>=20.0,<21.0
   dj-database-url>=2.0,<3.0
   whitenoise[brotli]>=6.0,<7.0
   ```
2. Using both pip 23.1.2 and pip 23.2, run `pip install -r requirements.txt` and compare the output.

### Output

This was the output in pip 23.1.2:

```
Collecting django<5.0,>=4.2 (from -r requirements.txt (line 1))
  Downloading Django-4.2.3-py3-none-any.whl (8.0 MB)
Collecting gunicorn<21.0,>=20.0 (from -r requirements.txt (line 2))
  Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
Collecting dj-database-url<3.0,>=2.0 (from -r requirements.txt (line 3))
  Downloading dj_database_url-2.0.0-py3-none-any.whl (7.6 kB)
Collecting whitenoise[brotli]<7.0,>=6.0 (from -r requirements.txt (line 4))
  Downloading whitenoise-6.5.0-py3-none-any.whl (19 kB)
Collecting asgiref<4,>=3.6.0 (from django<5.0,>=4.2->-r requirements.txt (line 1))
  Downloading asgiref-3.7.2-py3-none-any.whl (24 kB)
Collecting sqlparse>=0.3.1 (from django<5.0,>=4.2->-r requirements.txt (line 1))
  Downloading sqlparse-0.4.4-py3-none-any.whl (41 kB)
Requirement already satisfied: setuptools>=3.0 in /layers/heroku_python/python/lib/python3.11/site-packages (from gunicorn<21.0,>=20.0->-r requirements.txt (line 2)) (68.0.0)
Collecting typing-extensions>=3.10.0.0 (from dj-database-url<3.0,>=2.0->-r requirements.txt (line 3))
  Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Collecting Brotli (from whitenoise[brotli]<7.0,>=6.0->-r requirements.txt (line 4))
  Downloading Brotli-1.0.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
Installing collected packages: Brotli, whitenoise, typing-extensions, sqlparse, gunicorn, asgiref, django, dj-database-url
```

This is the output in Pip 23.2:

(Note: The codeblock here hides the real impact of the extra log lines, since the code block doesn't wrap the lines, whereas in a terminal they are wrapped due to the long URLs and so take up extra screen estate.)

```
Collecting django<5.0,>=4.2 (from -r requirements.txt (line 1))
  Obtaining dependency information for django<5.0,>=4.2 from https://files.pythonhosted.org/packages/d4/83/227ebf197e413f3599cea96dddc7d6b8ff220310cc5b40dd0f1a15e5a9d1/Django-4.2.3-py3-none-any.whl.metadata
  Downloading Django-4.2.3-py3-none-any.whl.metadata (4.1 kB)
Collecting gunicorn<21.0,>=20.0 (from -r requirements.txt (line 2))
  Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
Collecting dj-database-url<3.0,>=2.0 (from -r requirements.txt (line 3))
  Downloading dj_database_url-2.0.0-py3-none-any.whl (7.6 kB)
Collecting whitenoise[brotli]<7.0,>=6.0 (from -r requirements.txt (line 4))
  Obtaining dependency information for whitenoise[brotli]<7.0,>=6.0 from https://files.pythonhosted.org/packages/3f/d1/ab1d78bcf3e78517f4c57d34c3b349f1289afb5b2dbf46e5bf5c96932be5/whitenoise-6.5.0-py3-none-any.whl.metadata
  Downloading whitenoise-6.5.0-py3-none-any.whl.metadata (3.7 kB)
Collecting asgiref<4,>=3.6.0 (from django<5.0,>=4.2->-r requirements.txt (line 1))
  Obtaining dependency information for asgiref<4,>=3.6.0 from https://files.pythonhosted.org/packages/9b/80/b9051a4a07ad231558fcd8ffc89232711b4e618c15cb7a392a17384bbeef/asgiref-3.7.2-py3-none-any.whl.metadata
  Downloading asgiref-3.7.2-py3-none-any.whl.metadata (9.2 kB)
Collecting sqlparse>=0.3.1 (from django<5.0,>=4.2->-r requirements.txt (line 1))
  Downloading sqlparse-0.4.4-py3-none-any.whl (41 kB)
Requirement already satisfied: setuptools>=3.0 in /layers/heroku_python/python/lib/python3.11/site-packages (from gunicorn<21.0,>=20.0->-r requirements.txt (line 2)) (68.0.0)
Collecting typing-extensions>=3.10.0.0 (from dj-database-url<3.0,>=2.0->-r requirements.txt (line 3))
  Obtaining dependency information for typing-extensions>=3.10.0.0 from https://files.pythonhosted.org/packages/ec/6b/63cc3df74987c36fe26157ee12e09e8f9db4de771e0f3404263117e75b95/typing_extensions-4.7.1-py3-none-any.whl.metadata
  Downloading typing_extensions-4.7.1-py3-none-any.whl.metadata (3.1 kB)
Collecting Brotli (from whitenoise[brotli]<7.0,>=6.0->-r requirements.txt (line 4))
  Downloading Brotli-1.0.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
Downloading Django-4.2.3-py3-none-any.whl (8.0 MB)
Downloading asgiref-3.7.2-py3-none-any.whl (24 kB)
Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Downloading whitenoise-6.5.0-py3-none-any.whl (19 kB)
Installing collected packages: Brotli, whitenoise, typing-extensions, sqlparse, gunicorn, asgiref, django, dj-database-url
```

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
By design the URL is always just the artifact’s URL + `.metadata` so the information is arguably redundant. Removing the metadata URL makes sense to me. A pull request would be welcomed.
Something else I noticed is the discrepancy between the log output between the new `.metadata` file fetching, and the experimental lazy-wheels feature.

ie compare these:

https://github.com/pypa/pip/blob/b252ad819bc7b998508a7ed8789b60dceddfd603/src/pip/_internal/operations/prepare.py#L388-L392

https://github.com/pypa/pip/blob/b252ad819bc7b998508a7ed8789b60dceddfd603/src/pip/_internal/operations/prepare.py#L436-L440

Specifically:
- The `.metadata` log entry includes the full URL, but the lazy-wheel one does not
- The `.metadata` log entry refers to the dependency **range** (eg `"... for django<5.0,>=4.2"`), but the lazy-wheel one references a specific version (eg `"... from django 4.2.3"`) -- the former seems incorrect (since a specific metadata file is being downloaded, there is no such generic metadata file for the range `django<5.0,>=4.2` per se)

Resolving patch:
diff --git a/src/pip/_internal/operations/prepare.py b/src/pip/_internal/operations/prepare.py
--- a/src/pip/_internal/operations/prepare.py
+++ b/src/pip/_internal/operations/prepare.py
@@ -4,7 +4,6 @@
 # The following comment should be removed at some point in the future.
 # mypy: strict-optional=False
 
-import logging
 import mimetypes
 import os
 import shutil
@@ -37,6 +36,7 @@
 from pip._internal.network.session import PipSession
 from pip._internal.operations.build.build_tracker import BuildTracker
 from pip._internal.req.req_install import InstallRequirement
+from pip._internal.utils._log import getLogger
 from pip._internal.utils.direct_url_helpers import (
     direct_url_for_editable,
     direct_url_from_link,
@@ -53,7 +53,7 @@
 from pip._internal.utils.unpacking import unpack_file
 from pip._internal.vcs import vcs
 
-logger = logging.getLogger(__name__)
+logger = getLogger(__name__)
 
 
 def _get_prepared_distribution(
@@ -394,7 +394,7 @@ def _fetch_metadata_using_link_data_attr(
         if metadata_link is None:
             return None
         assert req.req is not None
-        logger.info(
+        logger.verbose(
             "Obtaining dependency information for %s from %s",
             req.req,
             metadata_link,


Test patch: