Repository: pypa/pip

Issue:
Unable to clean http cache with pip
<!--
If you're reporting an issue for `--use-feature=2020-resolver`, use the "Dependency resolver failures / errors" template instead.
-->

**Environment**

* pip version: pip 20.2.3
* Python version: Python 2.7 (32bit)
* OS: Windows 10 (64bit)

<!-- Feel free to add more information about your environment here -->

**Description**
<!-- A clear and concise description of what the bug is. -->
We noticed pip will cache the http response for future installation. However if the cached http response includes corruped data (network issue for example), installing the same package will result in pip cache error as follow: 

```
(venv2) H:\>pip install pyside
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pyside
  Using cached PySide-1.2.4-cp27-none-win32.whl (41.0 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pyside from https://files.pythonhosted.org/packages/39/78/2b5fcd3b4ff4fc891f1c3a8bee09616d59fa6d644b0dc6252d46d8fbb423/PySide-1.2.4-cp27-none-win32.whl#sha256=104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908:
        Expected sha256 104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908
             Got        9c9bdae0c8b07e8e834b3bd9f35ab93f3ddbae8365047b1596553eb70e996427
```


The symption of this issue has already been reported in https://github.com/pypa/warehouse/issues/8330. 

**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->
If the error is caused by the data corruption in http response cache then pip should invalidate the cache and try to re-download for best attempt. 

Currently `pip cache purge` will not clear http response cache. The only solution to this case is to remove http directory from the file system forcing pip to rebuild http cache. 

**How to Reproduce**
<!-- Describe the steps to reproduce this bug. -->

1. Create a Python 2 virtual environment
2. Download the example corruped file https://mega.nz/file/WsthyLTS#AWD7NmS-w9B62Q3Y8Lb4SvCalqCb1d83a5FniKPmFqY
3. Overwrite http folder in C:\Users\<account name>\AppData\Local\pip\cache\http
4. Uninstall pyside and install pyside by `pip install pyside`


**Output**

```
(venv2) H:\>pip install pyside
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pyside
  Using cached PySide-1.2.4-cp27-none-win32.whl (41.0 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pyside from https://files.pythonhosted.org/packages/39/78/2b5fcd3b4ff4fc891f1c3a8bee09616d59fa6d644b0dc6252d46d8fbb423/PySide-1.2.4-cp27-none-win32.whl#sha256=104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908:
        Expected sha256 104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908
             Got        9c9bdae0c8b07e8e834b3bd9f35ab93f3ddbae8365047b1596553eb70e996427
```



Discussion:
Would it make sense for the `pip cache` commands to include the http directory too?

Looking at the `cache` subcommands (`dir, info, list, purge, remove`):

```console
$ # No change needed:
$ pip cache dir
/Users/hugo/Library/Caches/pip
```
```console
$ # Now:
$ pip cache info
Location: /Users/hugo/Library/Caches/pip/wheels
Size: 67.1 MB
Number of wheels: 139

$ # Next:
$ pip cache info
http location: /Users/hugo/Library/Caches/pip/http
http size: 1202.4 MB
Wheels location: /Users/hugo/Library/Caches/pip/wheels
Wheels size: 67.1 MB
Number of wheels: 139
```
```console
$ # Now (and there's probably no point listing any `http` files?):
$ pip cache list
Cache contents:

 - MacFSEvents-0.8.1-cp38-cp38-macosx_10_9_x86_64.whl (13 kB)
 - Pillow-4.3.0-cp38-cp38-macosx_10_9_x86_64.whl (491 kB)
...
 - ujson-2.0.3-cp38-cp38-macosx_10_9_x86_64.whl (45 kB)
 - wrapt-1.12.1-cp38-cp38-macosx_10_9_x86_64.whl (32 kB)
```
```console
$ # Now:
$ pip cache purge
Files removed: 139
$ # Next:
$ pip cache purge
Files removed: 2245
```
```console
$ # Now (and there's probably no point removing matched `http` files?):
$ pip cache remove "tiny*"
Files removed: 2
```


Resolving patch:
diff --git a/src/pip/_internal/commands/cache.py b/src/pip/_internal/commands/cache.py
--- a/src/pip/_internal/commands/cache.py
+++ b/src/pip/_internal/commands/cache.py
@@ -102,19 +102,30 @@ def get_cache_info(self, options, args):
         if args:
             raise CommandError('Too many arguments')
 
+        num_http_files = len(self._find_http_files(options))
         num_packages = len(self._find_wheels(options, '*'))
 
-        cache_location = self._wheels_cache_dir(options)
-        cache_size = filesystem.format_directory_size(cache_location)
+        http_cache_location = self._cache_dir(options, 'http')
+        wheels_cache_location = self._cache_dir(options, 'wheels')
+        http_cache_size = filesystem.format_directory_size(http_cache_location)
+        wheels_cache_size = filesystem.format_directory_size(
+            wheels_cache_location
+        )
 
         message = textwrap.dedent("""
-            Location: {location}
-            Size: {size}
+            Package index page cache location: {http_cache_location}
+            Package index page cache size: {http_cache_size}
+            Number of HTTP files: {num_http_files}
+            Wheels location: {wheels_cache_location}
+            Wheels size: {wheels_cache_size}
             Number of wheels: {package_count}
         """).format(
-            location=cache_location,
+            http_cache_location=http_cache_location,
+            http_cache_size=http_cache_size,
+            num_http_files=num_http_files,
+            wheels_cache_location=wheels_cache_location,
             package_count=num_packages,
-            size=cache_size,
+            wheels_cache_size=wheels_cache_size,
         ).strip()
 
         logger.info(message)
@@ -169,6 +180,11 @@ def remove_cache_items(self, options, args):
             raise CommandError('Please provide a pattern')
 
         files = self._find_wheels(options, args[0])
+
+        # Only fetch http files if no specific pattern given
+        if args[0] == '*':
+            files += self._find_http_files(options)
+
         if not files:
             raise CommandError('No matching packages')
 
@@ -184,13 +200,18 @@ def purge_cache(self, options, args):
 
         return self.remove_cache_items(options, ['*'])
 
-    def _wheels_cache_dir(self, options):
-        # type: (Values) -> str
-        return os.path.join(options.cache_dir, 'wheels')
+    def _cache_dir(self, options, subdir):
+        # type: (Values, str) -> str
+        return os.path.join(options.cache_dir, subdir)
+
+    def _find_http_files(self, options):
+        # type: (Values) -> List[str]
+        http_dir = self._cache_dir(options, 'http')
+        return filesystem.find_files(http_dir, '*')
 
     def _find_wheels(self, options, pattern):
         # type: (Values, str) -> List[str]
-        wheel_dir = self._wheels_cache_dir(options)
+        wheel_dir = self._cache_dir(options, 'wheels')
 
         # The wheel filename format, as specified in PEP 427, is:
         #     {distribution}-{version}(-{build})?-{python}-{abi}-{platform}.whl


Test patch:
diff --git a/tests/functional/test_cache.py b/tests/functional/test_cache.py
--- a/tests/functional/test_cache.py
+++ b/tests/functional/test_cache.py
@@ -15,11 +15,30 @@ def cache_dir(script):
     return result.stdout.strip()
 
 
+@pytest.fixture
+def http_cache_dir(cache_dir):
+    return os.path.normcase(os.path.join(cache_dir, 'http'))
+
+
 @pytest.fixture
 def wheel_cache_dir(cache_dir):
     return os.path.normcase(os.path.join(cache_dir, 'wheels'))
 
 
+@pytest.fixture
+def http_cache_files(http_cache_dir):
+    destination = os.path.join(http_cache_dir, 'arbitrary', 'pathname')
+
+    if not os.path.exists(destination):
+        return []
+
+    filenames = glob(os.path.join(destination, '*'))
+    files = []
+    for filename in filenames:
+        files.append(os.path.join(destination, filename))
+    return files
+
+
 @pytest.fixture
 def wheel_cache_files(wheel_cache_dir):
     destination = os.path.join(wheel_cache_dir, 'arbitrary', 'pathname')
@@ -34,6 +53,24 @@ def wheel_cache_files(wheel_cache_dir):
     return files
 
 
+@pytest.fixture
+def populate_http_cache(http_cache_dir):
+    destination = os.path.join(http_cache_dir, 'arbitrary', 'pathname')
+    os.makedirs(destination)
+
+    files = [
+        ('aaaaaaaaa', os.path.join(destination, 'aaaaaaaaa')),
+        ('bbbbbbbbb', os.path.join(destination, 'bbbbbbbbb')),
+        ('ccccccccc', os.path.join(destination, 'ccccccccc')),
+    ]
+
+    for _name, filename in files:
+        with open(filename, 'w'):
+            pass
+
+    return files
+
+
 @pytest.fixture
 def populate_wheel_cache(wheel_cache_dir):
     destination = os.path.join(wheel_cache_dir, 'arbitrary', 'pathname')
@@ -83,6 +120,29 @@ def list_matches_wheel_abspath(wheel_name, result):
                and os.path.exists(l), lines))
 
 
+@pytest.fixture
+def remove_matches_http(http_cache_dir):
+    """Returns True if any line in `result`, which should be the output of
+    a `pip cache purge` call, matches `http_filename`.
+
+    E.g., If http_filename is `aaaaaaaaa`, it searches for a line equal to
+    `Removed <http files cache dir>/arbitrary/pathname/aaaaaaaaa`.
+    """
+
+    def _remove_matches_http(http_filename, result):
+        lines = result.stdout.splitlines()
+
+        # The "/arbitrary/pathname/" bit is an implementation detail of how
+        # the `populate_http_cache` fixture is implemented.
+        path = os.path.join(
+            http_cache_dir, 'arbitrary', 'pathname', http_filename,
+        )
+        expected = 'Removed {}'.format(path)
+        return expected in lines
+
+    return _remove_matches_http
+
+
 @pytest.fixture
 def remove_matches_wheel(wheel_cache_dir):
     """Returns True if any line in `result`, which should be the output of
@@ -124,11 +184,17 @@ def test_cache_dir_too_many_args(script, cache_dir):
     assert 'ERROR: Too many arguments' in result.stderr.splitlines()
 
 
-@pytest.mark.usefixtures("populate_wheel_cache")
-def test_cache_info(script, wheel_cache_dir, wheel_cache_files):
+@pytest.mark.usefixtures("populate_http_cache", "populate_wheel_cache")
+def test_cache_info(
+        script, http_cache_dir, wheel_cache_dir, wheel_cache_files
+):
     result = script.pip('cache', 'info')
 
-    assert 'Location: {}'.format(wheel_cache_dir) in result.stdout
+    assert (
+        'Package index page cache location: {}'.format(http_cache_dir)
+        in result.stdout
+    )
+    assert 'Wheels location: {}'.format(wheel_cache_dir) in result.stdout
     num_wheels = len(wheel_cache_files)
     assert 'Number of wheels: {}'.format(num_wheels) in result.stdout
 
@@ -265,21 +331,28 @@ def test_cache_remove_name_and_version_match(script, remove_matches_wheel):
     assert not remove_matches_wheel('zzz-7.8.9', result)
 
 
-@pytest.mark.usefixtures("populate_wheel_cache")
-def test_cache_purge(script, remove_matches_wheel):
-    """Running `pip cache purge` should remove all cached wheels."""
+@pytest.mark.usefixtures("populate_http_cache", "populate_wheel_cache")
+def test_cache_purge(script, remove_matches_http, remove_matches_wheel):
+    """Running `pip cache purge` should remove all cached http files and
+    wheels."""
     result = script.pip('cache', 'purge', '--verbose')
 
+    assert remove_matches_http('aaaaaaaaa', result)
+    assert remove_matches_http('bbbbbbbbb', result)
+    assert remove_matches_http('ccccccccc', result)
+
     assert remove_matches_wheel('yyy-1.2.3', result)
     assert remove_matches_wheel('zzz-4.5.6', result)
     assert remove_matches_wheel('zzz-4.5.7', result)
     assert remove_matches_wheel('zzz-7.8.9', result)
 
 
-@pytest.mark.usefixtures("populate_wheel_cache")
-def test_cache_purge_too_many_args(script, wheel_cache_files):
+@pytest.mark.usefixtures("populate_http_cache", "populate_wheel_cache")
+def test_cache_purge_too_many_args(
+        script, http_cache_files, wheel_cache_files
+):
     """Running `pip cache purge aaa` should raise an error and remove no
-    cached wheels."""
+    cached http files or wheels."""
     result = script.pip('cache', 'purge', 'aaa', '--verbose',
                         expect_error=True)
     assert result.stdout == ''
@@ -289,7 +362,7 @@ def test_cache_purge_too_many_args(script, wheel_cache_files):
     assert 'ERROR: Too many arguments' in result.stderr.splitlines()
 
     # Make sure nothing was deleted.
-    for filename in wheel_cache_files:
+    for filename in http_cache_files + wheel_cache_files:
         assert os.path.exists(filename)