Repository: pypa/pip (python)

Issue:
[Improvement] Pip could resume download package at halfway the connection is poor
* Pip version: 9.0.1
* Python version: 3.6.2
* Operating system: macOS 10.13

### Description

When I have poor internet connection (the network is cut unexpectedly), updating pip package is painful. When I retry the pip install, it would stop at the midpoint and give me the same md5 error.

All I have to do is

1. Download the package from pypi (using browser or wget, both have retry/resume capability)
2. Pip install
3. remove the package

If pip download have resume feature then the problem could be solved.

### What I've run

`pip install -U jupyterlab` in poor network condition

```
Collecting jupyterlab
  Downloading jupyterlab-0.28.4-py2.py3-none-any.whl (8.7MB)
    4% |█▋                              | 430kB 1.1MB/s eta 0:00:08
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    jupyterlab from https://pypi.python.org/packages/b1/6d/d1d033186a07e08af9dc09db41401af7d6e18f98b73bd3bef75a1139dd1b/jupyterlab-0.28.4-py2.py3-none-any.whl#md5=9a93b1dc85f5924151f0ae9670024bd0:
        Expected md5 9a93b1dc85f5924151f0ae9670024bd0
             Got        4b6835257af9609a227a72b18ea011e3
```


Pull request:
Introduce resumable downloads with --resume-retries
Resolves #4796

Introduced the `--resume-retries` option in order to allow resuming incomplete downloads incase of dropped or timed out connections.

This option additionally uses the values specified for `--retries` and `--timeout` for each resume attempt, since they are passed in the session.

Used `0` as the default in order to keep backward compatibility.

This PR is based on https://github.com/pypa/pip/pull/11180

> The downloader will make new requests and attempt to resume downloading using a `Range` header. If the initial response includes an `ETag` (preferred) or `Date` header, the downloader will ask the server to resume downloading only when it is safe (i.e., the file hasn't changed since the initial request) using an `If-Range` header.
>
> If the server responds with a `200` (e.g. if the server doesn't support partial content or can't check if the file has changed), the downloader will restart the download (i.e. start from the very first byte); if the server responds with a `206 Partial Content`, the downloader will resume the download from the partially downloaded file.


Patch:
diff --git a/docs/html/cli/index.md b/docs/html/cli/index.md
index 4b56dbeb4..902fe406e 100644
--- a/docs/html/cli/index.md
+++ b/docs/html/cli/index.md
@@ -37,6 +37,7 @@ pip_hash
 :caption: Package Index information
 
 pip_search
+pip_index
 ```
 
 ```{toctree}
diff --git a/docs/html/cli/pip_index.rst b/docs/html/cli/pip_index.rst
new file mode 100644
index 000000000..89e56195c
--- /dev/null
+++ b/docs/html/cli/pip_index.rst
@@ -0,0 +1,55 @@
+.. _`pip index`:
+
+===========
+pip index
+===========
+
+
+
+Usage
+=====
+
+.. tab:: Unix/macOS
+
+   .. pip-command-usage:: index "python -m pip"
+
+.. tab:: Windows
+
+   .. pip-command-usage:: index "py -m pip"
+
+
+Description
+===========
+
+.. pip-command-description:: index
+
+
+
+Options
+=======
+
+.. pip-command-options:: index
+
+
+
+Examples
+========
+
+#. Search for "peppercorn" versions
+
+   .. tab:: Unix/macOS
+
+      .. code-block:: console
+
+         $ python -m pip index versions peppercorn
+         peppercorn (0.6)
+         Available versions: 0.6, 0.5, 0.4, 0.3, 0.2, 0.1
+
+
+   .. tab:: Windows
+
+      .. code-block:: console
+
+         C:\> py -m pip index peppercorn
+         peppercorn (0.6)
+         Available versions: 0.6, 0.5, 0.4, 0.3, 0.2, 0.1
diff --git a/docs/html/reference/pip_index.rst b/docs/html/reference/pip_index.rst
new file mode 100644
index 000000000..b80c1ada9
--- /dev/null
+++ b/docs/html/reference/pip_index.rst
@@ -0,0 +1,11 @@
+:orphan:
+
+.. meta::
+
+  :http-equiv=refresh: 3; url=../../cli/pip_index/
+
+This page has moved
+===================
+
+You should be redirected automatically in 3 seconds. If that didn't
+work, here's a link: :doc:`../cli/pip_index`
diff --git a/docs/man/commands/index.rst b/docs/man/commands/index.rst
new file mode 100644
index 000000000..fe00870b9
--- /dev/null
+++ b/docs/man/commands/index.rst
@@ -0,0 +1,20 @@
+:orphan:
+
+===========
+pip-index
+===========
+
+Description
+***********
+
+.. pip-command-description:: index
+
+Usage
+*****
+
+.. pip-command-usage:: index
+
+Options
+*******
+
+.. pip-command-options:: index
diff --git a/news/12991.feature.rst b/news/12991.feature.rst
new file mode 100644
index 000000000..26c2df389
--- /dev/null
+++ b/news/12991.feature.rst
@@ -0,0 +1,3 @@
+Add support to enable resuming incomplete downloads.
+
+Control the number of retry attempts using the ``--resume-retries`` flag.
diff --git a/news/13188.feature.rst b/news/13188.feature.rst
new file mode 100644
index 000000000..d8bd807fb
--- /dev/null
+++ b/news/13188.feature.rst
@@ -0,0 +1,1 @@
+Remove ``experimental`` warning from ``pip index versions`` command.
diff --git a/news/13330.bugfix.rst b/news/13330.bugfix.rst
new file mode 100644
index 000000000..ab244fc40
--- /dev/null
+++ b/news/13330.bugfix.rst
@@ -0,0 +1,1 @@
+Don't require the ``wheel`` library to be installed to use ``--no-use-pep517``, any more.
diff --git a/src/pip/_internal/cli/cmdoptions.py b/src/pip/_internal/cli/cmdoptions.py
index 89be4a7b4..88392836c 100644
--- a/src/pip/_internal/cli/cmdoptions.py
+++ b/src/pip/_internal/cli/cmdoptions.py
@@ -281,8 +281,17 @@ retries: Callable[..., Option] = partial(
     dest="retries",
     type="int",
     default=5,
-    help="Maximum number of retries each connection should attempt "
-    "(default %default times).",
+    help="Maximum attempts to establish a new HTTP connection. (default: %default)",
+)
+
+resume_retries: Callable[..., Option] = partial(
+    Option,
+    "--resume-retries",
+    dest="resume_retries",
+    type="int",
+    default=0,
+    help="Maximum attempts to resume or restart an incomplete download. "
+    "(default: %default)",
 )
 
 timeout: Callable[..., Option] = partial(
@@ -824,9 +833,9 @@ def _handle_no_use_pep517(
         """
         raise_option_error(parser, option=option, msg=msg)
 
-    # If user doesn't wish to use pep517, we check if setuptools and wheel are installed
+    # If user doesn't wish to use pep517, we check if setuptools is installed
     # and raise error if it is not.
-    packages = ("setuptools", "wheel")
+    packages = ("setuptools",)
     if not all(importlib.util.find_spec(package) for package in packages):
         msg = (
             f"It is not possible to use --no-use-pep517 "
@@ -1077,7 +1086,6 @@ use_deprecated_feature: Callable[..., Option] = partial(
     help=("Enable deprecated functionality, that will be removed in the future."),
 )
 
-
 ##########
 # groups #
 ##########
@@ -1110,6 +1118,7 @@ general_group: Dict[str, Any] = {
         no_python_version_warning,
         use_new_feature,
         use_deprecated_feature,
+        resume_retries,
     ],
 }
 
diff --git a/src/pip/_internal/cli/progress_bars.py b/src/pip/_internal/cli/progress_bars.py
index a7d77cfaa..ab9d76b25 100644
--- a/src/pip/_internal/cli/progress_bars.py
+++ b/src/pip/_internal/cli/progress_bars.py
@@ -29,6 +29,7 @@ def _rich_download_progress_bar(
     *,
     bar_type: str,
     size: Optional[int],
+    initial_progress: Optional[int] = None,
 ) -> Generator[bytes, None, None]:
     assert bar_type == "on", "This should only be used in the default mode."
 
@@ -54,6 +55,8 @@ def _rich_download_progress_bar(
 
     progress = Progress(*columns, refresh_per_second=5)
     task_id = progress.add_task(" " * (get_indentation() + 2), total=total)
+    if initial_progress is not None:
+        progress.update(task_id, advance=initial_progress)
     with progress:
         for chunk in iterable:
             yield chunk
@@ -86,12 +89,13 @@ def _raw_progress_bar(
     iterable: Iterable[bytes],
     *,
     size: Optional[int],
+    initial_progress: Optional[int] = None,
 ) -> Generator[bytes, None, None]:
     def write_progress(current: int, total: int) -> None:
         sys.stdout.write(f"Progress {current} of {total}\n")
         sys.stdout.flush()
 
-    current = 0
+    current = initial_progress or 0
     total = size or 0
     rate_limiter = RateLimiter(0.25)
 
@@ -105,7 +109,7 @@ def _raw_progress_bar(
 
 
 def get_download_progress_renderer(
-    *, bar_type: str, size: Optional[int] = None
+    *, bar_type: str, size: Optional[int] = None, initial_progress: Optional[int] = None
 ) -> ProgressRenderer[bytes]:
     """Get an object that can be used to render the download progress.
 
@@ -113,10 +117,17 @@ def get_download_progress_renderer(
     """
     if bar_type == "on":
         return functools.partial(
-            _rich_download_progress_bar, bar_type=bar_type, size=size
+            _rich_download_progress_bar,
+            bar_type=bar_type,
+            size=size,
+            initial_progress=initial_progress,
         )
     elif bar_type == "raw":
-        return functools.partial(_raw_progress_bar, size=size)
+        return functools.partial(
+            _raw_progress_bar,
+            size=size,
+            initial_progress=initial_progress,
+        )
     else:
         return iter  # no-op, when passed an iterator
 
diff --git a/src/pip/_internal/cli/req_command.py b/src/pip/_internal/cli/req_command.py
index 82164e8a7..d9b514270 100644
--- a/src/pip/_internal/cli/req_command.py
+++ b/src/pip/_internal/cli/req_command.py
@@ -144,6 +144,7 @@ class RequirementCommand(IndexGroupCommand):
             lazy_wheel=lazy_wheel,
             verbosity=verbosity,
             legacy_resolver=legacy_resolver,
+            resume_retries=options.resume_retries,
         )
 
     @classmethod
diff --git a/src/pip/_internal/commands/index.py b/src/pip/_internal/commands/index.py
index 14f0feb61..e8714a727 100644
--- a/src/pip/_internal/commands/index.py
+++ b/src/pip/_internal/commands/index.py
@@ -55,12 +55,6 @@ class IndexCommand(IndexGroupCommand):
             "versions": self.get_available_package_versions,
         }
 
-        logger.warning(
-            "pip index is currently an experimental command. "
-            "It may be removed/changed in a future release "
-            "without prior warning."
-        )
-
         # Determine action
         if not args or args[0] not in handlers:
             logger.error(
diff --git a/src/pip/_internal/exceptions.py b/src/pip/_internal/exceptions.py
index 313bbf51b..4fe4aadef 100644
--- a/src/pip/_internal/exceptions.py
+++ b/src/pip/_internal/exceptions.py
@@ -27,6 +27,7 @@ if TYPE_CHECKING:
     from pip._vendor.requests.models import Request, Response
 
     from pip._internal.metadata import BaseDistribution
+    from pip._internal.models.link import Link
     from pip._internal.req.req_install import InstallRequirement
 
 logger = logging.getLogger(__name__)
@@ -809,6 +810,38 @@ class InvalidInstalledPackage(DiagnosticPipError):
         )
 
 
+class IncompleteDownloadError(DiagnosticPipError):
+    """Raised when the downloader receives fewer bytes than advertised
+    in the Content-Length header."""
+
+    reference = "incomplete-download"
+
+    def __init__(
+        self, link: "Link", received: int, expected: int, *, retries: int
+    ) -> None:
+        # Dodge circular import.
+        from pip._internal.utils.misc import format_size
+
+        download_status = f"{format_size(received)}/{format_size(expected)}"
+        if retries:
+            retry_status = f"after {retries} attempts "
+            hint = "Use --resume-retries to configure resume attempt limit."
+        else:
+            retry_status = ""
+            hint = "Consider using --resume-retries to enable download resumption."
+        message = Text(
+            f"Download failed {retry_status}because not enough bytes "
+            f"were received ({download_status})"
+        )
+
+        super().__init__(
+            message=message,
+            context=f"URL: {link.redacted_url}",
+            hint_stmt=hint,
+            note_stmt="This is an issue with network connectivity, not pip.",
+        )
+
+
 class ResolutionTooDeepError(DiagnosticPipError):
     """Raised when the dependency resolver exceeds the maximum recursion depth."""
 
diff --git a/src/pip/_internal/models/link.py b/src/pip/_internal/models/link.py
index 0c7a1e9a6..f0560f6ec 100644
--- a/src/pip/_internal/models/link.py
+++ b/src/pip/_internal/models/link.py
@@ -380,9 +380,9 @@ class Link:
         else:
             rp = ""
         if self.comes_from:
-            return f"{redact_auth_from_url(self._url)} (from {self.comes_from}){rp}"
+            return f"{self.redacted_url} (from {self.comes_from}){rp}"
         else:
-            return redact_auth_from_url(str(self._url))
+            return self.redacted_url
 
     def __repr__(self) -> str:
         return f"<Link {self}>"
@@ -404,6 +404,10 @@ class Link:
     def url(self) -> str:
         return self._url
 
+    @property
+    def redacted_url(self) -> str:
+        return redact_auth_from_url(self.url)
+
     @property
     def filename(self) -> str:
         path = self.path.rstrip("/")
diff --git a/src/pip/_internal/network/download.py b/src/pip/_internal/network/download.py
index 9c6d70165..15ef58b9c 100644
--- a/src/pip/_internal/network/download.py
+++ b/src/pip/_internal/network/download.py
@@ -4,12 +4,14 @@ import email.message
 import logging
 import mimetypes
 import os
-from typing import Iterable, Optional, Tuple
+from http import HTTPStatus
+from typing import BinaryIO, Iterable, Optional, Tuple
 
 from pip._vendor.requests.models import Response
+from pip._vendor.urllib3.exceptions import ReadTimeoutError
 
 from pip._internal.cli.progress_bars import get_download_progress_renderer
-from pip._internal.exceptions import NetworkConnectionError
+from pip._internal.exceptions import IncompleteDownloadError, NetworkConnectionError
 from pip._internal.models.index import PyPI
 from pip._internal.models.link import Link
 from pip._internal.network.cache import is_from_cache
@@ -27,13 +29,21 @@ def _get_http_response_size(resp: Response) -> Optional[int]:
         return None
 
 
+def _get_http_response_etag_or_last_modified(resp: Response) -> Optional[str]:
+    """
+    Return either the ETag or Last-Modified header (or None if neither exists).
+    The return value can be used in an If-Range header.
+    """
+    return resp.headers.get("etag", resp.headers.get("last-modified"))
+
+
 def _prepare_download(
     resp: Response,
     link: Link,
     progress_bar: str,
+    total_length: Optional[int],
+    range_start: Optional[int] = 0,
 ) -> Iterable[bytes]:
-    total_length = _get_http_response_size(resp)
-
     if link.netloc == PyPI.file_storage_domain:
         url = link.show_url
     else:
@@ -42,10 +52,17 @@ def _prepare_download(
     logged_url = redact_auth_from_url(url)
 
     if total_length:
-        logged_url = f"{logged_url} ({format_size(total_length)})"
+        if range_start:
+            logged_url = (
+                f"{logged_url} ({format_size(range_start)}/{format_size(total_length)})"
+            )
+        else:
+            logged_url = f"{logged_url} ({format_size(total_length)})"
 
     if is_from_cache(resp):
         logger.info("Using cached %s", logged_url)
+    elif range_start:
+        logger.info("Resuming download %s", logged_url)
     else:
         logger.info("Downloading %s", logged_url)
 
@@ -65,7 +82,9 @@ def _prepare_download(
     if not show_progress:
         return chunks
 
-    renderer = get_download_progress_renderer(bar_type=progress_bar, size=total_length)
+    renderer = get_download_progress_renderer(
+        bar_type=progress_bar, size=total_length, initial_progress=range_start
+    )
     return renderer(chunks)
 
 
@@ -112,10 +131,27 @@ def _get_http_response_filename(resp: Response, link: Link) -> str:
     return filename
 
 
-def _http_get_download(session: PipSession, link: Link) -> Response:
+def _http_get_download(
+    session: PipSession,
+    link: Link,
+    range_start: Optional[int] = 0,
+    if_range: Optional[str] = None,
+) -> Response:
     target_url = link.url.split("#", 1)[0]
-    resp = session.get(target_url, headers=HEADERS, stream=True)
-    raise_for_status(resp)
+    headers = HEADERS.copy()
+    # request a partial download
+    if range_start:
+        headers["Range"] = f"bytes={range_start}-"
+    # make sure the file hasn't changed
+    if if_range:
+        headers["If-Range"] = if_range
+    try:
+        resp = session.get(target_url, headers=headers, stream=True)
+        raise_for_status(resp)
+    except NetworkConnectionError as e:
+        assert e.response is not None
+        logger.critical("HTTP error %s while getting %s", e.response.status_code, link)
+        raise
     return resp
 
 
@@ -124,30 +160,140 @@ class Downloader:
         self,
         session: PipSession,
         progress_bar: str,
+        resume_retries: int,
     ) -> None:
+        assert (
+            resume_retries >= 0
+        ), "Number of max resume retries must be bigger or equal to zero"
         self._session = session
         self._progress_bar = progress_bar
+        self._resume_retries = resume_retries
 
     def __call__(self, link: Link, location: str) -> Tuple[str, str]:
         """Download the file given by link into location."""
-        try:
-            resp = _http_get_download(self._session, link)
-        except NetworkConnectionError as e:
-            assert e.response is not None
-            logger.critical(
-                "HTTP error %s while getting %s", e.response.status_code, link
-            )
-            raise
+        resp = _http_get_download(self._session, link)
+        # NOTE: The original download size needs to be passed down everywhere
+        # so if the download is resumed (with a HTTP Range request) the progress
+        # bar will report the right size.
+        total_length = _get_http_response_size(resp)
+        content_type = resp.headers.get("Content-Type", "")
 
         filename = _get_http_response_filename(resp, link)
         filepath = os.path.join(location, filename)
 
-        chunks = _prepare_download(resp, link, self._progress_bar)
         with open(filepath, "wb") as content_file:
+            bytes_received = self._process_response(
+                resp, link, content_file, 0, total_length
+            )
+            # If possible, check for an incomplete download and attempt resuming.
+            if total_length and bytes_received < total_length:
+                self._attempt_resume(
+                    resp, link, content_file, total_length, bytes_received
+                )
+
+        return filepath, content_type
+
+    def _process_response(
+        self,
+        resp: Response,
+        link: Link,
+        content_file: BinaryIO,
+        bytes_received: int,
+        total_length: Optional[int],
+    ) -> int:
+        """Process the response and write the chunks to the file."""
+        chunks = _prepare_download(
+            resp, link, self._progress_bar, total_length, range_start=bytes_received
+        )
+        return self._write_chunks_to_file(
+            chunks, content_file, allow_partial=bool(total_length)
+        )
+
+    def _write_chunks_to_file(
+        self, chunks: Iterable[bytes], content_file: BinaryIO, *, allow_partial: bool
+    ) -> int:
+        """Write the chunks to the file and return the number of bytes received."""
+        bytes_received = 0
+        try:
             for chunk in chunks:
+                bytes_received += len(chunk)
                 content_file.write(chunk)
-        content_type = resp.headers.get("Content-Type", "")
-        return filepath, content_type
+        except ReadTimeoutError as e:
+            # If partial downloads are OK (the download will be retried), don't bail.
+            if not allow_partial:
+                raise e
+
+            # Ensuring bytes_received is returned to attempt resume
+            logger.warning("Connection timed out while downloading.")
+
+        return bytes_received
+
+    def _attempt_resume(
+        self,
+        resp: Response,
+        link: Link,
+        content_file: BinaryIO,
+        total_length: Optional[int],
+        bytes_received: int,
+    ) -> None:
+        """Attempt to resume the download if connection was dropped."""
+        etag_or_last_modified = _get_http_response_etag_or_last_modified(resp)
+
+        attempts_left = self._resume_retries
+        while total_length and attempts_left and bytes_received < total_length:
+            attempts_left -= 1
+
+            logger.warning(
+                "Attempting to resume incomplete download (%s/%s, attempt %d)",
+                format_size(bytes_received),
+                format_size(total_length),
+                (self._resume_retries - attempts_left),
+            )
+
+            try:
+                # Try to resume the download using a HTTP range request.
+                resume_resp = _http_get_download(
+                    self._session,
+                    link,
+                    range_start=bytes_received,
+                    if_range=etag_or_last_modified,
+                )
+
+                # Fallback: if the server responded with 200 (i.e., the file has
+                # since been modified or range requests are unsupported) or any
+                # other unexpected status, restart the download from the beginning.
+                must_restart = resume_resp.status_code != HTTPStatus.PARTIAL_CONTENT
+                if must_restart:
+                    bytes_received, total_length, etag_or_last_modified = (
+                        self._reset_download_state(resume_resp, content_file)
+                    )
+
+                bytes_received += self._process_response(
+                    resume_resp, link, content_file, bytes_received, total_length
+                )
+            except (ConnectionError, ReadTimeoutError, OSError):
+                continue
+
+        # No more resume attempts. Raise an error if the download is still incomplete.
+        if total_length and bytes_received < total_length:
+            os.remove(content_file.name)
+            raise IncompleteDownloadError(
+                link, bytes_received, total_length, retries=self._resume_retries
+            )
+
+    def _reset_download_state(
+        self,
+        resp: Response,
+        content_file: BinaryIO,
+    ) -> Tuple[int, Optional[int], Optional[str]]:
+        """Reset the download state to restart downloading from the beginning."""
+        content_file.seek(0)
+        content_file.truncate()
+        bytes_received = 0
+        total_length = _get_http_response_size(resp)
+        etag_or_last_modified = _get_http_response_etag_or_last_modified(resp)
+
+        return bytes_received, total_length, etag_or_last_modified
 
 
 class BatchDownloader:
@@ -155,32 +301,14 @@ class BatchDownloader:
         self,
         session: PipSession,
         progress_bar: str,
+        resume_retries: int,
     ) -> None:
-        self._session = session
-        self._progress_bar = progress_bar
+        self._downloader = Downloader(session, progress_bar, resume_retries)
 
     def __call__(
         self, links: Iterable[Link], location: str
     ) -> Iterable[Tuple[Link, Tuple[str, str]]]:
         """Download the files given by links into location."""
         for link in links:
-            try:
-                resp = _http_get_download(self._session, link)
-            except NetworkConnectionError as e:
-                assert e.response is not None
-                logger.critical(
-                    "HTTP error %s while getting %s",
-                    e.response.status_code,
-                    link,
-                )
-                raise
-
-            filename = _get_http_response_filename(resp, link)
-            filepath = os.path.join(location, filename)
-
-            chunks = _prepare_download(resp, link, self._progress_bar)
-            with open(filepath, "wb") as content_file:
-                for chunk in chunks:
-                    content_file.write(chunk)
-            content_type = resp.headers.get("Content-Type", "")
+            filepath, content_type = self._downloader(link, location)
             yield link, (filepath, content_type)
diff --git a/src/pip/_internal/operations/prepare.py b/src/pip/_internal/operations/prepare.py
index b7beee55d..531070a03 100644
--- a/src/pip/_internal/operations/prepare.py
+++ b/src/pip/_internal/operations/prepare.py
@@ -235,6 +235,7 @@ class RequirementPreparer:
         lazy_wheel: bool,
         verbosity: int,
         legacy_resolver: bool,
+        resume_retries: int,
     ) -> None:
         super().__init__()
 
@@ -242,8 +243,8 @@ class RequirementPreparer:
         self.build_dir = build_dir
         self.build_tracker = build_tracker
         self._session = session
-        self._download = Downloader(session, progress_bar)
-        self._batch_download = BatchDownloader(session, progress_bar)
+        self._download = Downloader(session, progress_bar, resume_retries)
+        self._batch_download = BatchDownloader(session, progress_bar, resume_retries)
         self.finder = finder
 
         # Where still-packed archives should be written to. If None, they are


Test patch:
diff --git a/tests/unit/test_network_download.py b/tests/unit/test_network_download.py
index 14998d229..128b62b06 100644
--- a/tests/unit/test_network_download.py
+++ b/tests/unit/test_network_download.py
@@ -1,53 +1,79 @@
 import logging
 import sys
-from typing import Dict
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+from unittest.mock import MagicMock, call, patch
 
 import pytest
 
+from pip._internal.exceptions import IncompleteDownloadError
 from pip._internal.models.link import Link
 from pip._internal.network.download import (
+    Downloader,
+    _get_http_response_size,
+    _http_get_download,
     _prepare_download,
     parse_content_disposition,
     sanitize_content_filename,
 )
+from pip._internal.network.session import PipSession
+from pip._internal.network.utils import HEADERS
 
 from tests.lib.requests_mocks import MockResponse
 
 
 @pytest.mark.parametrize(
-    "url, headers, from_cache, expected",
+    "url, headers, from_cache, range_start, expected",
     [
         (
             "http://example.com/foo.tgz",
             {},
             False,
+            None,
             "Downloading http://example.com/foo.tgz",
         ),
         (
             "http://example.com/foo.tgz",
             {"content-length": "2"},
             False,
+            None,
             "Downloading http://example.com/foo.tgz (2 bytes)",
         ),
         (
             "http://example.com/foo.tgz",
             {"content-length": "2"},
             True,
+            None,
             "Using cached http://example.com/foo.tgz (2 bytes)",
         ),
-        ("https://files.pythonhosted.org/foo.tgz", {}, False, "Downloading foo.tgz"),
+        (
+            "https://files.pythonhosted.org/foo.tgz",
+            {},
+            False,
+            None,
+            "Downloading foo.tgz",
+        ),
         (
             "https://files.pythonhosted.org/foo.tgz",
             {"content-length": "2"},
             False,
+            None,
             "Downloading foo.tgz (2 bytes)",
         ),
         (
             "https://files.pythonhosted.org/foo.tgz",
             {"content-length": "2"},
             True,
+            None,
             "Using cached foo.tgz",
         ),
+        (
+            "http://example.com/foo.tgz",
+            {"content-length": "200"},
+            False,
+            100,
+            "Resuming download http://example.com/foo.tgz (100 bytes/200 bytes)",
+        ),
     ],
 )
 def test_prepare_download__log(
@@ -55,6 +81,7 @@ def test_prepare_download__log(
     url: str,
     headers: Dict[str, str],
     from_cache: bool,
+    range_start: Optional[int],
     expected: str,
 ) -> None:
     caplog.set_level(logging.INFO)
@@ -64,7 +91,14 @@ def test_prepare_download__log(
     if from_cache:
         resp.from_cache = from_cache
     link = Link(url)
-    _prepare_download(resp, link, progress_bar="on")
+    total_length = _get_http_response_size(resp)
+    _prepare_download(
+        resp,
+        link,
+        progress_bar="on",
+        total_length=total_length,
+        range_start=range_start,
+    )
 
     assert len(caplog.records) == 1
     record = caplog.records[0]
@@ -114,6 +148,29 @@ def test_sanitize_content_filename__platform_dependent(
     assert sanitize_content_filename(filename) == expected
 
 
+@pytest.mark.parametrize(
+    "range_start, if_range, expected_headers",
+    [
+        (None, None, HEADERS),
+        (1234, None, {**HEADERS, "Range": "bytes=1234-"}),
+        (1234, '"etag"', {**HEADERS, "Range": "bytes=1234-", "If-Range": '"etag"'}),
+    ],
+)
+def test_http_get_download(
+    range_start: Optional[int],
+    if_range: Optional[str],
+    expected_headers: Dict[str, str],
+) -> None:
+    session = PipSession()
+    session.get = MagicMock()
+    link = Link("http://example.com/foo.tgz")
+    with patch("pip._internal.network.download.raise_for_status"):
+        _http_get_download(session, link, range_start, if_range)
+    session.get.assert_called_once_with(
+        "http://example.com/foo.tgz", headers=expected_headers, stream=True
+    )
+
+
 @pytest.mark.parametrize(
     "content_disposition, default_filename, expected",
     [
@@ -125,3 +182,180 @@ def test_parse_content_disposition(
 ) -> None:
     actual = parse_content_disposition(content_disposition, default_filename)
     assert actual == expected
+
+
+@pytest.mark.parametrize(
+    "resume_retries,mock_responses,expected_resume_args,expected_bytes",
+    [
+        # If content-length is not provided, the download will
+        # always "succeed" since we don't have a way to check if
+        # the download is complete.
+        (
+            0,
+            [({}, 200, b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89")],
+            [],
+            b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+        # Complete download (content-length matches body)
+        (
+            0,
+            [({"content-length": "36"}, 200, b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89")],
+            [],
+            b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+        # Incomplete download without resume retries
+        (
+            0,
+            [({"content-length": "36"}, 200, b"0cfa7e9d-1868-4dd7-9fb3-")],
+            [],
+            None,
+        ),
+        # Incomplete download with resume retries
+        (
+            5,
+            [
+                ({"content-length": "36"}, 200, b"0cfa7e9d-1868-4dd7-9fb3-"),
+                ({"content-length": "12"}, 206, b"f2561d5dfd89"),
+            ],
+            [(24, None)],
+            b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+        # If the server responds with 200 (e.g. no range header support or the file
+        # has changed between the requests) the downloader should restart instead of
+        # attempting to resume. The downloaded file should not be affected.
+        (
+            5,
+            [
+                ({"content-length": "36"}, 200, b"0cfa7e9d-1868-4dd7-9fb3-"),
+                ({"content-length": "36"}, 200, b"0cfa7e9d-1868-"),
+                (
+                    {"content-length": "36"},
+                    200,
+                    b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+                ),
+            ],
+            [(24, None), (14, None)],
+            b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+        # File size could change between requests. Make sure this is handled correctly.
+        (
+            5,
+            [
+                ({"content-length": "36"}, 200, b"0cfa7e9d-1868-4dd7-9fb3-"),
+                (
+                    {"content-length": "40"},
+                    200,
+                    b"new-0cfa7e9d-1868-4dd7-9fb3-f2561d5d",
+                ),
+                ({"content-length": "4"}, 206, b"fd89"),
+            ],
+            [(24, None), (36, None)],
+            b"new-0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+        # The downloader should fail after n resume_retries attempts.
+        # This prevents the downloader from getting stuck if the connection
+        # is unstable and the server doesn't not support range requests.
+        (
+            1,
+            [
+                ({"content-length": "36"}, 200, b"0cfa7e9d-1868-4dd7-9fb3-"),
+                ({"content-length": "36"}, 200, b"0cfa7e9d-1868-"),
+            ],
+            [(24, None)],
+            None,
+        ),
+        # The downloader should use If-Range header to make the range
+        # request conditional if it is possible to check for modifications
+        # (e.g. if we know the creation time of the initial response).
+        (
+            5,
+            [
+                (
+                    {
+                        "content-length": "36",
+                        "last-modified": "Wed, 21 Oct 2015 07:28:00 GMT",
+                    },
+                    200,
+                    b"0cfa7e9d-1868-4dd7-9fb3-",
+                ),
+                (
+                    {
+                        "content-length": "12",
+                        "last-modified": "Wed, 21 Oct 2015 07:54:00 GMT",
+                    },
+                    206,
+                    b"f2561d5dfd89",
+                ),
+            ],
+            [(24, "Wed, 21 Oct 2015 07:28:00 GMT")],
+            b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+        # ETag is preferred over Last-Modified for the If-Range condition.
+        (
+            5,
+            [
+                (
+                    {
+                        "content-length": "36",
+                        "last-modified": "Wed, 21 Oct 2015 07:28:00 GMT",
+                        "etag": '"33a64df551425fcc55e4d42a148795d9f25f89d4"',
+                    },
+                    200,
+                    b"0cfa7e9d-1868-4dd7-9fb3-",
+                ),
+                (
+                    {
+                        "content-length": "12",
+                        "last-modified": "Wed, 21 Oct 2015 07:54:00 GMT",
+                        "etag": '"33a64df551425fcc55e4d42a148795d9f25f89d4"',
+                    },
+                    206,
+                    b"f2561d5dfd89",
+                ),
+            ],
+            [(24, '"33a64df551425fcc55e4d42a148795d9f25f89d4"')],
+            b"0cfa7e9d-1868-4dd7-9fb3-f2561d5dfd89",
+        ),
+    ],
+)
+def test_downloader(
+    resume_retries: int,
+    mock_responses: List[Tuple[Dict[str, str], int, bytes]],
+    # list of (range_start, if_range)
+    expected_resume_args: List[Tuple[Optional[int], Optional[int]]],
+    # expected_bytes is None means the download should fail
+    expected_bytes: Optional[bytes],
+    tmpdir: Path,
+) -> None:
+    session = PipSession()
+    link = Link("http://example.com/foo.tgz")
+    downloader = Downloader(session, "on", resume_retries)
+
+    responses = []
+    for headers, status_code, body in mock_responses:
+        resp = MockResponse(body)
+        resp.headers = headers
+        resp.status_code = status_code
+        responses.append(resp)
+    _http_get_download = MagicMock(side_effect=responses)
+
+    with patch("pip._internal.network.download._http_get_download", _http_get_download):
+        if expected_bytes is None:
+            remove = MagicMock(return_value=None)
+            with patch("os.remove", remove):
+                with pytest.raises(IncompleteDownloadError):
+                    downloader(link, str(tmpdir))
+            # Make sure the incomplete file is removed
+            remove.assert_called_once()
+        else:
+            filepath, _ = downloader(link, str(tmpdir))
+            with open(filepath, "rb") as downloaded_file:
+                downloaded_bytes = downloaded_file.read()
+                assert downloaded_bytes == expected_bytes
+
+    calls = [call(session, link)]  # the initial request
+    for range_start, if_range in expected_resume_args:
+        calls.append(call(session, link, range_start=range_start, if_range=if_range))
+
+    # Make sure that the download makes additional requests for resumption
+    _http_get_download.assert_has_calls(calls)
diff --git a/tests/unit/test_operations_prepare.py b/tests/unit/test_operations_prepare.py
index 86e26c118..bc6edcb3b 100644
--- a/tests/unit/test_operations_prepare.py
+++ b/tests/unit/test_operations_prepare.py
@@ -32,7 +32,7 @@ def test_unpack_url_with_urllib_response_without_content_type(data: TestData) ->
 
     session = Mock()
     session.get = _fake_session_get
-    download = Downloader(session, progress_bar="on")
+    download = Downloader(session, progress_bar="on", resume_retries=0)
 
     uri = data.packages.joinpath("simple-1.0.tar.gz").as_uri()
     link = Link(uri)
@@ -78,7 +78,7 @@ def test_download_http_url__no_directory_traversal(
         "content-disposition": 'attachment;filename="../out_dir_file"',
     }
     session.get.return_value = resp
-    download = Downloader(session, progress_bar="on")
+    download = Downloader(session, progress_bar="on", resume_retries=0)
 
     download_dir = os.fspath(tmpdir.joinpath("download"))
     os.mkdir(download_dir)
diff --git a/tests/unit/test_req.py b/tests/unit/test_req.py
index e243a7187..327597a8c 100644
--- a/tests/unit/test_req.py
+++ b/tests/unit/test_req.py
@@ -109,6 +109,7 @@ class TestRequirementSet:
                 lazy_wheel=False,
                 verbosity=0,
                 legacy_resolver=True,
+                resume_retries=0,
             )
             yield Resolver(
                 preparer=preparer,