Repository: pypa/bandersnatch (python)

Issue:
Add config/CLI arg to override https:// check for local mirrors (development)
When attempting to run a banderstnatch mirror against a locally-running index on port 80, a an exception is raised.
https://github.com/pypa/bandersnatch/blob/131438f40536c83c5074af90e2e337dfdadffebb/src/bandersnatch/master.py#L50-L53

Feature: Add a configuration parameter or CLI argument to skip the `https://` check and allow insecure targets.

Pull request description:
feat: allow using a non-https mirror via config
Resolves #1759

Patch:
diff --git a/CHANGES.md b/CHANGES.md
index 6973aa8..c42e692 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -9,13 +9,14 @@
 - Move Docker images to 3.12 (PR #1733)
   - Removing swift builds due to lack or assistance - Happy to bring back if you're willing to help maintian
 - Move black, mypy + pyupgrade to >= 3.11 codebase (PR #1734)
+- Allow non-HTTPS-enabled mirrors (PR #1765)
 
 ## Documentation
 
 - Updated documentation for `[mirror]` configuration options `PR #1669`
 - Updated documentation `PR #1760`
 
-## Big Fixes
+## Bug Fixes
 
 - Fix config file value interpolation for the `diff-file` option `PR #1715`
 - Fix diff-file being created when the option wasn't set `PR #1716`
diff --git a/docs/mirror_configuration.md b/docs/mirror_configuration.md
index bb5d471..b080227 100644
--- a/docs/mirror_configuration.md
+++ b/docs/mirror_configuration.md
@@ -52,6 +52,18 @@ download-mirror = https://pypi-mirror.example.com/
 
 This will download release files from `https://pypi-mirror.example.com` if possible and fall back to PyPI if a download fails. See [](#download-mirror). Add [](#download-mirror-no-fallback) to download release files exclusively from `download-mirror`.
 
+If you are using a download source that does not have HTTPS support, you can supply this configuration:
+
+```ini
+[mirror]
+...
+allow-non-https = true
+...
+```
+
+**Note**: It is not recommended to use this option in production environments as it may expose you to security vulnerabilities.
+Always ensure your PyPI server is running over `https://` in production.
+
 ### Index Files Only
 
 It is possible to mirror just index files without downloading any package release files:
diff --git a/src/bandersnatch/defaults.conf b/src/bandersnatch/defaults.conf
index ca1cbc7..ee041db 100644
--- a/src/bandersnatch/defaults.conf
+++ b/src/bandersnatch/defaults.conf
@@ -9,6 +9,7 @@ master = https://pypi.org
 proxy =
 download-mirror =
 download-mirror-no-fallback = false
+allow-non-https = false
 
 json = false
 release-files = true
diff --git a/src/bandersnatch/master.py b/src/bandersnatch/master.py
index 6a8ed6c..34c4ed8 100644
--- a/src/bandersnatch/master.py
+++ b/src/bandersnatch/master.py
@@ -41,13 +41,15 @@ class Master:
         timeout: float = 10.0,
         global_timeout: float | None = FIVE_HOURS_FLOAT,
         proxy: str | None = None,
+        allow_non_https: bool = False,
     ) -> None:
         self.proxy = proxy
         self.loop = asyncio.get_event_loop()
         self.timeout = timeout
         self.global_timeout = global_timeout or FIVE_HOURS_FLOAT
         self.url = url
-        if self.url.startswith("http://"):
+        self.allow_non_https = allow_non_https
+        if self.url.startswith("http://") and not self.allow_non_https:
             err = f"Master URL {url} is not https scheme"
             logger.error(err)
             raise ValueError(err)
diff --git a/src/bandersnatch/mirror.py b/src/bandersnatch/mirror.py
index 4f552d5..2cda422 100644
--- a/src/bandersnatch/mirror.py
+++ b/src/bandersnatch/mirror.py
@@ -970,6 +970,7 @@ async def mirror(
         )
 
     mirror_url = config.get("mirror", "master")
+    allow_non_https = config.getboolean("mirror", "allow-non-https")
     timeout = config.getfloat("mirror", "timeout")
     global_timeout = config.getfloat("mirror", "global-timeout", fallback=None)
     proxy = config.get("mirror", "proxy", fallback=None)
@@ -978,7 +979,9 @@ async def mirror(
 
     # Always reference those classes here with the fully qualified name to
     # allow them being patched by mock libraries!
-    async with Master(mirror_url, timeout, global_timeout, proxy) as master:
+    async with Master(
+        mirror_url, timeout, global_timeout, proxy, allow_non_https
+    ) as master:
         mirror = BandersnatchMirror(
             homedir,
             master,


Test patch:
diff --git a/src/bandersnatch/tests/test_configuration.py b/src/bandersnatch/tests/test_configuration.py
index 9005741..b4969b9 100644
--- a/src/bandersnatch/tests/test_configuration.py
+++ b/src/bandersnatch/tests/test_configuration.py
@@ -59,6 +59,7 @@ class TestBandersnatchConf(TestCase):
         self.assertSetEqual(
             options,
             {
+                "allow-non-https",
                 "cleanup",
                 "compare-method",
                 "diff-append-epoch",