Repository: beeware/briefcase

Issue:
Briefcase doesn't use system certificate store for lookups
### Describe the bug

We have had persistent issues with users - especially users with corporate VPNs - where downloads will fail with an "Are you connected to the internet?" error. 

Thanks to @pradyunsg, we've found the cause - httpx, by default, uses the `certifi` certificate store. This is almost always best practice, as it is updated regularly. However, some corporate VPNs mess with connections at the global level, even if the VPN isn't active. These VPNs patch/modify the system certificate store, but the certifi store isn't modified. As a result, these users get the bogus "connection" error.

### Steps to reproduce

1. On a laptop that has a "global" VPN installed, run `briefcase create`.


### Expected behavior

The download should complete if there is a valid certificate store.

### Screenshots

_No response_

### Environment

- Operating System: Any
- Python version: Any
- Software versions:
  - Briefcase: 0.3.23


### Logs

```

```


### Additional context

There are a couple of possible mitigations here:

1. The "Self-signed certificate" error should be caught explicitly, rather than manifesting as a generic "connection" error
2. There should be a `BRIEFCASE_USE_SYSTEM_CERTIFICATES` environment variable option that will [enable the use of system certificates in `httpx`](https://www.python-httpx.org/advanced/ssl/).
3. If a self-signed certificate is detected, an error directing the user to enable `BRIEFCASE_USE_SYSTEM_CERTIFICATES` should be displayed.



Discussion:
It might also be worthwhile checking for the `SSL_CERT_FILE ` and `SSL_CERT_DIR` environment variables.
To be a bit provocative perhaps, why not just use the window cert store by default? Or some sort of automatic fallback if necessary?

To me, I think it is most frustrating to users that Briefcase won't download the file but their web browser does. So, if the web browser defers to the machine's trusted certs, I think it makes sense for Briefcase to as well.
> To be a bit provocative perhaps, why not just use the window cert store by default? Or some sort of automatic fallback if necessary?

That's a fair question. The conversation we had at the sprints was of the opinion that `certifi` certificates are more likely to be up to date, because they reflect changes in root certificates that some OS vendors can be... less than responsive in that regard :-)

However, it was also noted that `pip` *does* do a "merged" certificate store - accepting both certifi and system store certificates. This is arguably problematic because a certificate that is known to be bad will be accepted if the system store hasn't been updated.

So - I guess it comes down to "pragmatism vs purity". `certifi` provides a way to get the guaranteed most up-to-date version of root stores; System certs are the ones that are most likely to reflect browser and OS behavior. On that basis, maybe it's not worth supporting the `certifi` option? After all, "it works in my browser" is the in-practice experience that we're seeing... while certifi might be theoretically correct, maybe it's better to just do what the system does, especially for file downloads where our fallback (that works) is "download using the browser". 

On that basis, maybe just using the system certificates, *not* using certifi at all, and skip the `BRIEFCASE_USE_SYSTEM_CERTIFICATES` option entirely.
> However, some corporate VPNs mess with connections at the global level, even if the VPN isn't active.

Specifically, they're doing an MITM on every SSL connection so they can monitor their employees' traffic. On such systems, only the system certificate store will work. 

> On that basis, maybe it's not worth supporting the certifi option? 

It does still have the advantage that it's pretty much guaranteed to support every legitimate server that's active at the time of release, whereas the system store might be missing a certificate if it hasn't been updated for a long time.

So I think we should go with the "merged" option, if it isn't too difficult.
After discussion, we decided that the system certificate store is probably good enough by itself, otherwise the user wouldn't even be able to do regular web browsing. So the plan is:

* On Python 3.10+, use [truststore](https://pypi.org/project/truststore/).
* On Python 3.9, which doesn't support truststore, continue using certifi. This will be EOL in about 6 months anyway.



Patch:
diff --git a/changes/2296.bugfix.rst b/changes/2296.bugfix.rst
new file mode 100644
index 000000000..5b93c5d09
--- /dev/null
+++ b/changes/2296.bugfix.rst
@@ -0,0 +1,1 @@
+Briefcase will now default to using the system certificate store when performing HTTPS downloads.
diff --git a/pyproject.toml b/pyproject.toml
index 9323fdd33..7ec0db34a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -74,6 +74,8 @@ dependencies = [
     "setuptools >= 60",
     "wheel >= 0.37",
     "build >= 0.10",
+    # truststore is only available on 3.10+
+    "truststore >= 0.10.1; python_version >= '3.10'",
     #
     # For the remaining packages: We set the lower version limit to the lowest possible
     # version that satisfies our API needs. If the package uses semver, we set a limit
diff --git a/src/briefcase/integrations/file.py b/src/briefcase/integrations/file.py
index 91738a0c0..f46b73d87 100644
--- a/src/briefcase/integrations/file.py
+++ b/src/briefcase/integrations/file.py
@@ -3,6 +3,7 @@
 import itertools
 import os
 import shutil
+import ssl
 import sys
 import tempfile
 from collections.abc import Iterable, Sequence
@@ -12,6 +13,12 @@
 
 import httpx
 
+if sys.version_info >= (3, 10):  # pragma: no-cover-if-lt-py310
+    import truststore
+else:  # pragma: no-cover-if-gte-py310
+    # truststore is only available for python 3.10+
+    truststore = None
+
 from briefcase.exceptions import (
     BadNetworkResourceError,
     MissingNetworkResourceError,
@@ -93,6 +100,22 @@ def sorted_depth_first_groups(
             )
         )
 
+    @property
+    def ssl_context(self):
+        """The SSL context to use for downloads."""
+        try:
+            return self._ssl_context
+        except AttributeError:
+            if sys.version_info >= (3, 10):  # pragma: no-cover-if-lt-py310
+                # Set up a TLS trust store based on the system root certificates
+                self._ssl_context = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+            else:  # pragma: no-cover-if-gte-py310
+                # Truststore requires Python 3.10; on older versions, fall back to the
+                # certifi-based store.
+                self._ssl_context = True
+
+            return self._ssl_context
+
     def is_archive(self, filename: str | os.PathLike) -> bool:
         """Can a file be unpacked via `shutil.unpack_archive()`?
 
@@ -174,7 +197,12 @@ def download(self, url: str, download_path: Path, role: str | None = None) -> Pa
         download_path.mkdir(parents=True, exist_ok=True)
         filename: Path = None
         try:
-            with self.tools.httpx.stream("GET", url, follow_redirects=True) as response:
+            with self.tools.httpx.stream(
+                "GET",
+                url,
+                follow_redirects=True,
+                verify=self.ssl_context,
+            ) as response:
                 if response.status_code == 404:
                     raise MissingNetworkResourceError(url=url)
                 elif response.status_code != 200:


Test patch:
diff --git a/tests/integrations/file/test_File__download.py b/tests/integrations/file/test_File__download.py
index fffd57ec9..046825ca0 100644
--- a/tests/integrations/file/test_File__download.py
+++ b/tests/integrations/file/test_File__download.py
@@ -161,6 +161,7 @@ def test_new_download_oneshot(mock_tools, file_perms, url, content_disposition):
         "GET",
         "https://example.com/support?useful=Yes",
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
     response.headers.get.assert_called_with("content-length")
     response.read.assert_called_once()
@@ -213,6 +214,7 @@ def test_new_download_chunked(mock_tools, file_perms):
         "GET",
         "https://example.com/support?useful=Yes",
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
     response.headers.get.assert_called_with("content-length")
     response.iter_bytes.assert_called_once_with(chunk_size=1048576)
@@ -276,6 +278,7 @@ def test_already_downloaded(mock_tools):
         "GET",
         url,
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
 
     # The request's Content-Disposition header is consumed to
@@ -316,6 +319,7 @@ def test_missing_resource(mock_tools):
         "GET",
         url,
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
     response.headers.get.assert_not_called()
 
@@ -351,6 +355,7 @@ def test_bad_resource(mock_tools):
         "GET",
         url,
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
     response.headers.get.assert_not_called()
 
@@ -389,6 +394,7 @@ def test_iter_bytes_connection_error(mock_tools):
         "GET",
         "https://example.com/something.zip?useful=Yes",
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
     response.headers.get.assert_called_with("content-length")
 
@@ -430,6 +436,7 @@ def test_connection_error(mock_tools):
         "GET",
         url,
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
 
     # The file doesn't exist as a result of the download failure
@@ -466,6 +473,7 @@ def test_redirect_connection_error(mock_tools):
         "GET",
         "https://example.com/something.zip?useful=Yes",
         follow_redirects=True,
+        verify=mock_tools.file.ssl_context,
     )
 
     # The file doesn't exist as a result of the download failure
diff --git a/tests/integrations/file/test_File__ssl_context.py b/tests/integrations/file/test_File__ssl_context.py
new file mode 100644
index 000000000..74e9cddeb
--- /dev/null
+++ b/tests/integrations/file/test_File__ssl_context.py
@@ -0,0 +1,17 @@
+import sys
+
+if sys.version_info >= (3, 10):  # pragma: no-cover-if-lt-py310
+    import truststore
+else:  # pragma: no-cover-if-gte-py310
+    # truststore is only available for python 3.10+
+    truststore = None
+
+
+def test_ssl_context(mock_tools):
+    """The SSL context is of the expected type."""
+    if sys.version_info >= (3, 10):
+        expected_type = truststore.SSLContext
+    else:
+        expected_type = bool
+
+    assert isinstance(mock_tools.file.ssl_context, expected_type)