Repository: pypa/cibuildwheel

Issue:
SSL issues with OSX on Azure
A contributor to my project recently enabled a test that attempts to download a resource using HTTPS. It seems that running these tests on Azure fails for the OSX environments we have with errors like:

```
 (<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)>)
```

Which leads to issues on github like https://github.com/scikit-learn/scikit-learn/issues/10201#issuecomment-365734582. It seems the "proper" solution is to properly install the SSL certificates that come with the python installation: https://bugs.python.org/issue29480 which can apparently be fixed with:

```
sudo /Applications/Python\ 3.6/Install\ Certificates.command
```

I'm not really familiar with how cibuildwheel is using the base image from the build environment, but is this something I should be able to control in my Azure environments? Is this something that can and should be fixed in cibuildwheel? Thanks for any help.

I see there are issues already related to SSL stuff, but they all seem a little different than this or say that this should already be fixed. I must be missing something.


Discussion:
> A contributor to my project recently enabled a test that attempts to download a resource using HTTPS. It seems that running these tests on Azure fails for the OSX environments we have with errors like:

Could you provide information about cibuildwheel version, python version (running test)? Which macos version? 
Yes, sorry. Of course.

Here is the full azure config if necessary: https://github.com/vispy/vispy/blob/master/azure-pipelines.yml

The two OSX systems are both OSX 10.14, but one uses the default build target while the other does `MACOSX_DEPLOYMENT_TARGET="10.14"` before calling cibuildwheel.

Our currently build skips are:

```
  CIBW_SKIP: "cp27-* cp34-* cp35-* cp39-* pp27-* pp36-*"
```

So python 3.6, 3.7, and 3.8 should be built.

Edit: And this should be using the latest cibuildwheel release (1.6.1).
As project is public. Could You provide link to failing build? 
Let me know if you can see these: https://dev.azure.com/vispy/vispy/_build/results?buildId=574&view=results
*Ping @mayeut, who's been patching SSL in older Python versions for cibuildwheel before.*
@djhoese I see it. 
Could You ignore `cp36` and check if everything pass? 

It looks like outdated certificate center. 
Maybe wee need to add below call to install python on macos

```
sudo /Applications/Python\ 3.6/Install\ Certificates.command
``` 

Did we know any MacPython expert who could tell us for which version of python this should be called? 
I could, but I had our internal pytest test get skipped for python 3.6 and it then failed for python 3.7. My understanding based on the python issue mentioned above is that this is the intended way for Python 3+ to work now with regards to certificate installation on OSX.
Here's what I get for a list of CPython versions:
```
find /Applications/Python* -name 'Inst*'
/Applications/Python 2.7/Install Certificates.command
/Applications/Python 3.6/Install Certificates.command
/Applications/Python 3.7/Install Certificates.command
/Applications/Python 3.8/Install Certificates.command
/Applications/Python 3.9/Install Certificates.command
```

Only Python 3.5 doesn't come with that script but probably should (since it reports location of cert file like the others):
```
Matt$ /Library/Frameworks/Python.framework/Versions/3.5/bin/python3
Python 3.5.4 (v3.5.4:3f56838976, Aug  7 2017, 12:56:33) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.5/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.6/bin/python3
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 03:03:55) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
Python 3.7.5 (v3.7.5:5c02a39a0b, Oct 14 2019, 19:07:34) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.8/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/3.9/bin/python3
Python 3.9.0 (v3.9.0:9cf6752276, Oct  5 2020, 11:29:23) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/3.9/etc/openssl/cert.pem'
>>> ^D
Matt$ /Library/Frameworks/Python.framework/Versions/2.7/bin/python
Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 16:07:15) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.get_default_verify_paths().openssl_cafile
'/Library/Frameworks/Python.framework/Versions/2.7/etc/openssl/cert.pem'
>>> ^D
```
The script does the same thing for all versions, install/upgrade `certifi`, overwrite the `ssl.get_default_verify_paths().openssl_cafile` by a symlink to `certifi.where()` 

Resolving patch:
diff --git a/cibuildwheel/macos.py b/cibuildwheel/macos.py
--- a/cibuildwheel/macos.py
+++ b/cibuildwheel/macos.py
@@ -11,7 +11,7 @@
 from .environment import ParsedEnvironment
 from .util import (BuildOptions, BuildSelector, NonPlatformWheelError, download,
                    get_build_verbosity_extra_flags, get_pip_script,
-                   prepare_command)
+                   prepare_command, install_certifi_script)
 
 
 def call(args: Union[str, Sequence[Union[str, PathLike]]], env: Optional[Dict[str, str]] = None, cwd: Optional[str] = None, shell: bool = False) -> int:
@@ -72,6 +72,9 @@ def install_cpython(version: str, url: str) -> Path:
 
     # if this version of python isn't installed, get it from python.org and install
     python_package_identifier = f'org.python.Python.PythonFramework-{version}'
+    python_executable = 'python3' if version[0] == '3' else 'python'
+    installation_bin_path = Path(f'/Library/Frameworks/Python.framework/Versions/{version}/bin')
+
     if python_package_identifier not in installed_system_packages:
         # download the pkg
         download(url, Path('/tmp/Python.pkg'))
@@ -83,8 +86,8 @@ def install_cpython(version: str, url: str) -> Path:
             download(open_ssl_patch_url, Path('/tmp/python-patch.tar.gz'))
             call(['sudo', 'tar', '-C', f'/Library/Frameworks/Python.framework/Versions/{version}/', '-xmf', '/tmp/python-patch.tar.gz'])
 
-    installation_bin_path = Path(f'/Library/Frameworks/Python.framework/Versions/{version}/bin')
-    python_executable = 'python3' if version[0] == '3' else 'python'
+        call(["sudo", str(installation_bin_path/python_executable), str(install_certifi_script)])
+
     pip_executable = 'pip3' if version[0] == '3' else 'pip'
     make_symlinks(installation_bin_path, python_executable, pip_executable)
 
diff --git a/cibuildwheel/resources/install_certifi.py b/cibuildwheel/resources/install_certifi.py
new file mode 100644
--- /dev/null
+++ b/cibuildwheel/resources/install_certifi.py
@@ -0,0 +1,52 @@
+# Based on: https://github.com/python/cpython/blob/master/Mac/BuildScript/resources/install_certificates.command
+
+# install_certifi.py
+#
+# sample script to install or update a set of default Root Certificates
+# for the ssl module.  Uses the certificates provided by the certifi package:
+#       https://pypi.org/project/certifi/
+
+import os
+import os.path
+import ssl
+import stat
+import subprocess
+import sys
+
+STAT_0o775 = (stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
+              | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP
+              | stat.S_IROTH | stat.S_IXOTH)
+
+if sys.version_info[0] == 2:
+    FileNotFoundError = OSError
+
+
+def main():
+    openssl_dir, openssl_cafile = os.path.split(
+        ssl.get_default_verify_paths().openssl_cafile)
+    print(" -- pip install --upgrade certifi")
+    subprocess.check_call([sys.executable,
+                           "-E", "-s", "-m", "pip", "install", "--upgrade", "certifi"])
+
+    import certifi
+    # change working directory to the default SSL directory
+    if sys.version_info[0:2] == (3, 5):
+        os.makedirs(openssl_dir, exist_ok=True, mode=0o775)
+    os.chdir(openssl_dir)
+    relpath_to_certifi_cafile = os.path.relpath(certifi.where())
+
+    print(" -- removing any existing file or link")
+    try:
+        os.remove(openssl_cafile)
+    except FileNotFoundError:
+        pass
+    print(" -- creating symlink to certifi certificate bundle")
+    os.symlink(relpath_to_certifi_cafile, openssl_cafile)
+
+    print(" -- setting permissions")
+    os.chmod(openssl_cafile, STAT_0o775)
+    print(" -- update complete")
+
+
+if __name__ == '__main__':
+    main()
diff --git a/cibuildwheel/util.py b/cibuildwheel/util.py
--- a/cibuildwheel/util.py
+++ b/cibuildwheel/util.py
@@ -127,6 +127,7 @@ class BuildOptions(NamedTuple):
 
 resources_dir = Path(__file__).resolve().parent / 'resources'
 get_pip_script = resources_dir / 'get-pip.py'
+install_certifi_script = resources_dir / "install_certifi.py"
 
 
 class NonPlatformWheelError(Exception):


Test patch:
diff --git a/test/test_ssl.py b/test/test_ssl.py
--- a/test/test_ssl.py
+++ b/test/test_ssl.py
@@ -13,11 +13,10 @@
         else:
             from urllib.request import urlopen
 
-        if sys.version_info[0:2] == (3, 3):
-            data = urlopen("https://www.nist.gov")
-        else:
-            context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
-            data = urlopen("https://www.nist.gov", context=context)
+        context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
+        data = urlopen("https://www.nist.gov", context=context)
+        data = urlopen("https://raw.githubusercontent.com/joerick/cibuildwheel/master/CI.md", context=context)
+        data = urlopen("https://raw.githubusercontent.com/joerick/cibuildwheel/master/CI.md")
     ''')
 )