Repository: readthedocs/readthedocs.org

Issue:
Invalid certificates (Let's Encrypt related)
Probably due to the fact that the Let's Encrypt root certificate expired on 30 September 2021, trying to pull from many non-Github repositories leads to ``server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none``.
Would it be possible to update the certificates to resolve this?
This is for instance an issue when using submodules that are not hosted on GitHub.


Discussion:
>  Would it be possible to update the certificates to resolve this?

What are the required steps that Read the Docs should follow to do this?
> trying to pull from many non-Github repositories leads to `server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none`.

Do you have an example of a build URL failing with this error?
> Do you have an example of a build URL failing with this error?

https://readthedocs.org/api/v2/build/14898416.txt

> What are the required steps that Read the Docs should follow to do this?

I don't know exactly how you manage your VMs but you need to patch the system certificate of your Linux (Ubuntu LTS?) to use the new Let'sEncrypt certificate (most probably Ubuntu release an update for OpenSSL or something like that, I don't use this distro so I don't know exactly)
Thanks for your reply.

I found what's required: `apt-get update` and `apt-get install ca-certificates`. After doing that, I'm able to clone the repository and its submodules from inside the VMs.

Now the problems 😞 . To update `ca-certificates` in our build images, we have to rebuild them all and that will bring known issues that we can't cleanly solve (basically, re-building an image doesn't make it exactly the same with the same software versions and people's builds start failing because of that). So, I'm not sure we will be able to re-build all the Docker images (`stable`, `latest`, `testing`).

However, we can easily rebuild our latest one (`readthedocs/build:ubuntu-20.04`) with the latest `ca-certificates` version (`20210119~20.04.2`) installed and tell users to define this image on their config file.
Another workaround would be to execute `apt-get install ca-certificates` as part of the build process from inside Read the Docs application itself.
I think we should be able to rebuild all the images, not just our beta image. Ubuntu releases are stable enough that I would trust things to work. Package versions do not drift heavily once in an LTS
> However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

That's fine with me :+1: can you ping me once that's done?

I don't think a workaround is a good idea, the first option would probably save you some work when others start having that same issue
> However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

Wouldn't that introduce a chicken and egg problem whereby the repo needs to be able to be cloned so that the configuration required to clone the repo can be read?
I am experiencing the same problem as op described. My code repository is hosted on a GitLab server.

If there is something with which I (a noob) could help, please let me know.
@rjw57 

> Wouldn't that introduce a chicken and egg problem whereby the repo needs to be able to be cloned so that the configuration required to clone the repo can be read?

Yes. I may express myself wrong here.

We use a default Docker image to run `git` commands inside it (currently we use `readthedocs/build:latest`) and I think we could use the new image `readthedocs/build:ubuntu-20.04` with a newer version of `ca-certificates` to do the `git` commands.

However, even if the `git` commands succeed, there may be other commands that may fail in the build process due to the old CA certificates. In that case, users could define `readthedocs/build:ubuntu-20.04` as the build image to fix that problem.
xref https://stackoverflow.com/q/69439345/554319
@humitos can I approve the PR or does it have to be someone from RtD?
> @humitos can I approve the PR or does it have to be someone from RtD?

that's a core team decision, but if you have any feedback there, feel free to comment!
> that's a core team decision, but if you have any feedback there, feel free to comment!

thought so, so no, nothing to change on my side; I think this is the right server-side fix for now
We deployed a quick and temporal fix for now. Please, let us know if you still have issues with the certificates. Thanks!
I can confirm that the hotfix fixed our project build. Thanks a lot everyone!

Resolving patch:
diff --git a/readthedocs/doc_builder/environments.py b/readthedocs/doc_builder/environments.py
--- a/readthedocs/doc_builder/environments.py
+++ b/readthedocs/doc_builder/environments.py
@@ -493,6 +493,7 @@ def __init__(
             config=None,
             environment=None,
             record=True,
+            **kwargs,
     ):
         super().__init__(project, environment)
         self.version = version
@@ -552,6 +553,8 @@ class DockerBuildEnvironment(BuildEnvironment):
     container_time_limit = DOCKER_LIMITS.get('time')
 
     def __init__(self, *args, **kwargs):
+        container_image = kwargs.pop('container_image', None)
+
         super().__init__(*args, **kwargs)
         self.client = None
         self.container = None
@@ -568,6 +571,16 @@ def __init__(self, *args, **kwargs):
         if self.project.container_image:
             self.container_image = self.project.container_image
 
+        # Override the ``container_image`` if we pass it via argument.
+        #
+        # FIXME: This is a temporal fix while we explore how to make
+        # ``ubuntu-20.04`` the default build image without breaking lot of
+        # builds. For now, we are passing
+        # ``container_image='readthedocs/build:ubuntu-20.04'`` for the setup
+        # VCS step.
+        if container_image:
+            self.container_image = container_image
+
         if self.project.container_mem_limit:
             self.container_mem_limit = self.project.container_mem_limit
         if self.project.container_time_limit:
diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py
--- a/readthedocs/projects/models.py
+++ b/readthedocs/projects/models.py
@@ -1766,7 +1766,6 @@ def add_features(sender, **kwargs):
     ALL_VERSIONS_IN_HTML_CONTEXT = 'all_versions_in_html_context'
     CACHED_ENVIRONMENT = 'cached_environment'
     LIMIT_CONCURRENT_BUILDS = 'limit_concurrent_builds'
-    UPDATE_CA_CERTIFICATES = 'update_ca_certificates'
     CDN_ENABLED = 'cdn_enabled'
 
     # Versions sync related features
@@ -1851,10 +1850,6 @@ def add_features(sender, **kwargs):
             LIMIT_CONCURRENT_BUILDS,
             _('Limit the amount of concurrent builds'),
         ),
-        (
-            UPDATE_CA_CERTIFICATES,
-            _('Update ca-certificates Ubuntu package before VCS clone'),
-        ),
         (
             CDN_ENABLED,
             _('CDN support for a project\'s public versions when privacy levels are enabled.'),
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
--- a/readthedocs/projects/tasks/builds.py
+++ b/readthedocs/projects/tasks/builds.py
@@ -561,6 +561,9 @@ def run_setup(self):
             version=self.data.version,
             build=self.data.build,
             environment=self.get_vcs_env_vars(),
+            # Force the ``container_image`` to use one that has the latest
+            # ca-certificate package which is compatible with Lets Encrypt
+            container_image=settings.RTD_DOCKER_BUILD_SETTINGS['os']['ubuntu-20.04']
         )
 
         # Environment used for code checkout & initial configuration reading
@@ -656,25 +659,6 @@ def setup_vcs(self, environment):
         This also syncs versions in the DB.
         """
         self.update_build(state=BUILD_STATE_CLONING)
-
-        # Install a newer version of ca-certificates packages because it's
-        # required for Let's Encrypt certificates
-        # https://github.com/readthedocs/readthedocs.org/issues/8555
-        # https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
-        # TODO: remove this when a newer version of ``ca-certificates`` gets
-        # pre-installed in the Docker images
-        if self.data.project.has_feature(Feature.UPDATE_CA_CERTIFICATES):
-            environment.run(
-                'apt-get', 'update', '--assume-yes', '--quiet',
-                user=settings.RTD_DOCKER_SUPER_USER,
-                record=False,
-            )
-            environment.run(
-                'apt-get', 'install', '--assume-yes', '--quiet', 'ca-certificates',
-                user=settings.RTD_DOCKER_SUPER_USER,
-                record=False,
-            )
-
         self.sync_repo(environment)
 
         commit = self.data.build_commit or self.get_vcs_repo(environment).commit


Test patch: