Repository: readthedocs/readthedocs.org

Issue:
Build: incompatible Mercurial versions from VCS setup than build commands
## Details

* Read the Docs project URL: https://readthedocs.org/projects/taskcluster-taskgraph/
* Build URL (if applicable): https://readthedocs.org/projects/taskcluster-taskgraph/builds/16259866/
* Read the Docs username (if applicable): ahal

## Expected Result

The builds should succeed :)

## Actual Result

Builds fail as seen [here](https://readthedocs.org/projects/taskcluster-taskgraph/builds/16259866/). I don't believe anything changed on the repo side, this was [working a week or two ago](https://readthedocs.org/projects/taskcluster-taskgraph/builds/).

```
hg identify --id
abort: repository requires features unknown to this Mercurial: sparserevlog!
(see https://mercurial-scm.org/wiki/MissingRequirement for more information) 
```



Discussion:
I only skimmed through the logs, but this is weird. It looks like the `hg id` call is suddently too old to be able to read the repository that just got cloned by that same hg.

So, might this is no longer the same hg and some of the virtualenv play in between downgraded it somehow ?
I don't have any insight into reathedocs infrastructure, though a co-worker pointed out a [somewhat suspicious commit](https://www.mercurial-scm.org/repo/hg/rev/6d2ddea0721a2a70dc399bd94829ee36bf7597af) to Mercurial which would have been released a couple weeks before this started failing. But yeah, it's weird that the earlier `hg` commands aren't failing.

Edit: Apparently marmoute ruled out this commit in conversation with said co-worker. I'll leave this comment up just in case.
The "suspicious commit" (and the related work in this area) only affects the "streaming clone" feature. Streaming clone does not seems to be involved in this build at all. So I would be surprised if these changes play any part in the current failures.
We are seeing this on the [PyPy doc builds](https://readthedocs.org/projects/pypy/builds/) as well. Is there a way to find all the projects that are using mercurial to find one that is succeeding?
According to #8840 there are ~46 repos using mercurial. Are all their builds failing?
@ahal can you try using a Read the Docs config file with build.os (https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os) declared as Ubuntu 20.04?
I guess the problem here is that we are using a newer version in the `hg clone` and then an old version for the second `hg identify` command. I think this issue was introduced in https://github.com/readthedocs/readthedocs.org/pull/8930 when we started using the `ubuntu-20.04` for the VCS steps.

I think you will be able to fix this problem by using Ubuntu 20.04 in your config file as I suggested in my previous comment.

It would be good to find a way to tell Mercurial to clone the repository to make it compatible with older versions.
You can use `--config format.sparse-revlog=no` when cloning (or update the config of the thing doing the clone).
This problem will arise in the future as we turn format variant on by default overtime. Maybe Mercurial need some generic flag to request clone compatible with version X 
> @ahal can you try using a Read the Docs config file with build.os (https://docs.readthedocs.io/en/stable/config-file/v2.html#build-os) declared as Ubuntu 20.04?

Thanks, this works around the issue for me.
Something is wrong. I cannot build the PyPy docs https://readthedocs.org/projects/pypy/builds/ for the last day: the build fails to get `hg id`.
@mattip for some reason `hg tags` is returning `Command exited abnormally`. I was able to reproduce this error locally when building your project with my Read the Docs development instance, but I don't understand why it's happening yet. However, I'm not able to reproduce this by running the commands manually.

Resolving patch:
diff --git a/readthedocs/doc_builder/director.py b/readthedocs/doc_builder/director.py
--- a/readthedocs/doc_builder/director.py
+++ b/readthedocs/doc_builder/director.py
@@ -65,7 +65,30 @@ def setup_vcs(self):
                 ),
             )
 
-        environment = self.data.environment_class(
+        before_vcs.send(
+            sender=self.data.version,
+            environment=self.vcs_environment,
+        )
+
+        # Create the VCS repository where all the commands are going to be
+        # executed for a particular VCS type
+        self.vcs_repository = self.data.project.vcs_repo(
+            version=self.data.version.slug,
+            environment=self.vcs_environment,
+            verbose_name=self.data.version.verbose_name,
+            version_type=self.data.version.type,
+        )
+
+        self.pre_checkout()
+        self.checkout()
+        self.post_checkout()
+
+        commit = self.data.build_commit or self.vcs_repository.commit
+        if commit:
+            self.data.build["commit"] = commit
+
+    def create_vcs_environment(self):
+        self.vcs_environment = self.data.environment_class(
             project=self.data.project,
             version=self.data.version,
             build=self.data.build,
@@ -74,28 +97,6 @@ def setup_vcs(self):
             # ca-certificate package which is compatible with Lets Encrypt
             container_image=settings.RTD_DOCKER_BUILD_SETTINGS["os"]["ubuntu-20.04"],
         )
-        with environment:
-            before_vcs.send(
-                sender=self.data.version,
-                environment=environment,
-            )
-
-            # Create the VCS repository where all the commands are going to be
-            # executed for a particular VCS type
-            self.vcs_repository = self.data.project.vcs_repo(
-                version=self.data.version.slug,
-                environment=environment,
-                verbose_name=self.data.version.verbose_name,
-                version_type=self.data.version.type,
-            )
-
-            self.pre_checkout()
-            self.checkout()
-            self.post_checkout()
-
-            commit = self.data.build_commit or self.vcs_repository.commit
-            if commit:
-                self.data.build["commit"] = commit
 
     def create_build_environment(self):
         self.build_environment = self.data.environment_class(
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
--- a/readthedocs/projects/tasks/builds.py
+++ b/readthedocs/projects/tasks/builds.py
@@ -543,11 +543,20 @@ def execute(self):
 
         # Clonning
         self.update_build(state=BUILD_STATE_CLONING)
-        self.data.build_director.setup_vcs()
 
-        # Sync tags/branches from VCS repository into Read the Docs' `Version`
-        # objects in the database
-        self.sync_versions(self.data.build_director.vcs_repository)
+        # TODO: remove the ``create_vcs_environment`` hack. Ideally, this should be
+        # handled inside the ``BuildDirector`` but we can't use ``with
+        # self.vcs_environment`` twice because it kills the container on
+        # ``__exit__``
+        self.data.build_director.create_vcs_environment()
+        with self.data.build_director.vcs_environment:
+            self.data.build_director.setup_vcs()
+
+            # Sync tags/branches from VCS repository into Read the Docs'
+            # `Version` objects in the database. This method runs commands
+            # (e.g. "hg tags") inside the VCS environment, so it requires to be
+            # inside the `with` statement
+            self.sync_versions(self.data.build_director.vcs_repository)
 
         # TODO: remove the ``create_build_environment`` hack. Ideally, this should be
         # handled inside the ``BuildDirector`` but we can't use ``with


Test patch: