Repository: readthedocs/readthedocs.org

Issue:
Shallow clone breaks build
## Details

* Read the Docs project URL: https://readthedocs.org/projects/patchwork/
* Build URL (if applicable): https://readthedocs.org/projects/patchwork/builds/8314815/
* Read the Docs username (if applicable):

## Expected Result

The build, which utilizes the `reno.sphinxext` module, should work.

## Actual Result

The build fails due to missing tags. These are presumably missing due to the shallow cloning of repos introduced in #4939.

## Additional Information

The original issue that #4939 purported to resolve was #1888, **Optional** use of shallow clone for git repos (emphasis mine). While shallow clone might be useful at large, it should be possible to disable it for projects that rely on git history to build their docs.



Discussion:
I'm guessing this is because the `reno` package needs the git history https://docs.openstack.org/reno/.
> I'm guessing this is because the `reno` package needs the git history [docs.openstack.org/reno](https://docs.openstack.org/reno/).

Indeed :disappointed: 
@stsewd I think we should add a Feature flag for these cases to omit shallowing.
We have to options here, feature flag (this can only be added by the core team), or an option in the web interface (this can be handled by the user).
Yeah, I've been looking at this. I see both approaches were adopted for submodules, with bd364946aa providing the feature flag approach and the later bfa6bdb346 providing a config file based approach. I'd been adding a `git.shallow_clone` setting using the latter approach based on the idea that pretty much everyone using reno or similar tools would be affected. I'm not sure how the config file ties into the web interface though and I don't see submodule configuration there. Is this a third option?
We can't have this option in the config file because the config file is parsed after the clone step. And we don't have the submodules option on the web because we are moving all per-version settings to the config only, all per-project settings will be available on the web.
Oh, of course 🤦 I guess web UI is the way to go so, though I'm not sure where to nestle that in.

Resolving patch:
diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py
--- a/readthedocs/projects/models.py
+++ b/readthedocs/projects/models.py
@@ -1013,6 +1013,7 @@ def add_features(sender, **kwargs):
     DONT_OVERWRITE_SPHINX_CONTEXT = 'dont_overwrite_sphinx_context'
     ALLOW_V2_CONFIG_FILE = 'allow_v2_config_file'
     MKDOCS_THEME_RTD = 'mkdocs_theme_rtd'
+    DONT_SHALLOW_CLONE = 'dont_shallow_clone'
 
     FEATURES = (
         (USE_SPHINX_LATEST, _('Use latest version of Sphinx')),
@@ -1021,10 +1022,12 @@ def add_features(sender, **kwargs):
         (PIP_ALWAYS_UPGRADE, _('Always run pip install --upgrade')),
         (SKIP_SUBMODULES, _('Skip git submodule checkout')),
         (DONT_OVERWRITE_SPHINX_CONTEXT, _(
-            'Do not overwrite context vars in conf.py with Read the Docs context',)),
+            'Do not overwrite context vars in conf.py with Read the Docs context')),
         (ALLOW_V2_CONFIG_FILE, _(
             'Allow to use the v2 of the configuration file')),
         (MKDOCS_THEME_RTD, _('Use Read the Docs theme for MkDocs as default theme')),
+        (DONT_SHALLOW_CLONE, _(
+            'Do not shallow clone when cloning git repos')),
     )
 
     projects = models.ManyToManyField(
diff --git a/readthedocs/vcs_support/backends/git.py b/readthedocs/vcs_support/backends/git.py
--- a/readthedocs/vcs_support/backends/git.py
+++ b/readthedocs/vcs_support/backends/git.py
@@ -131,11 +131,26 @@ def validate_submodules(self, config):
                 return False, []
         return True, submodules.keys()
 
+    def use_shallow_clone(self):
+        """
+        Test whether shallow clone should be performed.
+
+        .. note::
+
+            Temporarily, we support skipping this option as builds that rely on
+            git history can fail if using shallow clones. This should
+            eventually be configurable via the web UI.
+        """
+        from readthedocs.projects.models import Feature
+        return not self.project.has_feature(Feature.DONT_SHALLOW_CLONE)
+
     def fetch(self):
-        code, stdout, stderr = self.run(
-            'git', 'fetch', '--depth', str(self.repo_depth),
-            '--tags', '--prune', '--prune-tags',
-        )
+        cmd = ['git', 'fetch', '--tags', '--prune', '--prune-tags']
+
+        if self.use_shallow_clone():
+            cmd.extend(['--depth', str(self.repo_depth)])
+
+        code, stdout, stderr = self.run(*cmd)
         if code != 0:
             raise RepositoryError
         return code, stdout, stderr
@@ -152,10 +167,14 @@ def checkout_revision(self, revision=None):
 
     def clone(self):
         """Clones the repository."""
-        code, stdout, stderr = self.run(
-            'git', 'clone', '--depth', str(self.repo_depth),
-            '--no-single-branch', self.repo_url, '.'
-        )
+        cmd = ['git', 'clone', '--no-single-branch']
+
+        if self.use_shallow_clone():
+            cmd.extend(['--depth', str(self.repo_depth)])
+
+        cmd.extend([self.repo_url, '.'])
+
+        code, stdout, stderr = self.run(*cmd)
         if code != 0:
             raise RepositoryError
         return code, stdout, stderr


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_backend.py b/readthedocs/rtd_tests/tests/test_backend.py
--- a/readthedocs/rtd_tests/tests/test_backend.py
+++ b/readthedocs/rtd_tests/tests/test_backend.py
@@ -149,7 +149,7 @@ def test_skip_submodule_checkout(self):
         repo.update()
         repo.checkout('submodule')
         self.assertTrue(repo.are_submodules_available(self.dummy_conf))
-        feature = fixture.get(
+        fixture.get(
             Feature,
             projects=[self.project],
             feature_id=Feature.SKIP_SUBMODULES,
@@ -157,6 +157,19 @@ def test_skip_submodule_checkout(self):
         self.assertTrue(self.project.has_feature(Feature.SKIP_SUBMODULES))
         self.assertFalse(repo.are_submodules_available(self.dummy_conf))
 
+    def test_use_shallow_clone(self):
+        repo = self.project.vcs_repo()
+        repo.update()
+        repo.checkout('submodule')
+        self.assertTrue(repo.use_shallow_clone())
+        fixture.get(
+            Feature,
+            projects=[self.project],
+            feature_id=Feature.DONT_SHALLOW_CLONE,
+        )
+        self.assertTrue(self.project.has_feature(Feature.DONT_SHALLOW_CLONE))
+        self.assertFalse(repo.use_shallow_clone())
+
     def test_check_submodule_urls(self):
         repo = self.project.vcs_repo()
         repo.update()