Repository: readthedocs/readthedocs.org

Issue:
Optional use of shallow clone for git repos
It would be great if there was an option to enable shallow clone for repositories. If set, then RtD should use shallow clone (`--depth 1` option of `git clone`) instead of full clone if possible.
This will result in huge disk space saving for projects with large history.

E.g., for one of our projects `/www/readthedocs.org/user_builds/{project}/checkouts/latest` folder takes up to 400 Mb, when cloned with `--depth 1`, it takes up only 12 Mb.



Discussion:
This would definitely be worth looking into, this might be able to take a chunk of storage. I haven't looked into this at all, be the `--depth` option would need to be supported by `git fetch`, as we don't actually run a git clone except on initial import, afaik.

Looks like we can now support this with the `--no-single-branch` option on `git clone`. There doesn't appear to be a similar option for `git fetch`, though, so we might not be able to reliably update the list of branches while keeping a shallow clone on disk. 

If someone could test that this is possible, it would be a pretty sizable win in terms of disk space needed. 

What exactly do you mean by _update the list of branches_?

If it is simple obtaining a list of all branches and tags, then it can be done using `git ls-remote` command.

``` bash
git ls-remote --heads --tags https://github.com/rtfd/readthedocs.org.git
```

You don't even need to clone repository to do so.

See https://www.kernel.org/pub/software/scm/git/docs/git-ls-remote.html

travis uses --depth=50

We have the depth option in the fetch command too https://git-scm.com/docs/git-fetch/2.17.0#git-fetch---depthltdepthgt, I'm testing it, looks like is something that could work for every project. Any reason to not do that on every project? Also, depth=1 looks fine.
Do we still get branch/tag info with a shallow clone? 
Yeah, we do, actually I just have a POC for this. So, we could add a feature flag to test and force everyone to do a shallow clone, or make it an option for the users. The only thing I can see is where a custom extension needs info from other commits I guess?

Resolving patch:
diff --git a/readthedocs/vcs_support/backends/git.py b/readthedocs/vcs_support/backends/git.py
--- a/readthedocs/vcs_support/backends/git.py
+++ b/readthedocs/vcs_support/backends/git.py
@@ -35,6 +35,7 @@ class Backend(BaseVCS):
     supports_branches = True
     supports_submodules = True
     fallback_branch = 'master'  # default branch
+    repo_depth = 50
 
     def __init__(self, *args, **kwargs):
         super(Backend, self).__init__(*args, **kwargs)
@@ -131,7 +132,8 @@ def validate_submodules(self, config):
 
     def fetch(self):
         code, stdout, stderr = self.run(
-            'git', 'fetch', '--tags', '--prune', '--prune-tags',
+            'git', 'fetch', '--depth', str(self.repo_depth),
+            '--tags', '--prune', '--prune-tags',
         )
         if code != 0:
             raise RepositoryError
@@ -150,7 +152,8 @@ def checkout_revision(self, revision=None):
     def clone(self):
         """Clones the repository."""
         code, stdout, stderr = self.run(
-            'git', 'clone', self.repo_url, '.'
+            'git', 'clone', '--depth', str(self.repo_depth),
+            '--no-single-branch', self.repo_url, '.'
         )
         if code != 0:
             raise RepositoryError


Test patch: