Repository: readthedocs/readthedocs.org

Issue:
Execute all checkout steps inside docker
Right now we execute the checkout step (clone, pull) in the build itself (outside docker). The rest of commands (install deps, build html, etc) are executed inside docker.

We can't execute all the commands inside docker because we are using gitpython to parse branches/tags and other operations.

Possible solutions:

1) Go back to parse the git output ourselves and drop gitpython (we basically only uses gitpython to list branches/tags/submodules).
2) Use gitpython inside docker and stream the results back. We can have the results back using pickle (maybe it can break due different python versions) or using something else like json (we only need branches/tags/submodules, so it can be just one call to generate a json file and read from there)


Discussion:
I think we should definitely we trying to run everything in docker, so #2 sounds the best to me. I think we can do it in stages though. Let's move the clone inside docker first, then we can figure out the best approach for the other commands after that. 
What's exactly the idea behind 2? If I understand correctly, we call `gitpython` from inside Docker querying for all the info we need (branches, tags, commit hash, etc) and save all of that into a JSON file. Then, from the Host we read that file. Am I right?

> I wouldn't use `pickle` as it's known to have security issues
@humitos yeah, something like mounting the script in `rtd/bin/get_vcs_information.py`
I'm thinking that could be tricky. That script would need a Python environment with `gitpython` installed to be executed, right?

Another approach could be mixture of this: query git repository data from the host (using gitpython), and use this data from the container to run the git commands inside it.
I don't think installing gitpython in the build env is too much. We already have lots of other Python dependencies there. I think the bigger question is around architecture, and if we want more running in the build env. I think that seems like a reasonable approach, and we should at least put together a proof of concept, it shouldn't be too hard. 
So, looks like it doesn't matter if we install gitpython in the user env, bc we can't. We have a chicken/egg problem. We do all the git steps (get branches and tags) and after that we create the python env.
But I think we can use the global python interpreter or create an env just to execute gitpython
I have this silly POC https://github.com/readthedocs/readthedocs.org/compare/master...stsewd:stsewd/get-git-info-from-container

It works(1) in the local environment and docker, there are some things to decide:

- Use a virtualenv to install gitpython (we can refactor our current Virtualenv class to work without version, config etc) or just install it in the current virtualenv (local environment gets polluted with gitpython here)
- How to _copy_ the script. In the POC I'm mounting the script with docker, but we can make the script an installable package
- Define the error codes from the script. Right now it only exists with 1 for all errors, but I think we want to have better error msgs to show to the user.

1. Only creates a json object form the output, but it would be easy to refactor the  other methods to make use of it.

Resolving patch:
diff --git a/readthedocs/projects/tasks.py b/readthedocs/projects/tasks.py
--- a/readthedocs/projects/tasks.py
+++ b/readthedocs/projects/tasks.py
@@ -358,17 +358,17 @@ def __init__(
 
     # pylint: disable=arguments-differ
     def run(
-            self, version_pk, build_pk=None, commit=None, record=True, docker=None,
+            self, version_pk, build_pk=None, commit=None, record=True,
             force=False, **__
     ):
         """
         Run a documentation sync n' build.
 
         This is fully wrapped in exception handling to account for a number of
-        failure cases. We first run a few commands in a local build environment,
+        failure cases. We first run a few commands in a build environment,
         but do not report on environment success. This avoids a flicker on the
         build output page where the build is marked as finished in between the
-        local environment steps and the docker build steps.
+        checkout steps and the build steps.
 
         If a failure is raised, or the build is not successful, return
         ``False``, otherwise, ``True``.
@@ -381,8 +381,6 @@ def run(
         :param build_pk int: Build id (if None, commands are not recorded)
         :param commit: commit sha of the version required for sending build status reports
         :param record bool: record a build object in the database
-        :param docker bool: use docker to build the project (if ``None``,
-            ``settings.DOCKER_ENABLE`` is used)
         :param force bool: force Sphinx build
 
         :returns: whether build was successful or not
@@ -390,8 +388,6 @@ def run(
         :rtype: bool
         """
         try:
-            if docker is None:
-                docker = settings.DOCKER_ENABLE
             self.version = self.get_version(version_pk)
             self.project = self.version.project
             self.build = self.get_build(build_pk)
@@ -403,7 +399,7 @@ def run(
             setup_successful = self.run_setup(record=record)
             if not setup_successful:
                 return False
-            self.run_build(docker=docker, record=record)
+            self.run_build(record=record)
             return True
         except Exception:
             log.exception(
@@ -444,11 +440,16 @@ def run(
 
     def run_setup(self, record=True):
         """
-        Run setup in the local environment.
+        Run setup in a build environment.
 
         Return True if successful.
         """
-        environment = LocalBuildEnvironment(
+        if settings.DOCKER_ENABLE:
+            env_cls = DockerBuildEnvironment
+        else:
+            env_cls = LocalBuildEnvironment
+
+        environment = env_cls(
             project=self.project,
             version=self.version,
             build=self.build,
@@ -524,19 +525,16 @@ def additional_vcs_operations(self, environment):
         if version_repo.supports_submodules:
             version_repo.update_submodules(self.config)
 
-    def run_build(self, docker, record):
+    def run_build(self, record):
         """
         Build the docs in an environment.
 
-        :param docker: if ``True``, the build uses a ``DockerBuildEnvironment``,
-            otherwise it uses a ``LocalBuildEnvironment`` to run all the
-            commands to build the docs
         :param record: whether or not record all the commands in the ``Build``
             instance
         """
         env_vars = self.get_env_vars()
 
-        if docker:
+        if settings.DOCKER_ENABLE:
             env_cls = DockerBuildEnvironment
         else:
             env_cls = LocalBuildEnvironment


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_config_integration.py b/readthedocs/rtd_tests/tests/test_config_integration.py
--- a/readthedocs/rtd_tests/tests/test_config_integration.py
+++ b/readthedocs/rtd_tests/tests/test_config_integration.py
@@ -493,7 +493,7 @@ def test_conda_environment(self, build_failed, checkout_path, tmpdir):
         )
 
         update_docs = self.get_update_docs_task()
-        update_docs.run_build(docker=False, record=False)
+        update_docs.run_build(record=False)
 
         assert update_docs.config.conda.environment == conda_file
         assert isinstance(update_docs.python_env, Conda)