Repository: buildbot/buildbot

Issue:
Allow slaves to substantiate in parallel
BuildRequestDistributor's _maybeStartBuildsOnBuilder() method is
decorated with @defer.inlineCallbacks.  This means that when it
uses "yield bldr.maybeStartBuild", the entire slave substantiation
process is sequentialized at that point.

slave substantiation can take considerable time, especially with
something like EC2 latent slaves.  Even without anything substantial
happening in the cloud init (user-data) script, it can easily take
a couple of minutes.  If any substantial configuration of the
OS takes place ins the user-data script, maybeStartBuild() can
easily take many minutes.

To enable parallel slave substantiation, we need to stop waiting on
bldr.maybeStartBuild() inline.  Instead we move the error
handling code into a callback function that can be added to the
Deferred that is returned by bldr.maybeStartBuild().

At least for for the EC2LatentBuildSlave class, everything else down
the chain is properly event based and/or threaded.  This small
code change allows slaves to substantiate in parallel.

Signed-off-by: Christopher J. Morrone morrone2@llnl.gov



Discussion:

Resolving patch:
diff --git a/master/buildbot/steps/source/git.py b/master/buildbot/steps/source/git.py
--- a/master/buildbot/steps/source/git.py
+++ b/master/buildbot/steps/source/git.py
@@ -227,12 +227,14 @@ def mode_incremental(self, _):
         else:
             yield self._fetchOrFallback()
 
+        yield self._syncSubmodule(None)
         yield self._updateSubmodule(None)
 
     def clean(self):
         command = ['clean', '-f', '-f', '-d']
         d = self._dovccmd(command)
         d.addCallback(self._fetchOrFallback)
+        d.addCallback(self._syncSubmodule)
         d.addCallback(self._updateSubmodule)
         d.addCallback(self._cleanSubmodule)
         return d
@@ -253,6 +255,7 @@ def fresh(self):
         else:
             yield self._doClobber()
             yield self._fullCloneOrFallback()
+        yield self._syncSubmodule()
         yield self._updateSubmodule()
         yield self._cleanSubmodule()
 
@@ -523,6 +526,12 @@ def computeSourceRevision(self, changes):
             return None
         return changes[-1].revision
 
+    def _syncSubmodule(self, _=None):
+        if self.submodules:
+            return self._dovccmd(['submodule', 'sync'])
+        else:
+            return defer.succeed(0)
+
     def _updateSubmodule(self, _=None):
         if self.submodules:
             return self._dovccmd(['submodule', 'update',


Test patch:
diff --git a/master/buildbot/test/unit/test_steps_source_git.py b/master/buildbot/test/unit/test_steps_source_git.py
--- a/master/buildbot/test/unit/test_steps_source_git.py
+++ b/master/buildbot/test/unit/test_steps_source_git.py
@@ -930,6 +930,9 @@ def test_mode_full_fresh_submodule(self):
             ExpectShell(workdir='wkdir',
                         command=['git', 'reset', '--hard', 'FETCH_HEAD', '--'])
             + 0,
+            ExpectShell(workdir='wkdir',
+                        command=['git', 'submodule', 'sync'])
+            + 0,
             ExpectShell(workdir='wkdir',
                         command=['git', 'submodule', 'update', '--init', '--recursive', '--force'])
             + 0,