Repository: buildbot/buildbot

Issue:
Transferring Files with LocalWorker seems not working
This ticket is a migrated Trac ticket [3628](http://trac.buildbot.net/ticket/3628)

People contributed to the original ticket: @unknown_contributor, @sa2ajj
Ticket created on: `Oct 20 2016`
Ticket last modified on: `Oct 27 2016`

---

Hi,

I just update my buildbot to 0.9. And i convert my old Slave to [[LocalWorker]].

But now my builds are stuck during the copy process "DirectoryUpload" or "!Multiple[[FileUpload]]"

I check the `twistd.log` and it looks like the first light files are copied but then the copy is stuck during a transfer of 5MB. The file has always a size of 3227648o.

Thanks for your help


---





Discussion:
Just encountered the same problem: setup [["Python","3.5.1"],["Buildbot","0.9.15.post1"],["Twisted","17.9.0"]] on Win7-64. Configured one local worker. Try to steps.FileDowload() a 8Mb executable and get the following error. Dedicated worker is OK on localhost.

```
Traceback (most recent call last):
  File "c:\03-dev\buildbot\bb-py3-env\lib\site-packages\twisted\internet\defer.py", line 500, in errback
    self._startRunCallbacks(fail)
  File "c:\03-dev\buildbot\bb-py3-env\lib\site-packages\twisted\internet\defer.py", line 567, in _startRunCallbacks
    self._runCallbacks()
  File "c:\03-dev\buildbot\bb-py3-env\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "c:\03-dev\buildbot\bb-py3-env\lib\site-packages\twisted\internet\defer.py", line 1442, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "c:\03-dev\buildbot\bb-py3-env\lib\site-packages\twisted\internet\defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "c:\03-dev\buildbot\bb-py3-env\lib\site-packages\buildbot\process\buildstep.py", line 705, in run
    results = yield self._start_deferred
buildbot_worker.exceptions.AbandonChain: -1
```
Its weird, because there is an integration test supposed to tests all transfer capabilities for the null protocol (i.e LocalWorker). We test on windows linux and pythonw vs python3 for those tests.
https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/integration/interop/test_transfer.py#L79

Actually the exact step FileDownload is not tested there (only StringDownload), maybe we can add that, but I doubt it would change anything.

The stack trace is weird. There is not much info in there.. AbandonChain -1... Could that be a permission issue?


it creates a partial file with 3.211.264 bytes (so I get less than mentioned in the original post ;-) ) which is also "partially" correct: self-extracting zip which I can start and  then it says (correctly) that the payload is corrupt. so it seams that simply the remaining part is missing. 

This doesn't look like a permission issue to me... Anything else that I could help to narrow down this issue? 
Interresting, so this could be an issue with the size of the transfer. Is this always only transfering only that exact number of bytes?
some manual attempts with completely newly setup masters / local-workers result in exactly 3211264 bytes per transfer
And I missed one line from the error txt

```
[...]
builtins.RecursionError: maximum recursion depth exceeded while calling a Python object
```
I checked it with a different master.cfg on my linux system (Manjaro 64.bit, [["Python","3.6.4"],["Buildbot","0.9.15.post1"],["Twisted","17.9.0"]]) with a builder that only contained the FileDownload step and a 46Mb .deb file:
1st transfer 3194880 bytes, 
every run therafter (even after completely wiping state.sqlite, worker directories ...) : 3211264 (remember that number ?-) 

Funny side note : I found the following comment in the file twisted\internet\defer.py:1386 where the traceback is coming from before the excption happens in the buildbot callback

```
    # This function is complicated by the need to prevent unbounded recursion
    # arising from repeatedly yielding immediately ready deferreds.  This while
    # loop and the waiting variable solve that by manually unfolding the
    # recursion.
```
I can reproduce, it looks like indeed an infinite recursion problem

Resolving patch:
diff --git a/master/buildbot/worker/protocols/null.py b/master/buildbot/worker/protocols/null.py
--- a/master/buildbot/worker/protocols/null.py
+++ b/master/buildbot/worker/protocols/null.py
@@ -20,6 +20,7 @@
 from twisted.internet import defer
 from twisted.python import log
 
+from buildbot.util.eventual import fireEventually
 from buildbot.worker.protocols import base
 
 
@@ -43,7 +44,8 @@ def callRemote(self, message, *args, **kw):
         except TypeError:
             log.msg("%s didn't accept %s and %s" % (method, args, kw))
             raise
-        return defer.maybeDeferred(lambda: state)
+        # break callback recursion for large transfers by using fireEventually
+        return fireEventually(state)
 
     def notifyOnDisconnect(self, cb):
         pass


Test patch:
diff --git a/master/buildbot/test/integration/interop/test_transfer.py b/master/buildbot/test/integration/interop/test_transfer.py
--- a/master/buildbot/test/integration/interop/test_transfer.py
+++ b/master/buildbot/test/integration/interop/test_transfer.py
@@ -45,7 +45,7 @@ def readMasterDirContents(self, top):
 
     @defer.inlineCallbacks
     def test_transfer(self):
-        yield self.setupConfig(masterConfig())
+        yield self.setupConfig(masterConfig(bigfilename=self.mktemp()))
 
         build = yield self.doForceBuild(wantSteps=True, wantLogs=True)
         self.assertEqual(build['results'], SUCCESS)
@@ -81,7 +81,7 @@ class TransferStepsMasterNull(TransferStepsMasterPb):
 
 
 # master configuration
-def masterConfig():
+def masterConfig(bigfilename):
     c = {}
     from buildbot.config import BuilderConfig
     from buildbot.process.factory import BuildFactory
@@ -96,6 +96,14 @@ def masterConfig():
     # do a bunch of transfer to exercise the protocol
     f.addStep(steps.StringDownload("filecontent", workerdest="dir/file1.txt"))
     f.addStep(steps.StringDownload("filecontent2", workerdest="dir/file2.txt"))
+    # create 8 MB file
+    with open(bigfilename, 'w') as o:
+        buf = "xxxxxxxx" * 1024
+        for i in range(1000):
+            o.write(buf)
+    f.addStep(
+        steps.FileDownload(
+            mastersrc=bigfilename, workerdest="bigfile.txt"))
     f.addStep(
         steps.FileUpload(workersrc="dir/file2.txt", masterdest="master.txt"))
     f.addStep(