Repository: buildbot/buildbot

Issue:
Maximum recursion depth exceeded while calling a Python object
Noticed this exception during buildbot start. I am using Buildbot 1.1.1

2018-04-11 21:06:54-0700 [-] Unhandled error in Deferred:
2018-04-11 21:06:54-0700 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1429, in _inlineCallbacks
            deferred.callback(e.value)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 459, in callback
            self._startRunCallbacks(result)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib64/python2.7/site-packages/twisted/spread/pb.py", line 1117, in _sendFailureOrError
            self._sendFailure(fail, requestID)
          File "/usr/lib64/python2.7/site-packages/twisted/spread/pb.py", line 1129, in _sendFailure
            log.msg("Peer will receive following PB traceback:")
          File "/usr/lib64/python2.7/site-packages/twisted/python/threadable.py", line 53, in sync 
            return function(self, *args, **kwargs)
          File "/usr/lib64/python2.7/site-packages/twisted/python/log.py", line 286, in msg
            _publishNew(self._publishPublisher, actualEventDict, textFromEventDict)
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver
            observer(eventDict)
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_observer.py", line 133, in __call__
            brokenObservers.append((observer, Failure()))
        exceptions.RuntimeError: maximum recursion depth exceeded while calling a Python object


Discussion:
Many of the workers are not able to connect to the master, can that be due to this error.
Another exception:

```
        Traceback (most recent call last):
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1429, in _inlineCallbacks
            deferred.callback(e.value)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 459, in callback
            self._startRunCallbacks(result)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib64/python2.7/site-packages/twisted/spread/pb.py", line 1117, in _sendFailureOrError
            self._sendFailure(fail, requestID)
          File "/usr/lib64/python2.7/site-packages/twisted/spread/pb.py", line 1129, in _sendFailure
            log.msg("Peer will receive following PB traceback:")
          File "/usr/lib64/python2.7/site-packages/twisted/python/threadable.py", line 53, in sync 
            return function(self, *args, **kwargs)
          File "/usr/lib64/python2.7/site-packages/twisted/python/log.py", line 286, in msg
            _publishNew(self._publishPublisher, actualEventDict, textFromEventDict)
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver
            observer(eventDict)
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_observer.py", line 136, in __call__
            errorLogger = self._errorLoggerForObserver(brokenObserver)
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_observer.py", line 156, in _errorLoggerForObserver
            if obs is not observer
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_observer.py", line 81, in __init__
            self.log = Logger(observer=self)
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_logger.py", line 52, in __init__
            namespace = self._namespaceFromCallingContext()
          File "/usr/lib64/python2.7/site-packages/twisted/logger/_logger.py", line 32, in _namespaceFromCallingContext
            return currentframe(2).f_globals["__name__"]
        exceptions.RuntimeError: maximum recursion depth exceeded
```
Another one:
```
        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/buildbot/pbmanager.py", line 217, in requestAvatarId
            yield self.master.initLock.release()
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1640, in release
            d.callback(self)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 459, in callback
            self._startRunCallbacks(result)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
        --- <exception caught here> ---
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 1432, in _inlineCallbacks
            deferred.errback()
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 500, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 666, in _runCallbacks
            current.result = failure.Failure(captureVars=self.debug)
          File "/usr/lib64/python2.7/site-packages/twisted/python/failure.py", line 206, in __init__
            if isinstance(exc_value, str) and exc_type is None:
        exceptions.RuntimeError: maximum recursion depth exceeded while calling a Python object
```
Adding "sys.setrecursionlimit(10000)" in master.cfg helps. Probably this issue is visible only when there are large number of workers configured on a master.
Did you upgrade your twisted as well?
This exception looks in the logging system. Did you change the way the logging is handled?
> This exception looks in the logging system. Did you change the way the logging is handled?

No, I didn't touch the Twisted version (it was 17.5.0) or anything in logging.

Resolving patch:
diff --git a/master/buildbot/pbmanager.py b/master/buildbot/pbmanager.py
--- a/master/buildbot/pbmanager.py
+++ b/master/buildbot/pbmanager.py
@@ -30,6 +30,7 @@
 from buildbot.util import bytes2unicode
 from buildbot.util import service
 from buildbot.util import unicode2bytes
+from buildbot.util.eventual import eventually
 
 debug = False
 
@@ -214,4 +215,4 @@ def requestAvatarId(self, creds):
             log.msg("invalid login from unknown user '%s'" % creds.username)
             raise error.UnauthorizedLogin()
         finally:
-            yield self.master.initLock.release()
+            eventually(self.master.initLock.release)


Test patch: