Repository: buildbot/buildbot

Issue:
Exception sometimes thrown by steps.MsBuild14
Hello,

Since my upgrade to BuildBot 1.9.2, if using the `MsBuild14` step to build a Visual Studio 2015 solution, I sometimes (not always) obtain the following trace in the `twistd.log`:

```
2020-12-15 10:53:20+0100 [-] when trying to finish a log
	Traceback (most recent call last):
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 501, in errback
	    self._startRunCallbacks(fail)
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
	    self._runCallbacks()
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1475, in gotResult
	    _inlineCallbacks(r, g, status)
	--- <exception caught here> ---
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/usr/local/lib/python3.5/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/usr/local/lib/python3.5/dist-packages/buildbot/process/log.py", line 203, in finish
	    yield super().finish()
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
	    result = g.send(result)
	  File "/usr/local/lib/python3.5/dist-packages/buildbot/process/log.py", line 115, in finish
	    self._had_errors = len(self.subPoint.pop_exceptions()) > 0
	builtins.TypeError: object of type 'NoneType' has no len()
	
2020-12-15 10:53:20+0100 [-] releaseLocks(MsBuild14(env={'PreferredToolArchitecture': 'x64'}, workdir='build/Build', projectfile='OrthancMySQL.sln', config='Release', platform='x64')): []
2020-12-15 10:53:20+0100 [-]  step 'compile' complete: exception (['built solution for Release|x64'])
2020-12-15 10:53:20+0100 [-]  <Build DB mainline - MySQL - VC2015-64 Release number:699 results:exception>: build finished
```

The build is then aborted and tagged with the `EXCEPTION` tag.  When I look at the `stdio` stream from Visual Studio, the build is reported as a success, and the binaries are properly generated:

```
    2551 Warning(s)
    0 Error(s)
Time Elapsed 00:07:53.31
program finished with exit code 0
elapsedTime=475.270000
```

This issue might be related to a large number of warnings being generated by Visual Studio, which seems to take long time to be parsed from `stdio` to `warnings` (this parsing seems to last more than 1 minute, which I find highly suspicious), although I am unsure about this. This issue didn't occur on releases 1.8.x of BuildBot.

Thanks in advance for your help, and thanks for your hard work on BuildBot!


Discussion:
Thanks a lot for the bug report. It seems that it's an issue with general Buildbot logs infrastructure as the error is not coming from the step itself.
Are you sure? I think this particular error is caused by oversight in 571a54916f commit. It's `self._got_exceptions = []` in `__init__`, but `self._got_exceptions = None` in `pop_exceptions` method. I wanted to write a fix, but have no idea how to write a test for this issue.
Hello, thanks for your feedback. I see that BuildBot to 1.9.4 is now released. I'm giving another try with this new version in order to see whether this makes the issue vanish on my setup, and I'll keep you updated.
I confirm that the same issue is still present after upgrading to 1.9.4:
```
2020-12-28 01:37:46+0100 [Broker,3,185.35.172.226] <RemoteShellCommand ''"%VCENV_BAT%" x86 && msbuild "OrthancMySQL.sln" /p:Configuration="Release" /p:Platform="x64" /maxcpucount /t:Rebuild''> rc=0
2020-12-28 01:41:23+0100 [-] when trying to finish a log
	Traceback (most recent call last):
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 501, in errback
	    self._startRunCallbacks(fail)
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
	    self._runCallbacks()
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
	    current.result = callback(current.result, *args, **kw)
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1475, in gotResult
	    _inlineCallbacks(r, g, status)
	--- <exception caught here> ---
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
	    result = result.throwExceptionIntoGenerator(g)
	  File "/usr/local/lib/python3.5/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
	    return g.throw(self.type, self.value, self.tb)
	  File "/usr/local/lib/python3.5/dist-packages/buildbot/process/log.py", line 203, in finish
	    yield super().finish()
	  File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
	    result = g.send(result)
	  File "/usr/local/lib/python3.5/dist-packages/buildbot/process/log.py", line 115, in finish
	    self._had_errors = len(self.subPoint.pop_exceptions()) > 0
	builtins.TypeError: object of type 'NoneType' has no len()
	
2020-12-28 01:41:24+0100 [-] releaseLocks(MsBuild14(platform='x64', projectfile='OrthancMySQL.sln', env={'PreferredToolArchitecture': 'x64'}, workdir='build/Build', config='Release')): []
2020-12-28 01:41:24+0100 [-]  step 'compile' complete: exception (['built solution for Release|x64'])
2020-12-28 01:41:24+0100 [-]  <Build DB mainline - MySQL - VC2015-64 Release number:719 results:exception>: build finished
```
The binaries are still correctly generated by the compiler:
```
    3073 Warning(s)
    0 Error(s)
Time Elapsed 00:10:37.03
program finished with exit code 0
elapsedTime=639.281000
```
Note that I use other types of builders (gcc, clang, OS X, Visual Studio 2008), but this problem only occurs with `steps.MsBuild14`.
Got the same error in 3.1.0. @p12tic are you sure that `_got_exceptions` https://github.com/buildbot/buildbot/blob/571a54916ff186afa345d501a5ab9e66b2ffb439/master/buildbot/util/subscription.py#L63 should become `None`, and not `[]`?
Hello everyone,

I'm still having this issue with :
Python version: 3.8.10
Buildbot version: 3.3.0
Twisted version: 21.7.0

The log has slightly changed, but it seem to me it is the same issue : 
```
2021-10-15 14:12:57+0000 [Broker,6,192.168.14.30] <RemoteShellCommand ''FOR /F "tokens=*" %%I in (\'vswhere.exe -property  installationPath -products *\') do "%%I\\%VCENV_BAT%" x86 && msbuild "XXXXXXX.sln" /p:Configuration="Release" /p:Platform="x64" /maxcpucount /t:"GbxGame" /p:DefineConstants="__XXXXXXX__"''> rc=1
2021-10-15 14:12:57+0000 [-] when trying to finish a log
        Traceback (most recent call last):
          File "/usr/local/lib/python3.8/dist-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
            result = current_context.run(gen.send, result)
          File "/usr/local/lib/python3.8/dist-packages/buildbot/process/log.py", line 207, in finish
            yield super().finish()
          File "/usr/local/lib/python3.8/dist-packages/twisted/internet/defer.py", line 1909, in unwindGenerator
            return _cancellableInlineCallbacks(gen)  # type: ignore[unreachable]
          File "/usr/local/lib/python3.8/dist-packages/twisted/internet/defer.py", line 1816, in _cancellableInlineCallbacks
            _inlineCallbacks(None, gen, status)
        --- <exception caught here> ---
          File "/usr/local/lib/python3.8/dist-packages/buildbot/process/log.py", line 207, in finish
            yield super().finish()
          File "/usr/local/lib/python3.8/dist-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
            result = current_context.run(gen.send, result)
          File "/usr/local/lib/python3.8/dist-packages/buildbot/process/log.py", line 101, in finish
            assert not self._finishing, "Did you maybe forget to yield the method?"
        builtins.AssertionError: Did you maybe forget to yield the method?
```

I've got the issue with 2 different workers, with versions : 3.3.0 and 2019.07.04, but the issue was not here before upgrading buildbot main server to 3.3.0, so my guess is that the bug is in the main server code.
I hope I'm not mistaken in supposing this is the same issue.

 I can provide more information if needed. 

Additionally : I reproduce these exceptions multiple times a day, and it's easy for me to test any patch, or log anything in my instance, but I'm not familiar enough with python and buildbot to solve that bug alone.

Resolving patch:
diff --git a/master/buildbot/steps/gitdiffinfo.py b/master/buildbot/steps/gitdiffinfo.py
--- a/master/buildbot/steps/gitdiffinfo.py
+++ b/master/buildbot/steps/gitdiffinfo.py
@@ -72,7 +72,7 @@ def run(self):
 
         yield self.runCommand(cmd)
         log = yield self.getLog("stdio-merge-base")
-        log.finish()
+        yield log.finish()
 
         if cmd.results() != results.SUCCESS:
             return cmd.results()
diff --git a/master/buildbot/steps/vstudio.py b/master/buildbot/steps/vstudio.py
--- a/master/buildbot/steps/vstudio.py
+++ b/master/buildbot/steps/vstudio.py
@@ -208,9 +208,9 @@ def getResultSummary(self):
     @defer.inlineCallbacks
     def finish_logs(self):
         log = yield self.getLog("warnings")
-        log.finish()
+        yield log.finish()
         log = yield self.getLog("errors")
-        log.finish()
+        yield log.finish()
 
 
 class VC6(VisualStudio):


Test patch:
diff --git a/master/buildbot/test/unit/process/test_log.py b/master/buildbot/test/unit/process/test_log.py
--- a/master/buildbot/test/unit/process/test_log.py
+++ b/master/buildbot/test/unit/process/test_log.py
@@ -68,7 +68,7 @@ def test_updates_plain(self):
         _log.addContent('hello ')
         _log.addContent('cruel ')
         _log.addContent('world\nthis is a second line')  # unfinished
-        _log.finish()
+        yield _log.finish()
 
         self.assertEqual(self.master.data.updates.logs[_log.logid], {
             'content': ['hello\n', 'hello cruel world\n',
@@ -83,7 +83,7 @@ def test_updates_different_encoding(self):
         _log = yield self.makeLog('t', logEncoding='latin-1')
         # 0xa2 is latin-1 encoding for CENT SIGN
         _log.addContent('$ and \xa2\n')
-        _log.finish()
+        yield _log.finish()
 
         self.assertEqual(self.master.data.updates.logs[_log.logid]['content'],
                          ['$ and \N{CENT SIGN}\n'])
@@ -92,7 +92,7 @@ def test_updates_different_encoding(self):
     def test_updates_unicode_input(self):
         _log = yield self.makeLog('t', logEncoding='something-invalid')
         _log.addContent('\N{SNOWMAN}\n')
-        _log.finish()
+        yield _log.finish()
 
         self.assertEqual(self.master.data.updates.logs[_log.logid]['content'],
                          ['\N{SNOWMAN}\n'])
@@ -171,7 +171,7 @@ def test_updates_stream(self):
         _log.addStderr('oh noes!\n')
         _log.addStdout('cruel world\n')
         _log.addStderr('bad things!')  # unfinished
-        _log.finish()
+        yield _log.finish()
 
         self.assertEqual(self.master.data.updates.logs[_log.logid], {
             'content': ['ohello\n', 'eoh noes!\n', 'ohello cruel world\n',