Repository: google/clusterfuzz

Issue:
Infinite recursion/stack-overflow in stack_symbolizer.py:72 when starting new bot "thread"
A couple times when I've left the local development server and a bot running for a while I see this issue happening repeatedly. See
[bot-log.txt](https://github.com/google/clusterfuzz/files/2894457/bot-log.txt)
([server-log.txt](https://github.com/google/clusterfuzz/files/2894469/server-log.txt) posted for completeness). This happens over and over again. The bot instance can't make any progress.

This is currently happening on my machine. It might be hard to debug because starting a new bot doesn't seem to repro the issue right away. I think I will need to attach to the currently running process if I want to debug this issue without waiting days possibly to repro this again.

I don't really understand what is wrong with the line where this issue occurs. It seems correct to me.
But I can't say I'm surprised that something like stack_symbolizer.py:disable_buffering() goes wrong with multiprocessing involved :-\




Discussion:
I think I know what might be causing this issue.
disable_buffering does this hacky thing to force `sys.stdout` to be unbuffered if 'PYTHONUNBUFFERED' isn't set in the environment.
But we reset the environment in between tasks. So this will be done many times, which can lead to stack overflow, see what happens in the example below:
```
import os
import sys

def disable_buffering():
  """Make this process and child processes stdout unbuffered."""
  if not os.environ.get('PYTHONUNBUFFERED'):
    # Since sys.stdout is a C++ object, it's impossible to do
    # sys.stdout.write = lambda...
    sys.stdout = LineBuffered(sys.stdout)
    os.environ['PYTHONUNBUFFERED'] = 'x'

class LineBuffered(object):
  """Disable buffering on a file object."""

  def __init__(self, stream):
    self.stream = stream

  def write(self, data):
    self.stream.write(data)
    if '\n' in data:
      self.stream.flush()

  def __getattr__(self, attr):
    return getattr(self.stream, attr)

for _ in xrange(1000):
  disable_buffering()
  del os.environ['PYTHONUNBUFFERED']

sys.stdout.flush()
disable_buffering()
```

Then run it
```
  $ python example.py
      return getattr(self.stream, attr)
    File "/tmp/example.py", line 32, in __getattr__
      return getattr(self.stream, attr)
    File "/tmp/example.py", line 32, in __getattr__
      return getattr(self.stream, attr)
    File "/tmp/example.py", line 32, in __getattr__
      return getattr(self.stream, attr)
  RuntimeError: maximum recursion depth exceeded while calling a Python object
  ```
Because this issue comes up for a bot that sees <PYTHON MAX STACK DEPTH> crashes it is probably more prone to happening in development server when using examples like the heartbleed example. Fix incoming.

Resolving patch:
diff --git a/src/python/crash_analysis/stack_parsing/stack_symbolizer.py b/src/python/crash_analysis/stack_parsing/stack_symbolizer.py
--- a/src/python/crash_analysis/stack_parsing/stack_symbolizer.py
+++ b/src/python/crash_analysis/stack_parsing/stack_symbolizer.py
@@ -138,11 +138,14 @@ def chrome_dsym_hints(binary):
 
 def disable_buffering():
   """Make this process and child processes stdout unbuffered."""
-  if not os.environ.get('PYTHONUNBUFFERED'):
-    # Since sys.stdout is a C++ object, it's impossible to do
-    # sys.stdout.write = lambda...
+  os.environ['PYTHONUNBUFFERED'] = '1'
+
+  if not isinstance(sys.stdout, LineBuffered):
+    # Don't wrap sys.stdout if it is already wrapped.
+    # See https://github.com/google/clusterfuzz/issues/234 for why.
+    # Since sys.stdout is a C++ object, it's impossible to do sys.stdout.write =
+    # lambda...
     sys.stdout = LineBuffered(sys.stdout)
-    os.environ['PYTHONUNBUFFERED'] = 'x'
 
 
 def fix_filename(file_name):


Test patch: