Repository: mitmproxy/mitmproxy

Issue:
mitmdump jumps to 100% CPU when parent process exits
#### Problem Description

It took me two days to make this reproduce in isolation. I hope someone with Python skills can figure out what is happening here. Depending on what the root cause is this might not even be related to my funny architecture.

I'm spawning `mitmdump` from Node.js. If the node process exits mitmdump will be re-assigned to become a child of `systemd` (some unix wizardry). It will then immediately jump to 100% CPU and stay there. This _only_ happens when an addon is using at least one network event (go figure...). E.g. I'm using `client_connected` (works with `clientconnect` on v6 as well). If the addon is only using sth. like `running` the bug does not occur. Even better: if the addon originally only has "running" nothing bad happens. But if I then add a `client_connected` and save the file (and the addon is automatically reloaded) it will instantly jump to 100% CPU.

My guess is that it might be related to stdout and the switcheroo with the parent process? In my actual architecture the mitmdump process will poll the parent via gRPC every second and shutdown if it's gone. But the 100% CPU prevents that.

Update: while trying to write down the exact steps it turns out this might only reproduce via local venv and and not if you download the binary. I'm not sure, it's confusing. I'm confused. But I have video proof, so I'm not completely insane.

#### Steps to reproduce the behavior:

index.js

```js
const path = require('path');
const { spawn } = require('child_process');

function handleStdOut(data) {
  console.log(`mitmdump stdout: ${data}`);
}

function handleStdError(data) {
  console.error(`mitmdump stderr: ${data}`);
}

function handleExit(code) {
  console.log(`mitm process exited with code ${code}`);
}

const mitm = spawn(
  // Adjust this path
  '/home/alex/Projects/Bandsalat/src/forks/mitmproxy/venv/bin/mitmdump',
  ['--quiet', '--set', 'connection_strategy=lazy', '--scripts', 'addon.py'],
  {
    detached: true,
    windowsHide: true,
    env: {
      PYTHONUNBUFFERED: '1',
    },
  }
);

console.log(mitm.spawnargs);

mitm.unref();
mitm.on('exit', handleExit);
mitm.stdout.on('data', handleStdOut);
mitm.stderr.on('data', handleStdError);
```
addon.py

```py
class MyAddon:
  def running(self):
    print('running')

  def client_connected(self, client):
    print('client_connected')

addons = [
  MyAddon()
]
```

1. I'm on Ubuntu
2. Adjust index.js to point to your local mitmproxy git venv
3. Launch `node index.js` (Node 14 or 16 work both for me)
4. Now open Chromium with mitmproxy configured. You don't need to enter any URL, Chromium will phone home anyway.
5. Keep Chromium open and ctrl+c the node process
6. Observe your fan getting louder and `top` showing mitmdump at 100% CPU

https://user-images.githubusercontent.com/679144/124594746-740a7080-de60-11eb-9ffb-a5fc4b3ba24a.mp4

#### System Information

Happens with both v6 and HEAD.

```
Mitmproxy: 7.0.0.dev (+492, commit af27556)
Python:    3.8.10
OpenSSL:   OpenSSL 1.1.1i  8 Dec 2020
Platform:  Linux-5.8.0-59-generic-x86_64-with-glibc2.29
```



Discussion:
Current working theory: the addon `print` raises a broken pipe and the mitmdump error handling wants to log that broken pipe error to stderr, causing another broken pipe. And this somehow causes an infinite recursion of trying to write to stderr? Maybe this is just a matter of `if e.errno != errno.EPIPE:` in the right spot?
Fundamentally this is my problem and not a bug in mitmdump.
> Fundamentally this is my problem and not a bug in mitmdump.

I've changed my mind :smile: . While the bug only surfaces in the way I use mitmdump, I cannot fix this on my end. When the parent process unexpectedly closes (without cleaning up, i.e. killing the mitmdump child), Unix will re-parent mitmdump to the next parent. But this breaks the stdin/stdout pipes I had. This will also cause the GRPC service to go away (which the parent was running). That means the next time my addon tries to call the service it will crash (yes, I _could_ catch that). And then it tries to write to stderr, but it can't, it's gone. And I _think_ it will try so over and over and over. Which will then cause mitmdump to run at 100% CPU. 

Does my explanation make sense? I've seen my mitmdump at 100% CPU (with an exited parent) multiple times in the past days and I'd love to fix it.

Where would be the right place to check for `if e.errno != errno.EPIPE:`? Does that sound like a good fix? Exiting the process when we can't write to stderr? I think everyone running mitmdump as a child process would benefit from that.
I've traced it down to

https://github.com/mitmproxy/mitmproxy/blob/f85c64401b0a9d1f60f85eaaf810fef1c46c4e08/mitmproxy/addonmanager.py#L50-L54

plus

https://github.com/mitmproxy/mitmproxy/blob/f85c64401b0a9d1f60f85eaaf810fef1c46c4e08/mitmproxy/log.py#L70-L74

If I add this before the ctx.log.error call

```py
file1 = open('/tmp/mitm/e.txt', 'a')
file1.write("Addon error: %s" % "".join(
    traceback.format_exception(etype, value, tb)
))
file1.close()
```

I get an endless growing `e.txt` with

```
Addon error: Traceback (most recent call last):
  File "addon.py", line 6, in client_connected
    print('client_connected')
BrokenPipeError: [Errno 32] Broken pipe
Addon error: Traceback (most recent call last):
  File "addon.py", line 6, in client_connected
    print('client_connected')
BrokenPipeError: [Errno 32] Broken pipe
Addon error: Traceback (most recent call last):
  File "addon.py", line 6, in client_connected
    print('client_connected')
BrokenPipeError: [Errno 32] Broken pipe
Addon error: Traceback (most recent call last):
  File "addon.py", line 6, in client_connected
    print('client_connected')
BrokenPipeError: [Errno 32] Broken pipe
Addon error: Traceback (most recent call last):
  File "/home/alex/Projects/mitmproxy/mitmproxy/addons/termlog.py", line 22, in add_log
    click.secho(
  File "/home/alex/Projects/mitmproxy/venv/lib/python3.8/site-packages/click/termui.py", line 659, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File "/home/alex/Projects/mitmproxy/venv/lib/python3.8/site-packages/click/utils.py", line 298, in echo
    file.write(out)  # type: ignore
BrokenPipeError: [Errno 32] Broken pipe
Addon error: Traceback (most recent call last):
  File "/home/alex/Projects/mitmproxy/mitmproxy/addons/termlog.py", line 22, in add_log
    click.secho(
  File "/home/alex/Projects/mitmproxy/venv/lib/python3.8/site-packages/click/termui.py", line 659, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File "/home/alex/Projects/mitmproxy/venv/lib/python3.8/site-packages/click/utils.py", line 298, in echo
    file.write(out)  # type: ignore
BrokenPipeError: [Errno 32] Broken pipe
Addon error: Traceback (most recent call last):
  File "/home/alex/Projects/mitmproxy/mitmproxy/addons/termlog.py", line 22, in add_log
    click.secho(
  File "/home/alex/Projects/mitmproxy/venv/lib/python3.8/site-packages/click/termui.py", line 659, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File "/home/alex/Projects/mitmproxy/venv/lib/python3.8/site-packages/click/utils.py", line 298, in echo
    file.write(out)  # type: ignore
BrokenPipeError: [Errno 32] Broken pipe
```

So this 100% confirms that when the parent process is gone, any error that the addon causes will cause an endless loop. The repro uses a simple `print` which itself will fail with EPIPE, but any error in the addon works. Because mitmproxy will try to write to a broken stderr.

Now the question is what is the proper fix? Ignoring all `EPIPE` doesn't sound like a good idea, because you might have other pipes that break in your addon. I don't know how the scheduling of the whole logging thing works, can't we wrap the final print in a `try`?

Edit: nvm, just a `try` is not enough. If logging the error fails the process needs to go down.
Thanks for looking into this! I think `TermLog.add_log` should just catch any OSErrors and then `sys.exit()` with a distinguishable exit code. As with streaming flows to disk, if we cannot write logfiles anymore we want to go down hard.

Resolving patch:
diff --git a/mitmproxy/addons/termlog.py b/mitmproxy/addons/termlog.py
--- a/mitmproxy/addons/termlog.py
+++ b/mitmproxy/addons/termlog.py
@@ -51,7 +51,9 @@ def __init__(
         self.formatter = log.MitmFormatter(self.has_vt_codes)
 
     def emit(self, record: logging.LogRecord) -> None:
-        print(
-            self.format(record),
-            file=self.file
-        )
+        try:
+            print(self.format(record), file=self.file)
+        except OSError:
+            # We cannot print, exit immediately.
+            # See https://github.com/mitmproxy/mitmproxy/issues/4669
+            sys.exit(1)


Test patch:
diff --git a/test/mitmproxy/addons/test_termlog.py b/test/mitmproxy/addons/test_termlog.py
--- a/test/mitmproxy/addons/test_termlog.py
+++ b/test/mitmproxy/addons/test_termlog.py
@@ -1,4 +1,5 @@
 import asyncio
+import builtins
 import io
 import logging
 
@@ -55,3 +56,20 @@ async def test_styling(monkeypatch) -> None:
 
     assert "\x1b[33mhello\x1b[0m" in f.getvalue()
     t.done()
+
+
+async def test_cannot_print(monkeypatch) -> None:
+    def _raise(*args, **kwargs):
+        raise OSError
+
+    monkeypatch.setattr(builtins, "print", _raise)
+
+    t = termlog.TermLog()
+    with taddons.context(t) as tctx:
+        tctx.configure(t)
+        with pytest.raises(SystemExit) as exc_info:
+            logging.info("Should not log this, but raise instead")
+
+        assert exc_info.value.args[0] == 1
+
+    t.done()