nathanmarz/storm — pull request #753 by poros (2013-11-21 01:43:07+00:00)
Title: Handle EOF in multilang Python

This is to highlight an issue that I encountered using the Python multilang support.

If I run a Python topology in local mode and the Storm process who spawned all the ShellComponents dies abnormally (e.g. an exception is raised in the Python code, the process is killed), the Python processes will still keep on running.

In addition, I noticed that these “orphan” processes tend to increase unboundedly their memory consumption. I tracked down the memory leak and I found that, when the Storm parent process dies, the stdin stream the Python processes are reading from seems to be closed. For this reason, the sys.stdin.readline() call returns an empty string, that will be concatenated to the msg string. And since all this code is within a while true loop, it will quickly consume a lot of memory.

If we consider an empty string as the signal that something did go wrong with the parent process, the component should exit. This behaviour should solve the problem described in the discussion: https://groups.google.com/forum/#!searchin/storm-user/multilang/storm-user/X90d2q6dC0o/ZWT__mBnBKcJ

Since I am running a patched version of Storm, I cannot easily run all the tests, but I wanted to open this pull request at least to keep track of the issue and of the possible solution.