bwhite/hadoopy — Issue #49: OOM killer on the Java process can cause the job may complete silently
Author: bwhite
Created: 2012-01-06 14:08:58+00:00
State: open

I have noticed this bug occasionally and it is unacceptable. It will only happen if you don't have enough memory on your machine. Linux has an algorithm for killing tasks when the system is low on memory, if it picks the Java process for hadoop streaming (Hadoopy isn't responsible for it) it may complete without actually finishing the work with no perceptible error. I'm not sure which java process needs to be killed for this to occur. I'll look into this more and try to post a minimum example.