bwhite/hadoopy — Issue #45: If setting jobconf mapred.child.java.opts too high, hard to diagnose failure occurs
Author: bwhite
Created: 2011-12-29 02:35:54+00:00
State: open

As the java heap space can't be created, it fails in a really early stage, even before it starts hadoop streaming. This can be forced by doing
jobconfs=['mapred.child.java.opts=-Xmx1024M'] where 1024 can be upped to anything larger than the memory available (might need to turn off swap).

Here is what the user log looks like. This is the only way I was able to diagnose it.

root@ip-10-102-69-20:/var/log/hadoop/userlogs/job_201112281934_0017/attempt_201112281934_0017_r_000016_0# cat *
Could not create the Java virtual machine.
Error occurred during initialization of VM
Could not reserve enough space for object heap