infochimps-labs/ironfan-pantry — pull request #17 by mrflip (2013-01-12 20:48:15+00:00)
Title: Hadoop too-small stack size killed tasks on Ubuntu 12 / Oracle Java

Finally figured out that Hadoop was dying because (according to oracle java, but not the prior sun java):

```
The stack size specified is too small, Specify at least 160k
Could not create the Java virtual machine.
```

This wasn't appearing in the jobtracker's log view; I had to look in var/log/hadoop/userlogs to see the JVM's epitaph. Note that that directory is read-protected, so you will have to sudo to see it. Error in jobtracker logs looked like

```
2013-01-12 20:30:47,605 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201301120857_0005_m_000001_3' to tip task_201301120857_0005_m_000001, for tracker 'tracker_ip-10-113-42-223.ec2.internal:localhost/127.0.0.1:54742'
2013-01-12 20:30:47,605 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201301120857_0005_m_000001_2'
2013-01-12 20:30:47,908 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201301120857_0005_m_000001_3: java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
```