discoproject/disco — Issue #300: Tracking node memory usage in job scheduler
Author: pmundkur
Created: 2012-02-15 22:37:17+00:00
State: open

Although we use ulimit to limit an individual tasks memory usage, it is possible for several large tasks to consume all the memory on a node, without consuming all the slots on the node. If we don't leave some memory headroom on the node, strange failures occur, both in tasks as well as in disco/ddfs.