fix flaky test
implement memory limiting

Writes memory.limit_in_bytes and memory.memsw.limit_in_bytes in the
container's cgroup. Spawns the oom notifier process, and stops the container
when the oom notifier exits 0.