GoogleCloudPlatform/kubernetes — pull request #1962 by dchen1107 (2014-10-22 18:55:39+00:00)
Title: Increase memory limit of frontend-controller in our guestbook example.

I ran guestbook example against the latest containervm image, which has memory cgroup enabled by default, and noticed that frontend job is run into exited, restart, then exited again loop forever. Docker container logs and daemon logs show nothing about why it is killed. Kubelet never kill those containers.

I spent quite sometime on debugging this since there were several open issues related to this kind of failure and replicaController related bad loop. At the end, I suspected I might have a sys oom issue. It turns out it is memory-limit-exceeded oom killed:

Oct 21 21:08:58 kubernetes-minion-2 kernel: [75254.207495] supervisord invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0
Oct 21 21:08:58 kubernetes-minion-2 kernel: [75254.229995] [] ? oom_kill_process+0x28a/0x3e0
Oct 21 21:08:58 kubernetes-minion-2 kernel: [75254.231977] [] ? mem_cgroup_oom_synchronize+0x50f/0x580
Oct 21 21:08:58 kubernetes-minion-2 kernel: [75254.262210] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name

Vish has a pending PR to docker: docker/docker/pull/8479 which converts non-sense termination reason: "Exited (137) ... " into proper oom-killed message. Once this is merged to next docker release, we wouldn't waste our time on debugging this kind of issue, also upper layer components can do better management job based on the proper death reason. @bgrant0607 @vishh

Meanwhile, I did a hell of a lot tests against containervm image, and this is the only issue blocking us to move kubernetes on the top of containervm image (There are some small issues can be addressed later).

@jbeda @brendanburns