Change:
The cluster I created on Friday (at head) was running over the weekend, and when I tried to talk to it this morning, I couldn't. SSH'ing into the master, I couldn't hit and Docker isn't running and neither is monit so what happened to monit? found in monit. My questions are: "Why?" and "Should we try to debug this further?" pointed out that a possible preventative measure is to have a cron job running that kicks monit if it's dead. I left the cluster running, so if anyone would like me to run commands or would like to dig in more, please let me know.
I haven't thought much about this, but it may be related to Docker being slow at restarting? As a side note, it seems like pretty poor design that monit dies if it fails to restart something.
Could you please take a look at this? Thanks!
you started the cluster on Friday, could you connect to apiserver using kubectl ? you see any errors in the file ?
cc
, the cluster functioned on Friday; I was running tests against it. (If you look at the monit log I pasted above, it appears it was functioning for quite a while.) don't see any errors there, but I can dump it here if it would help (it's quite long, obviously).
out-of-memory errors seen/logged around the time when monit failed to fork a child process. seems to consume 400m of memory. not sure why it is so high the fork did fail because of insufficient memory, but this is a guess and no logs to substantiate it
Just as an explicit followup, there were at least 8 out-of-memory errors that happened on that instance (a master) from the logs in —is this normal? I can provide several different logs or VM access for interested parties...
Regardless of the cause of the problem (which I agree would be interesting to discover) it seems like we need a cron job or something to make sure that monit is always running. can you look into how we can babysit our babysitter?
I will take a look this one today since there are several ones related to each other: , , ...

Discussion:
- group: root - mode: 644 <ins> /etc/monit/monit_watcher.sh: file.managed: - source: salt://monit/monit_watcher.sh - user: root - group: root - mode: 755 crontab -l | { cat; echo "* * * * * /etc/monit/monit_watcher.sh 2>&1 | logger"; } | crontab -: cmd.run: - unless: crontab -l | grep "* * * * * /etc/monit/monit_watcher.sh 2>&1 | logger" </ins> monit-service: service: - running

<ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script is invoked by crond every minute to check if monit is # up and oom protected. If down it restarts monit; otherwise, it exits # after applying oom_score_adj PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROGNAME=monit FULLPATH=/usr/bin/$PROGNAME pids=$(pidof $FULLPATH) if [[ "${pids}" == "" ]]; then service $PROGNAME start sleep 10 fi # Apply oom_score_adj: -901 to processes pids=$(pidof $FULLPATH) for pid in "${pids}"; do echo -901 > /proc/$pid/oom_score_adj done </ins>