mojombo/god — Issue #89: Missing proc_exit event
Author: matthewcalebsmith
Created: 2012-03-05 16:44:16+00:00
State: open

I have used a different version of this God config file in the past an it worked like a charm. However, now will start unicorn and check memory usage but does not seem to notice when the process exists.

I am using god to monitor several other resque processes on the same server using events and things seem to be working well. I am running Ubuntu 10.04.4 LTS. Linux 2.6.32-38-generic-pae #83-Ubuntu SMP Wed Jan 4 12:11:13 UTC 2012 i686 GNU/Linux

Where am I going wrong? (Config files below.)

```` bash
$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god -D --log-level debug
[sudo] password for deploy:
I [2012-03-05 08:07:20] INFO: Syslog enabled.
I [2012-03-05 08:07:20] INFO: Using pid file directory: /var/run/god
I [2012-03-05 08:07:20] INFO: Started on drbunix:///tmp/god.17165.sock
I [2012-03-05 08:07:32] INFO: unicorn-master Loaded config
I [2012-03-05 08:07:32] INFO: unicorn-master move 'unmonitored' to 'init'
D [2012-03-05 08:07:32] DEBUG: driver schedule # in 0 seconds
I [2012-03-05 08:07:32] INFO: unicorn-master moved 'unmonitored' to 'init'
I [2012-03-05 08:07:32] INFO: unicorn-master [trigger] process is not running (ProcessRunning)
D [2012-03-05 08:07:32] DEBUG: unicorn-master ProcessRunning [false] {true=>:up, false=>:start}
I [2012-03-05 08:07:32] INFO: unicorn-master move 'init' to 'start'
I [2012-03-05 08:07:32] INFO: unicorn-master before_start: no pid file to delete (CleanPidFile)
I [2012-03-05 08:07:32] INFO: unicorn-master start: /home/deploy/.rbenv/versions/1.9.3-p0/bin/unicorn -D -c /home/deploy/apps/project/current/config/unicorn.rb
D [2012-03-05 08:07:35] DEBUG: driver schedule # in 0 seconds
D [2012-03-05 08:07:35] DEBUG: driver schedule # in 0 seconds
I [2012-03-05 08:07:35] INFO: unicorn-master moved 'init' to 'start'
I [2012-03-05 08:07:35] INFO: unicorn-master [trigger] process is running (ProcessRunning)
D [2012-03-05 08:07:35] DEBUG: unicorn-master ProcessRunning [true] {true=>:up}
I [2012-03-05 08:07:35] INFO: unicorn-master move 'start' to 'up'
I [2012-03-05 08:07:35] INFO: unicorn-master registered 'proc_exit' event for pid 3155
D [2012-03-05 08:07:35] DEBUG: driver schedule # in 0 seconds
D [2012-03-05 08:07:35] DEBUG: driver schedule # in 0 seconds
I [2012-03-05 08:07:35] INFO: unicorn-master moved 'start' to 'up'
I [2012-03-05 08:07:35] INFO: unicorn-master [ok] memory within bounds [7468kb] (MemoryUsage)
D [2012-03-05 08:07:35] DEBUG: unicorn-master MemoryUsage [false] {true=>:restart}
D [2012-03-05 08:07:35] DEBUG: driver schedule # in 30 seconds
I [2012-03-05 08:07:35] INFO: unicorn-master [ok] cpu within bounds [0.0%%] (CpuUsage)
D [2012-03-05 08:07:35] DEBUG: unicorn-master CpuUsage [false] {true=>:restart}
D [2012-03-05 08:07:35] DEBUG: driver schedule # in 30 seconds
````
Then `kill 3155`

```` bash
I [2012-03-05 08:16:05] INFO: unicorn-master [ok] memory within bounds [7476kb, 7476kb, 0kb] (MemoryUsage)
D [2012-03-05 08:16:05] DEBUG: unicorn-master MemoryUsage [false] {true=>:restart}
D [2012-03-05 08:16:05] DEBUG: driver schedule # in 30 seconds
I [2012-03-05 08:16:05] INFO: unicorn-master [ok] cpu within bounds [0.0022087244616234127%%, 0.002071465561885034%%, 0%%] (CpuUsage)
D [2012-03-05 08:16:05] DEBUG: unicorn-master CpuUsage [false] {true=>:restart}
D [2012-03-05 08:16:05] DEBUG: driver schedule # in 30 seconds

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god status
unicorn:
unicorn-master: up
````

As you can see god still thinks the process is up, even though it shows 0 for memory usage. The pid file does exist in the specified location when the process is running. Below is the out put from -V and check, along with the God and Unicorn config files.

``` bash
$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god -V
Version: 0.12.1
Polls: enabled
Events: netlink

$ sudo /home/deploy/.rbenv/versions/1.9.3-p0/bin/god check
using event system: netlink
starting event handler
forking off new process
forked process with pid = 3227
killing process
[ok] process exit event received
````
God config file.

```` ruby
God.watch do |w|
w.name = "unicorn-master"
w.group = "unicorn"

w.dir = "/home/deploy/apps/project/current"
w.interval = 30.seconds
w.env = { 'RAILS_ENV' => 'production' }
w.start = "/home/deploy/.rbenv/versions/1.9.3-p0/bin/unicorn -D -c /home/deploy/apps/project/current/config/unicorn.rb"
w.pid_file = "/home/deploy/apps/project/shared/pids/unicorn.pid"

w.uid = "deploy"
w.gid = "deploy"

w.behavior(:clean_pid_file)

# With events
w.transition(:init, { true => :up, false => :start }) do |on|
on.condition(:process_running) { |c| c.running = true }
end
w.transition([:start, :restart], :up) do |on|
on.condition(:process_running) { |c| c.running = true }
on.condition(:tries) { |c| c.times = 5; c.transition = :start }
end
w.transition(:up, :start) do |on|
on.condition(:process_exits)
end
w.transition(:up, :restart) do |on|
on.condition(:memory_usage) { |c| c.above = 24.megabytes; c.times = [2,3] }
on.condition(:cpu_usage) { |c| c.above = 8.percent; c.times = [2,3] }
end

w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minutes
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
end
````

Unicorn (ver 4.2.0) config file

```` ruby
worker_processes 2
working_directory "/home/deploy/apps/project/current"
listen "/tmp/project.socket", :backlog => 64
timeout 30
user 'deploy', 'deploy'
pid "/home/deploy/apps/project/shared/pids/unicorn.pid"
stderr_path "/home/deploy/apps/project/shared/log/unicorn.stderr.log"
stdout_path "/home/deploy/apps/project/shared/log/unicorn.stdout.log"
before_fork do |server, worker|
defined?(ActiveRecord::Base) and
ActiveRecord::Base.connection.disconnect!
old_pid = "#{server.config[:pid]}.oldbin"
if old_pid != server.pid
begin
sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
Process.kill(sig, File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
end
end
sleep 1 # Prevent signals from being lost when busy
end
after_fork do |server, worker|
defined?(ActiveRecord::Base) and
ActiveRecord::Base.establish_connection
end