daitss/core — Issue #617: apache graceful interrupts production
Author: avatar382
Created: 2012-01-20 16:45:10+00:00
State: open

Apache graceful restarts should not disrupt production -- and yet they are. text from PRB 15508:

==============

Yesterday, the DAITSS config file was updated and an apache graceful
was issued to read in the changes.

However, the apache graceful restart caused xymon to report that two
services were down, one of which was silo, interrupting production
by
causing an error.

I did some experimentation, and issued another graceful, and xymon
reported that two different DAITSS services were down. On a third
graceful, xymon reported nothing.

Any ideas on what could be causing this behavior? A graceful restart
in apache shouldn't interrupt any processes actively serving
requests.
----------------
Xymon polls every 5(?) minutes, so if the graceful happens completely
between polling periods, then there's no alert. Otherwise, what ever
it
catches in the race condition between polling and restarting are
considered errors.

Why does a Xymon error cause a production interruption? It may be that
you want to tune your system to accept brief interruptions, or else
squelch xymon when you restart apache.
Allan
-----------------
Have you changed your scripts to allow for temporary outages like httpd
graceful restarts?
Just wondering whether we need to leave this ticket open. Allan
-----------------

Since production is sometimes interrupted when a graceful is issued, it seems as though graceful is actually disruptive and the xymon errors are accurate -- graceful actually causes DAITSS services to go down momentarily.

My understanding on how graceful works is that processes actively serving a request are allowed to finish, but new requests are served with a brand new process. I think that the action item is to figure out why graceful is disrupting DAITSS, even though it shouldn't be in theory. That should solve both the production being interrupted and the xymon errors. This is a task for daitss programmers, so this ticket can be closed. I'm opening a daitss github ticket for this issue.