sakaiproject/Hilary — pull request #430 by mrvisser (2013-03-22 17:28:15+00:00)
Title: Shutdown lifecycle

I'd like to get a better idea of what happens when the node process receives a kill / "shutdown" signal. Specifically:

a) What signal *should* be sent to gracefully shutdown
b) Is there anything more we need to do to ensure that HTTP requests, internal node event processing and MQ event processing safely completes before the process exits.

There is a good write-up here about some ideas:

http://blog.argteam.com/coding/hardening-node-js-for-production-part-3-zero-downtime-deployments-with-nginx/

However it leaves more questions in my mind:

* After `express.close()` is completed, what about internal node events that were fired and are in the process of performing IO (e.g., firing our activity and search tasks)?
* We probably need to do something similar to `express.close()` with RabbitMQ so we allow ourselves to stop receiving new tasks and safely finish processing the ones we have
** Then what about internal node events resulting from this (e.g., notifications tasks)?
* I have read that when there are no more events in the node queue, the nodejs process will exit on its own. Is this the key to allowing the asynchronous internal events to finish, by simply not executing `process.exit()` after the Express / MQ servers stop receiving requests? If so, why does every "graceful shutdown" solution involve invoking `process.exit()` immediately after the express server closes, rather than allowing the queue to empty?