mozilla/fxa-auth-server — pull request #581 by dannycoates (2014-02-24 03:05:20+00:00)
Title: Some MySQL connection improvements

- added sql connection stat logging
- one retry on connection errors
- prevent pools from getting removed

Setting `removeNodeErrorCount = Infinity` seems better than listening to the 'remove' event because there's a slight gap in service between the pool being removed and re-added where things can get weird or stuck. Our usage of the pools with RDS isn't really what this was meant for, which seems to be direct connections to a cluster of db servers not behind a load balancer.

This does a fairly good job of handling connection errors. I ran this in a test environment with mysql set to 4 max connections and the app configured for 20 total against a load test of 8 concurrent clients. There were **2** 503 errors returned to the clients, but the connection retry was able to recover from over 10k `ER_CON_COUNT_ERROR` mysql errors (from a run of ~26k requests). Obviously the right thing to do is never set the app connection limit higher than the database, but at least we continue to service requests when we're over.

We can still do with a nice refactor of the query logic, transaction handling, etc.