authdns: make the Ganglia plugin more resilient

The graphs have a lot of spikes currently. The theory is that this
happens because the plugin occasionally returns 0 for values, making
Ganglia misdetect the rate of change.

The first problem is, the plugin has an issue right now that it does an
HTTP request for every metric, which is kind of silly (this is due to a
limitation of Ganglia's python module). This could aggravate the problem
by making a potential fetching failure appear much more often than it
otherwise would. So, we add a caching layer for this, fetching metrics
at most once a second (which is how often gdnsd updates stats anyway).

Since we now do requests much more seldomly, we can afford to repeat
them if they failed for whatever reason. Two attempts should limit the
occurence of an issue much more.

Finally, add a grace period of 15s (the plugin's refresh time) when
we'll serve stale data even if all attempts to fetch failed.

Change-Id: I94bbcf6db6f6389cd7fdbc500b7a16441b448dc3
Merge "authdns: make the Ganglia plugin more resilient" into production