Revert "Default to a ratelimit of 2/second for API calls"

This reverts commit 400284119dfe8118ba020f4795c448833a9c3977.

This can be changed in configuration.  1/second is a sane default.

Change-Id: I77db36badd8d82667cfab6974ff101607ffbad21
Revert "Provide diagnostics when task rate limiting."

This reverts commit a01956ed70132029c4770ba4e7a3aafa4d14b4d1.

This outputs a huge amount of not very useful information.

Change-Id: Ie72a207ced8bb64ae1a01c88ca396f9df633e79c
Cleanup nodes in state DELETE immediately.

The 900 second delay before deleting them from periodic cleanup just
causes excessive waste when we've restarted nodepool.

To avoid concurrent sqlalchemy access (particularly mutations) to node
objects conditions with other per-node operations, this delegates all
deletions to periodic cleanup.

This is done because with the high latency on API command submission
due to low rate limits with providers (e.g. 2 requests/second) it's
likely that a node deleted during busy periods will not have it's API
request *submitted* for up to 5 minutes. As such, the 5 minute cleanup
thread will actually submit the API requests with little or no
increase in latency. And of course that frequency could be increased,
permitting faster cleanup in low-load situations.

Change-Id: Iad7004b0d3389df1e1b687209fd5f6ed27b03239
Log how long nodes have been in DELETE state.

We have providers that are terrible at deleting.

Change-Id: Ia1e5466b607010e7f21f321775aa25ce9543f046
Consolidate duplicate logging messages.

Change-Id: If66df6f9b42cbfb00f8ed2d81e573a550c3e6db3
Make cleanupServer optionally nonblocking.

cleanupServer is used from both the CLI and the daemon, but in the
daemon we don't want to block - a single slow to delete server should
not block deleting all other servers. Instead we want to pass over all
the servers we have pending deletion, submitting them for deletion,
and after that loop back and check that they were in fact deleted. Or
not.

This change also fixes a minor bug where calling cleanupServer on a
server that was concurrently deleted from the provider (e.g. by the
CLI) would result in a traceback and deferral to the next loop of
periodic cleanup before the node record was deleted.

Change-Id: If40930238d82c1103b83fe7f46ef6f6e86efd624
Split out the logic for deleting a nodedb node.

In order to move to scatter-gather, we need to be able to run just the
tail of deleteNode independently.

Change-Id: If03753e98b423552c8fd6be4d2f5b6b543f3a1af
Merge "Cleanup nodes in state DELETE immediately."
Merge "Log how long nodes have been in DELETE state."
Merge "Consolidate duplicate logging messages."
Merge "Make cleanupServer optionally nonblocking."
Merge "Split out the logic for deleting a nodedb node."
Use the nonblocking cleanupServer.

In this patch we now change from doing one server deletion entirely at
a time to instead doing all the deletion requests and then all the
was-it-deleted checks. This gives the cloud all our requested work
at once.

Note that we still do lookups from cleanupServer, future work might
tune this more to minimise them. The reason we need to do a scan after
submitting all the deletions is to avoid waiting the whole poll period
between periodic cleanups before we notice a not-quite-immediate
delete actually happened.

Change-Id: I8f52b24ed911c1e6b8e04b8fe561100f8552fee6
Teach periodicCleanup how to do one provider.

This permits (once the cron changes are wired in) per-provider
threads, rather than a single global thread.

Change-Id: Ie666d03ee9493373de494c3e6664e4e7bf144dcb
Move cron loading below provider loading.

This is preparation for making the cleanup cron job run per-provider
rather than globally.

Change-Id: I2cc0ec62b9779fa2500c6c67f014739dfec9205c
Move cron definition out of the inner loop.

This makes creating multiple Cron entries for one config entry
cleaner, which we need to do per-provider cleanups.

Change-Id: Ic5fe8a57fec7aaca644da43f4c88209fbc5488dd
Decouple cron names from config file names.

This permits creating multiple cron entries for one config entry,
which per-provider cleanups requires.

Change-Id: Ied9db4a512d8efef355e911d6e5630697d6c38c9
Run per-provider cleanup threads.

Change-Id: I4c1c54190a5e254dd946ef308a11bb81907bb16c
Include check in fake.yaml.

Anyone starting from fake.yaml should have this.

Change-Id: Ieaa8a075cfff5259fabdcdada71c04cddb53cf9f
Merge "Revert "Default to a ratelimit of 2/second for API calls""