boto/botocore — pull request #74 by jamesls (2013-07-02 01:14:47+00:00)
Title: Add retry logic into botocore

This pull request adds support for retry logic into botocore.
Note that this also pulls in code from https://github.com/boto/botocore/pull/69
so the diff for #69 shows up here as well.

Essentially there are 4 parts to this pull request:

* The retry logic itself. This is in `retryhandler.py` and is decoupled from
anything session/service related. They all take a request and an attempt
count and figure out whether or not a retry is needed.
* The actual retry configuration. This is service specific and defined in
_retry.json. I've updated our translation module (`translate.py`) to inject
these into the service specific json models. The reason to put everything
into a single _retry.json is so that we can keep all the retry logic in a
single place, as well as reuse common definitions to avoid duplication,
(there's a minimal implementation of `$ref` to refer to common definitions).
* The `create_*` factory functions in `retryhandler.py` that takes the retry
config from the model and constructs the appropriate objects that comprise
the retry logic.
* The plumbing to hook this all into the event system. This reads in the retry
configs and calls the appropriate factory functions and then registers the
necessary events with the created retry handler objects. The retry mechanism
is all based on events (we fire an event that asks any handlers if we should
retry, and based on the responses we get back we do or don't retry).

## General Retry Mechanism

This is an overview of how the retry process works.

When a service is created from the `Session` a `service-created.` event is
fired. We register a handler for this event that injects the retry logic as
needed. This is done so that retry registration is lazy. We only register
event handlers for retry logic for a service when a service is created from the
session.

The retry registration registers events for `needs-retry.` and, if the
service has operation specific retry logic, for
`needs-retry..`.

Now, when an operation is called, we fire a `needs-retry..`
event to see if we need to retry the request. This all happens in the
`endpoint.py` classes. Handlers return the amount of time to sleep in the
event of a retry, otherwise they return None. If that happens, we sleep and
then resend the HTTP request.

The one thing that's not implemented is retries for socket level errors. The
model can handle these types of errors (as a new key to `applies_when`) but it
just hasn't been implemented yet (though I plan on implementing this).

Also, for ease of merging, I have not included the rebuilt models for the retry config.
Once this is merged we can run the `scripts/buildall` script to inject the retry config,
or I can add it to this pull request if desired.