Updated openstack/openstack
Project: openstack/heat  586395b1320a7dcb548f1cdf4364699c46d821bc

Retry resource create until success

On resource create, if a ResourceInFailure is raised then
repeated attempts are made to delete and recreate the resource
until success or a different error state is achieved.

Likewise, the prepare-retry deletes will be retried until
ResourceInFailure is not raised.

An exponentially increasing delay with jitter is introduced
between each create attempt, and attempts continue up to the configured
action_retry_limit or stack operation timeout.

Likewise An exponentially increasing delay with jitter is introduced
between each prepre-retry delete attempt, and delete attempts
continue up to the configured action_retry_limit or stack operation
timeout. The delete attempt count is reset to zero whenever a create
attempt has been performed.

Creates that result from an UpdateReplace will also go
through this path, so this is also helps some stack update scenarios.

This change is aimed at being part of an interim solution to making
heat resilient to transient cloud failures. Convergence is the
permanent solution however there may be benefits to the convergence
implementation from this interim effort.

Currently retry is only attempted on ResourceInFailure. Eventually
client plugins can indicate whether a given exception should lead
to a retry attempt (such as connection errors, some 500s).

Partial-Blueprint: retry-failed-api-calls
Change-Id: I07c3301349bcd24096f3cafbb6d82c43bccb93de