saltstack/salt — pull request #10154 by abemusic (2014-02-03 20:32:24+00:00)
Title: Retry query on certain EC2 status codes

The `query` method in the EC2 driver is used all over the place and there are select response codes returned from AWS that could possibly be retried. I've identified the following that should be retried:

* RequestLimitExceeded - should already be handled by the increasing polling intervals
* InsufficientInstanceCapacity - EC2 can't give us any instances at this time
* InternalError - an unknown error on EC2 occurred, suggested to retry, but contact AWS if it continues
* Unavailable - EC2 is overloaded and can't serve the request
* InsufficientAddressCapacity - EC2 can't assign any more addresses
* InsufficientReservedInstanceCapacity - EC2 can't assign any more reserved instances

Right now, I've hardcoded the number of attempts per call to the query method to 5 and a sleep time of 2 seconds between attempts when one of the response codes in the above list is received. After the 5 attempts, it fails like normal with the error code, message, and response data.