jenkinsci/ec2-plugin — pull request #27 by zzzeek (2012-11-17 21:34:55+00:00)
Title: fallback a manual or timeout-based terminate to stop if terminate fails

I'm observing that in some cases, Amazon EC2 will force-provision one of my instances that has termination protection applied to it as one of Jenkins new nodes. This leads to the undesirable state that the automated "termination" of this instance, as well as a manual attempt to terminate this instance, produces the exception "AWS Error Code: OperationNotPermitted, AWS Error Message: The instance 'i-5d694121' may not be terminated. Modify its 'disableApiTermination' instance attribute and try again."

As any failure like this regarding EC2 essentially means monetary charges to the owner's amazon account, it is critical that in the event of a failure like this, the instance is at least stopped; else automated builds start up instances which stay running and charging money indefinitely until someone manually intervenes.

This pull request reworks the terminate() method to fall back to a new stop() method in the event of any AmazonClientException. The implementation for stop() which was factored out of the idleTimeout() method, which now has a simple conditional of calling terminate() or stop() based on the value of stopOnTerminate. As is the case before, I'm running this on jenkins.sqlalchemy.org and if I encounter any issues with it, I'll update the pullreq over here.