axemblr/axemblr-provisionr — pull request #143 by cimi (2013-02-12 13:34:36+00:00)
Title: Idempotent request error when a pool is not terminated successfully

Here's the following scenario:

* We try and create a new pool of machines on Amazon:
````
provisionr:create --id amazon --key mypool --size 2 --hardware-type m1.small
````

* There's a chance it fails due to an AWS internal error:
````
Exception in thread "pool-10-thread-1" Status Code: 500, AWS Service: AmazonEC2, AWS Request ID: 83a647e2-8e93-4edc-8328-0f000ee83721, AWS Error Code: InternalError, AWS Error Message: An internal error has occurred
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:556)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:289)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:5850)
at com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:2546)
at com.axemblr.provisionr.amazon.activities.RunOnDemandInstances.execute(RunOnDemandInstances.java:54)
[...]
````
* We then try and terminate the pool:
`provisionr:destroy --key mypool`

* This can fail due to having a null process variable for instance Ids (see #142). You can then get this error message:
````
Exception in thread "pool-10-thread-26" java.lang.NullPointerException: process variable '{}' not found [instanceIds]
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:235)
at com.axemblr.provisionr.amazon.activities.AllInstancesMatchPredicate.execute(AllInstancesMatchPredicate.java:61)
at com.axemblr.provisionr.amazon.activities.AmazonActivity.execute(AmazonActivity.java:61)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
````
* After a while, if you run `provisionr:destroy --key mypool` again you'll get:
````
Error executing command: No executions found waiting for signal 'terminatePoolEvent' on process mypool
````
* `provisionr:pools` still shows `mypool`
* If we try to create a new pool, with a different key and a different hardware type, we get an idempotency error:
````
provisionr:create --id amazon --key newpool --size 2 --hardware-type m1.micro
````
````
Exception in thread "pool-10-thread-32" Status Code: 400, AWS Service: AmazonEC2, AWS Request ID: 1dca7317-cf89-4e0b-aaec-596b533df1d1, AWS Error Code: IdempotentParameterMismatch, AWS Error Message: Arguments on this idempotent request are inconsistent with arguments used in previous request(s).
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:556)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:289)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:5850)
at com.amazonaws.services.ec2.AmazonEC2Client.runInstances(AmazonEC2Client.java:2546)
at com.axemblr.provisionr.amazon.activities.RunOnDemandInstances.execute(RunOnDemandInstances.java:54)
````
* `provisionr:pools` shows `mypool` and `newpool`, neither of them have any instances running.

From my understanting the create command should make a new process with a distinct clientToken sent to Amazon, the idempotency error is strange here. The destroy exception can be fixed by adding a check to `AllInstancesMatchPredicate`.