airbnb/billow — pull request #10 by ktrice (2014-08-21 08:12:14+00:00)
Title: set the marker in the request to the marker from the result

There was a small bug in the code that executed a no-op when it was supposed to update a pagination key in the request.

The "marker" is a pagination key, specifying where the last request left off. The default maxRecords is 100, so before we had 100 RDS instances in us-east-1, this problem didn't occur. Once we went over 100 instances, billow would call describeDBInstances in a tight loop (possibly an infinite loop). Sometimes we may still have less than 100 RDS instances since it could be backups that put us over 100.

The new code updates the marker in the request to be the marker that was passed back in the result.

We've been getting high CPU alerts for billow since around Wed, Aug 13, 2014 at 6:56 PM PT, which is around the time AWS told us we were making way too many describeDBInstances requests.

This code change should fix that issue. Billow does not seem like a very intensive process when things are not buggy.

Billow would have been reporting stale results intermittently. I didn't look into how stale those results could have been. But if someone thinks it could be something causing lasting damage, let me know and I will look into it.

(Variable names could have been changed to be more clear, but I did not change them for now.)

@nelgau @jtai @joshbuddy