Repository: cloud-custodian/cloud-custodian

Issue:
Retry logic for paginator actions, and RequestLimitExceeded exceptions
Hi,

Relatively new to Cloud Custodian. We're using 0.8.31.0, and deploying controls as lambdas in AWS, triggering off cloudtrail events. I've searched for this issue already and haven't seen anything similar so posting here.

What we occasionally see, is that lambdas trigger off events, receive a resource id from the event body correctly but then fail to lookup the resource (even though the resource is there). The lambda then continues (does not fail) and finishes successfully:

eg looking a log in CloudWatch i would see:

Processing event {<real event>}
..
Found resource ids: [u'<resource id>]
Resources[] .  <-- ie no resource looked up, even though the resource is there.

After spending a bit of time debugging around utils.py and query.py, what seems to happen is that calls from CloudCustodian are hitting api resource limits (say, Client.RequestLimitExceeded) in our account but not handling it as I would have expected:

example:

<ec2.py> EC2
An EC2 object is initialised and makes a call to retry for a DescribeInstances operation

<query.py> ResourceQuery._invoke_client_enum
 An 'enum_op' is passed in, which can be paginated (DescribeInstances). A paginator is successfully created and the output of the paginate operation is sent to the pager function.

<query.py> ResourceQuery.pager
'page = retry(next, iterator)' called. The first time 'next' is executed with Utils._retry it fails with a Client.RequestLimitExceeded. The logic inside the retry function correctly catches and triggers a retry. However the next call to 'next' throws a StopIteration exception, this is caught back in 'pager' presumably as a sign there are no more results to iterate over, and the functions all return, but with no resources.

If i replay the event through the lambda, when the account is quiet, then it works as expected.

Is this something people have seen before or have solutions for? Or am I missing something here. I would have expected that given retry(..) is effectively called twice (once for the initial paginate call, and then secondly for iterator.next()) that we'd want to make the first retry(..) kick in again and redo the pagination operation.

thanks for any help
nic




Discussion:
Thanks for stepping through this, i'll try and dig deeper, the retry logic is against the botocore.pagination logic, the logic was that the behavior there was to raise without updating internal bookeeping data structures they have, which allowed the c7n pager / retry logic to catch and retry in place. 
thanks again for the report, I had a chance to dig deeper and I've opened pull request #3170 to address this.
Thanks for the quick turn around, confirming the issue! I'll pull down the fix as well and test.
I've been testing with a patched version of CC today (I took the new versions of shield.py and query.py). Whilst I've been struggling to hit the API limits, when I did i managed to get a different outcome than prior to the patch - the exception is handled more gracefully now but it still seems to only  do one call into retry(..) instead of multiple. I get now:

[WARNING] <timestamp>  event ids not resolved : [u/<valid ebs snap>]  error: An error occurred  (RequestLimitExceeded) when calling the DescribeSnapShots operation(reached max retries : 4) Request limit exceeded.

From the debug i had within utils.retry(..), i know its only been called once. I'll continue to test but just wanted to flag this up as a possibility.


I can confirm that running the patch as lambdas on AWS still has the issue above. However I think i have an additional fix for it (1 liner) that makes the retry logic work correctly. I'll continue testing and hopefully contribute it back if it works
hmm.. i'm curious what your seeing, i added a unit test to validate the retry is done in the previous pr (which is in the latest release as well), but two things to keep in mind here, retry is finite, we will eventually stop and give up. second, custodian is retrying on top of the sdk baseline configuration (which has a separate retry builtin). there is only one call into to retry, as on exit from there, we either have a result or have exhausted all retries.

Apologies for the delay.

So this is what i had to do to get the fix to fully work:  in query.py _invoke_client_enum(..):

` results = p.paginate(**params)`
` if retry`
 `          p.PAGE_ITERATOR_CLS = RetryPageIterator`

Inside the paginate call is where the PAGE_ITERATOR_CLS is set/returned. So setting it after the paginate call meant it didn't get used.  When i changed it to:

`if retry`
 `         p.PAGE_ITERATOR_CLS = RetryPageIterator`
`results = p.paginate(**params)
`

Then the retry functionality inside RetryPageIterator was enabled and it worked fine (for me at least!)

(EDIT: lost the indentation but you get the idea)
aha, thanks, just ended up debugging this today, and noticed the same thing.

Resolving patch:
diff --git a/c7n/query.py b/c7n/query.py
--- a/c7n/query.py
+++ b/c7n/query.py
@@ -61,9 +61,9 @@ def resolve(resource_type):
     def _invoke_client_enum(self, client, enum_op, params, path, retry=None):
         if client.can_paginate(enum_op):
             p = client.get_paginator(enum_op)
-            results = p.paginate(**params)
             if retry:
                 p.PAGE_ITERATOR_CLS = RetryPageIterator
+            results = p.paginate(**params)
             data = results.build_full_result()
         else:
             op = getattr(client, enum_op)


Test patch: