Repository: cloud-custodian/cloud-custodian

Issue:
Inability to use config-poll-rule for > 100 resources due to limits in PutEvaluation
**Describe the bug**

When doing a config-poll-rule for when aws config does not natively support a resource like kinesis, custodian fails when updating config because of what appears to be an AWS limitation on the number of evaluations that can be put into a request.


**To Reproduce**
Create more than 100 kinesis data streams (code to follow??)

Steps to reproduce the behavior:

**Expected behavior**
A clear and concise description of what you expected to happen.


**Background (please complete the following information):**
 - Python Version: 3.9.1
 - Custodian Version: 0.9.10
 - Tool Version: [if applicable]
 - Cloud Provider: aws
 - Policy: [please exclude any account/sensitive information]
```yaml
policies: 
- name: kinesis-tag-compliance
  resource: aws.kinesis
  mode:
      # This is a different mode than the other resources as kinesis streams are not currently one
      # of the supported AWS Config resources
      role: custodian
      type: config-poll-rule
      schedule: One_Hour
  tag-filters: 
      - "tag:Name": absent
      - "tag:Contact": absent
      - "tag:Environment": absent
      - "tag:Pipeline": absent
      - "tag:System": absent
      - "tag:TaggingVersion": absent
```
 - Traceback: [if applicable, please exclude sensitive/account information]
```
[ERROR]	2021-02-16T16:23:00.481Z	befa31da-95e0-41c6-b47c-c7eb9c0338d7	error during policy execution
Traceback (most recent call last):
  File "/var/task/c7n/handler.py", line 165, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1140, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 790, in run
    self.policy.resource_manager.retry(
  File "/var/task/c7n/utils.py", line 444, in _retry
    return func(*args, **kw)
  File "/var/runtime/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the PutEvaluations operation: 1 validation error detected: Value '[com.amazonaws.starling.dove.Evaluation@27de0032, com.amazonaws.starling.dove.Evaluation@35e1583c, com.amazonaws.starling.dove.Evaluation@57c577dd, com.amazonaws.starling.dove.Evaluation@83d31304, com.amazonaws.starling.dove.Evaluation@88179743, com.amazonaws.starling.dove.Evaluation@43843883, com.amazonaws.starling.dove.Evaluation@5275a3f1, com.amazonaws.starling.dove.Evaluation@eb15ef74, com.amazonaws.starling.dove.Evaluation@85ae8df3, com.amazonaws.starling.dove.Evaluation@986519ca, com.amazonaws.starling.dove.Evaluation@a66adf4, com.amazonaws.starling.dove.Evaluation@14c8caae, com.amazonaws.starling.dove.Evaluation@732687a2, com.amazonaws.starling.dove.Evaluation@ab113137, com.amazonaws.starling.dove.Evaluation@f732086e, com.amazonaws.starling.dove.Evaluation@ee1be098, com.amazonaws.starling.dove.Evaluation@49e0cace, com.amazonaws.starling.dove.Evaluation@de6946db, com.amazonaws.starling.dove.Evaluation@5b4bc798, com.amazonaws.starling.dove.Evaluation@795bd423, com.amazonaws.starling.dove.Evaluation@ec34a188, com.amazonaws.starling.dove.Evaluation@a919f601, com.amazonaws.starling.dove.Evaluation@fa876e93, com.amazonaws.starling.dove.Evaluation@a8bd23c9, com.amazonaws.starling.dove.Evaluation@b4e94a84, com.amazonaws.starling.dove.Evaluation@cb541157, com.amazonaws.starling.dove.Evaluation@d5fe6f5a, com.amazonaws.starling.dove.Evaluation@5845d674, com.amazonaws.starling.dove.Evaluation@7a304149, com.amazonaws.starling.dove.Evaluation@c9ced591, com.amazonaws.starling.dove.Evaluation@17cebbd2, com.amazonaws.starling.dove.Evaluation@830c779b, com.amazonaws.starling.dove.Evaluation@197b7b20, com.amazonaws.starling.dove.Evaluation@e55e1cb8, com.amazonaws.starling.dove.Evaluation@e728aeb, com.amazonaws.starling.dove.Evaluation@77cc8b4e, com.amazonaws.starling.dove.Evaluation@25a3f0b0, com.amazonaws.starling.dove.Evaluation@fc7ea9c9, com.amazonaws.starling.dove.Evaluation@12ed64d9, com.amazonaws.starling.dove.Evaluation@49d3cedc, com.amazonaws.starling.dove.Evaluation@36cb954a, com.amazonaws.starling.dove.Evaluation@e913a4de, com.amazonaws.starling.dove.Evaluation@7d2fe739, com.amazonaws.starling.dove.Evaluation@39b2db73, com.amazonaws.starling.dove.Evaluation@54de31b4, com.amazonaws.starling.dove.Evaluation@ecf16a63, com.amazonaws.starling.dove.Evaluation@196bf89a, com.amazonaws.starling.dove.Evaluation@c6614340, com.amazonaws.starling.dove.Evaluation@4508b7b0, com.amazonaws.starling.dove.Evaluation@d2cbce8f, com.amazonaws.starling.dove.Evaluation@4829b45a, com.amazonaws.starling.dove.Evaluation@32db6718, com.amazonaws.starling.dove.Evaluation@10000039, com.amazonaws.starling.dove.Evaluation@e78f25b1, com.amazonaws.starling.dove.Evaluation@d840dd29, com.amazonaws.starling.dove.Evaluation@8490f6c8, com.amazonaws.starling.dove.Evaluation@b6f85207, com.amazonaws.starling.dove.Evaluation@bb73afa0, com.amazonaws.starling.dove.Evaluation@81d07ab3, com.amazonaws.starling.dove.Evaluation@af6c90ca, com.amazonaws.starling.dove.Evaluation@3a5e59ad, com.amazonaws.starling.dove.Evaluation@75610b6c, com.amazonaws.starling.dove.Evaluation@563ac45a, com.amazonaws.starling.dove.Evaluation@f24b0c9c, com.amazonaws.starling.dove.Evaluation@96899edb, com.amazonaws.starling.dove.Evaluation@2c852755, com.amazonaws.starling.dove.Evaluation@8d39927e, com.amazonaws.starling.dove.Evaluation@17f6fbf3, com.amazonaws.starling.dove.Evaluation@2ae3f156, com.amazonaws.starling.dove.Evaluation@31563b2a, com.amazonaws.starling.dove.Evaluation@ed11d1d, com.amazonaws.starling.dove.Evaluation@b8f93562, com.amazonaws.starling.dove.Evaluation@b6ec6db8, com.amazonaws.starling.dove.Evaluation@d18ba10, com.amazonaws.starling.dove.Evaluation@aac714f3, com.amazonaws.starling.dove.Evaluation@c8983162, com.amazonaws.starling.dove.Evaluation@29b97b21, com.amazonaws.starling.dove.Evaluation@675b2efe, com.amazonaws.starling.dove.Evaluation@34b6e207, com.amazonaws.starling.dove.Evaluation@657ef526, com.amazonaws.starling.dove.Evaluation@fd2aedee, com.amazonaws.starling.dove.Evaluation@656c316, com.amazonaws.starling.dove.Evaluation@9c977e7f, com.amazonaws.starling.dove.Evaluation@4cc810f7, com.amazonaws.starling.dove.Evaluation@f4110bc4, com.amazonaws.starling.dove.Evaluation@a3e31c83, com.amazonaws.starling.dove.Evaluation@da1c5818, com.amazonaws.starling.dove.Evaluation@24695f80, com.amazonaws.starling.dove.Evaluation@c35b521c, com.amazonaws.starling.dove.Evaluation@79e698c5, com.amazonaws.starling.dove.Evaluation@b959fff1, com.amazonaws.starling.dove.Evaluation@5a215163, com.amazonaws.starling.dove.Evaluation@d479f02d, com.amazonaws.starling.dove.Evaluation@5ae04031, com.amazonaws.starling.dove.Evaluation@90335f18, com.amazonaws.starling.dove.Evaluation@51ef96b1, com.amazonaws.starling.dove.Evaluation@16a4459d, com.amazonaws.starling.dove.Evaluation@57bf4387, com.amazonaws.starling.dove.Evaluation@c1f5b8ed, com.amazonaws.starling.dove.Evaluation@997681d5, com.amazonaws.starling.dove.Evaluation@8ecc23d2, com.amazonaws.starling.dove.Evaluation@d0d49359, com.amazonaws.starling.dove.Evaluation@a1a1361d, com.amazonaws.starling.dove.Evaluation@228372b5, com.amazonaws.starling.dove.Evaluation@e4bfafb1, com.amazonaws.starling.dove.Evaluation@7a48a110, com.amazonaws.starling.dove.Evaluation@e7602849, com.amazonaws.starling.dove.Evaluation@dca3e7dc, com.amazonaws.starling.dove.Evaluation@d91f6c04, com.amazonaws.starling.dove.Evaluation@89232735, com.amazonaws.starling.dove.Evaluation@616889dd, com.amazonaws.starling.dove.Evaluation@ac4b9420, com.amazonaws.starling.dove.Evaluation@e0b3d2ba, com.amazonaws.starling.dove.Evaluation@97f202a8, com.amazonaws.starling.dove.Evaluation@d1ea9d66, com.amazonaws.starling.dove.Evaluation@e84562e1, com.amazonaws.starling.dove.Evaluation@4d14e584, com.amazonaws.starling.dove.Evaluation@5a15dd5b, com.amazonaws.starling.dove.Evaluation@2187c06f, com.amazonaws.starling.dove.Evaluation@c536cd0b, com.amazonaws.starling.dove.Evaluation@a0dd940a, com.amazonaws.starling.dove.Evaluation@b2d6026f, com.amazonaws.starling.dove.Evaluation@e966adc1, com.amazonaws.starling.dove.Evaluation@1c9c77ca, com.amazonaws.starling.dove.Evaluation@bd051259, com.amazonaws.starling.dove.Evaluation@b99cd7c6, com.amazonaws.starling.dove.Evaluation@ab8355ed, com.amazonaws.starling.dove.Evaluation@7d7cd341, com.amazonaws.starling.dove.Evaluation@87b1f372, com.amazonaws.starling.dove.Evaluation@f28c8c97]' at 'evaluations' failed to satisfy constraint: Member must have length less than or equal to 100
```






Discussion:
ah... beautiful java tracebacks with internal codenames :-) `com.amazonaws.starling.dove`

Resolving patch:
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -763,11 +763,19 @@ def _get_client(self):
         return utils.local_session(
             self.policy.session_factory).client('config')
 
+    def put_evaluations(self, client, token, evaluations):
+        for eval_set in utils.chunks(evaluations, 100):
+            self.policy.resource_manager.retry(
+                client.put_evaluations,
+                Evaluations=eval_set,
+                ResultToken=token)
+
     def run(self, event, lambda_context):
         cfg_event = json.loads(event['invokingEvent'])
         resource_type = self.policy.resource_manager.resource_type.cfn_type
         resource_id = self.policy.resource_manager.resource_type.id
         client = self._get_client()
+        token = event.get('resultToken')
 
         matched_resources = set()
         for r in PullMode.run(self):
@@ -786,11 +794,8 @@ def run(self, event, lambda_context):
             Annotation='The resource is not compliant with policy:%s.' % (
                 self.policy.name))
             for r in matched_resources]
-        if evaluations:
-            self.policy.resource_manager.retry(
-                client.put_evaluations,
-                Evaluations=evaluations,
-                ResultToken=event.get('resultToken', 'No token found.'))
+        if evaluations and token:
+            self.put_evaluations(client, token, evaluations)
 
         evaluations = [dict(
             ComplianceResourceType=resource_type,
@@ -800,11 +805,8 @@ def run(self, event, lambda_context):
             Annotation='The resource is compliant with policy:%s.' % (
                 self.policy.name))
             for r in unmatched_resources]
-        if evaluations:
-            self.policy.resource_manager.retry(
-                client.put_evaluations,
-                Evaluations=evaluations,
-                ResultToken=event.get('resultToken', 'No token found.'))
+        if evaluations and token:
+            self.put_evaluations(client, token, evaluations)
         return list(matched_resources)
 
 


Test patch: