Repository: cloud-custodian/cloud-custodian

Issue:
aws.elasticsearch Error Scanning More Than 5 domains
**Describe the bug**
When running any elasticsearch policy on an account region with more than 5 elasticsearch domains the policy now bombs out with the error - ```error:An error occurred (ValidationException) when calling the DescribeElasticsearchDomains operation: Please provide a maximum of 5 Elasticsearch domain names to describe.```

**To Reproduce**
Create 6 es domains and run an elasticsearch c7n policy, error will occur

**Expected behavior**
It should chunk the calls into domains of 5 or less


**Background (please complete the following information):**
 - OS: Ubuntu v20
 - Python Version: 3.8
 - Custodian Version: 0.9.2.0
 - Tool Version: [if applicable]
 - Cloud Provider: aws
 - Policy:  any policy which queries ES
 - Traceback: 
```
[ERROR]	2020-05-22T14:51:25.978Z	9ef7929b-b494-434e-9f9f-dfdfdfdfdfdfd Error while executing policy
Traceback (most recent call last):
  File "/var/task/c7n/policy.py", line 291, in run
    resources = self.policy.resource_manager.resources()
  File "/var/task/c7n/query.py", line 466, in resources
    resources = self.augment(resources)
  File "/var/task/c7n/query.py", line 521, in augment
    return self.source.augment(resources)
  File "/var/task/c7n/resources/elasticsearch.py", line 48, in augment
    return _augment(domains)
  File "/var/task/c7n/resources/elasticsearch.py", line 39, in _augment
    resources = self.manager.retry(
  File "/var/task/c7n/utils.py", line 373, in _retry
    return func(*args, **kw)
  File "/var/task/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/task/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the DescribeElasticsearchDomains operation: Please provide a maximum of 5 Elasticsearch domain names to describe.
```
 - `custodian version --debug` output

**Additional context**
Seems to be introduced with 0.9.2.0


Discussion:

Patch:
diff --git a/c7n/resources/asg.py b/c7n/resources/asg.py
index 9d01ee426..181141dd9 100644
--- a/c7n/resources/asg.py
+++ b/c7n/resources/asg.py
@@ -1215,7 +1215,7 @@ class PropagateTags(Action):
         if self.data.get('trim', False):
             instances = [self.instance_map[i] for i in instance_ids]
             self.prune_instance_tags(client, asg, tag_set, instances)
-        if not self.manager.config.dryrun:
+        if not self.manager.config.dryrun and instances:
             client.create_tags(
                 Resources=instance_ids,
                 Tags=[{'Key': k, 'Value': v} for k, v in tag_map.items()])
diff --git a/c7n/resources/elasticsearch.py b/c7n/resources/elasticsearch.py
index e1355b824..76a50680f 100644
--- a/c7n/resources/elasticsearch.py
+++ b/c7n/resources/elasticsearch.py
@@ -18,7 +18,7 @@ from c7n.filters import MetricsFilter
 from c7n.filters.vpc import SecurityGroupFilter, SubnetFilter, VpcFilter
 from c7n.manager import resources
 from c7n.query import ConfigSource, DescribeSource, QueryResourceManager, TypeInfo
-from c7n.utils import local_session, type_schema
+from c7n.utils import chunks, local_session, type_schema
 from c7n.tags import Tag, RemoveTag, TagActionFilter, TagDelayedAction
 
 from .securityhub import PostFinding
@@ -34,6 +34,7 @@ class DescribeDomain(DescribeSource):
     def augment(self, domains):
         client = local_session(self.manager.session_factory).client('es')
         model = self.manager.get_model()
+        results = []
 
         def _augment(resource_set):
             resources = self.manager.retry(
@@ -45,7 +46,10 @@ class DescribeDomain(DescribeSource):
                     client.list_tags, ARN=rarn).get('TagList', [])
             return resources
 
-        return _augment(domains)
+        for resource_set in chunks(domains, 5):
+            results.extend(_augment(resource_set))
+
+        return results
 
 
 @resources.register('elasticsearch')
diff --git a/docs/source/aws/examples/eipgarbagecollect.rst b/docs/source/aws/examples/eipgarbagecollect.rst
new file mode 100644
index 000000000..24c846d9d
--- /dev/null
+++ b/docs/source/aws/examples/eipgarbagecollect.rst
@@ -0,0 +1,95 @@
+EIP - Garbage Collect Unattached Elastic IPs
+============================================
+
+Use the ``mark-for-op`` action to mark a resource for action later. One common
+pattern to follow is to mark a resource with an operation (example: release) in
+n days. In the subsequent days leading up to the marked date, run a unmark or
+untag policy if the resource has become compliant in the mean time.
+
+You can use this principle to implement garbage collection on resources. In this
+example, Custodian will first mark all unattached Elastic IPs for removal. The
+next policy will then unmark any EIP that has been attached and has the
+``maid_status`` tag, indicating that it had been previously marked. Finally, the
+third policy will filter in any resources that have been marked and run the
+``release`` action.
+
+It is important to note that the release policy will need to be run on the day that
+the resource is marked for, else the resource will still exist in the account.
+The mark operation only tags the resource with metadata about the upcoming operation.
+Operationally, the policy still must be executed on the day that is specified in
+the tag.
+
+Note: all resources that are ``marked-for-op`` up to and including the current
+date will be filtered in when utilizing the ``marked-for-op`` filter.
+
+  .. code-block:: yaml
+
+    vars:
+      notify: &notify
+        type: notify
+        to:
+          - slack://#slack-channel
+        subject: "EIP - No Instances Attached - [custodian {{ account }} - {{ region }}]"
+        transport:
+          type: sqs
+          queue: https://sqs.us-east-2.amazonaws.com/123456789012/mailer
+          region: us-east-2
+      run_mode: &run_mode
+        type: periodic
+        schedule: "rate(1 day)"
+        tags:
+          app: "c7n"
+          env: "tools"
+          account: "{account_id}"
+      eip_filters: &eip_filters
+        - type: value
+          key: InstanceId
+          value: absent
+        - type: value
+          key: AssociationId
+          value: absent
+
+    policies:
+      - name: unused-eip-mark
+        resource: network-addr
+        description: "Mark any EIP with no instances attached for action in 7 days"
+        filters:
+          - "tag:maid_status_eip": absent
+          - and: *eip_filters
+        mode:
+          <<: *run_mode
+        actions:
+          - type: mark-for-op
+            tag: maid_status_eip
+            days: 7
+            op: release
+
+      - name: unused-eip-unmark-if-in-use
+        resource: network-addr
+        description: |
+          Remove the maid_status_eip tag from any eip which has instances attached
+        filters:
+          - "tag:maid_status_eip": not-null
+          - not: 
+            - or: *eip_filters
+        mode:
+          <<: *run_mode
+        actions:
+          - type: remove-tag
+            tags: [maid_status_eip]
+
+      - name: unused-eip-action
+        resource: network-addr
+        description: "Release EIP after 7 days of having no instances"
+        filters:
+          - "tag:maid_status_eip": not-null
+          - type: marked-for-op
+            op: release
+            tag: maid_status_eip
+        mode:
+          <<: *run_mode
+        actions:
+          - type: release
+          - <<: *notify
+            action_desc: "EIP released"
+            violation_desc: "EIP has been unused for 7 days"


Test patch:
diff --git a/tests/data/placebo/test_asg_propagate_tag_no_instances/autoscaling.DescribeAutoScalingGroups_1.json b/tests/data/placebo/test_asg_propagate_tag_no_instances/autoscaling.DescribeAutoScalingGroups_1.json
new file mode 100644
index 000000000..3e8368da5
--- /dev/null
+++ b/tests/data/placebo/test_asg_propagate_tag_no_instances/autoscaling.DescribeAutoScalingGroups_1.json
@@ -0,0 +1,64 @@
+{
+    "status_code": 200,
+    "data": {
+        "AutoScalingGroups": [
+            {
+                "AutoScalingGroupARN": "arn:aws:autoscaling:us-east-1:123456789012:autoScalingGroup:5fd42b45-825e-45c1-b02a-6a97d9512f30:autoScalingGroupName/c7n.asg.ec2.01",
+                "TargetGroupARNs": [],
+                "SuspendedProcesses": [],
+                "DesiredCapacity": 1,
+                "Tags": [
+                    {
+                        "ResourceType": "auto-scaling-group",
+                        "ResourceId": "CustodianASG",
+                        "PropagateAtLaunch": true,
+                        "Value": "ubuntu",
+                        "Key": "Platform"
+                    }
+                ],
+                "EnabledMetrics": [],
+                "LoadBalancerNames": [],
+                "AutoScalingGroupName": "c7n.asg.ec2.01",
+                "DefaultCooldown": 300,
+                "MinSize": 1,
+                "Instances": [
+                ],
+                "MaxSize": 1,
+                "VPCZoneIdentifier": "subnet-ea6429a3,subnet-2c5dcf01",
+                "HealthCheckGracePeriod": 300,
+                "TerminationPolicies": [
+                    "Default"
+                ],
+                "LaunchConfigurationName": "c7n.asg.01",
+                "CreatedTime": {
+                    "hour": 19,
+                    "__class__": "datetime",
+                    "month": 2,
+                    "second": 3,
+                    "microsecond": 171000,
+                    "year": 2017,
+                    "day": 24,
+                    "minute": 0
+                },
+                "AvailabilityZones": [
+                    "us-east-1a",
+                    "us-east-1b"
+                ],
+                "HealthCheckType": "EC2",
+                "NewInstancesProtectedFromScaleIn": false
+            }
+        ],
+        "ResponseMetadata": {
+            "RetryAttempts": 0,
+            "HTTPStatusCode": 200,
+            "RequestId": "770b2115-fac8-11e6-ac2c-5ff00137fb8b",
+            "HTTPHeaders": {
+                "x-amzn-requestid": "770b2115-fac8-11e6-ac2c-5ff00137fb8b",
+                "vary": "Accept-Encoding",
+                "content-length": "7006",
+                "content-type": "text/xml",
+                "date": "Fri, 24 Feb 2017 19:35:57 GMT"
+            }
+        }
+    }
+}
diff --git a/tests/data/placebo/test_asg_propagate_tag_no_instances/ec2.CreateTags_1.json b/tests/data/placebo/test_asg_propagate_tag_no_instances/ec2.CreateTags_1.json
new file mode 100644
index 000000000..420881faf
--- /dev/null
+++ b/tests/data/placebo/test_asg_propagate_tag_no_instances/ec2.CreateTags_1.json
@@ -0,0 +1,20 @@
+{
+    "status_code": 400,
+    "data": {
+        "Error": {
+            "Code": "MissingParameter",
+            "Message": "The request must contain the parameter resourceIdSet"
+        },
+        "ResponseMetadata": {
+            "RequestId": "f229a23a-def5-4288-8ee0-f754ea63eda2",
+            "HTTPStatusCode": 400,
+            "HTTPHeaders": {
+                "transfer-encoding": "chunked",
+                "date": "Thu, 21 Nov 2019 18:49:02 GMT",
+                "connection": "close",
+                "server": "AmazonEC2"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
diff --git a/tests/data/placebo/test_asg_propagate_tag_no_instances/ec2.DeleteTags_1.json b/tests/data/placebo/test_asg_propagate_tag_no_instances/ec2.DeleteTags_1.json
new file mode 100644
index 000000000..8a17226c3
--- /dev/null
+++ b/tests/data/placebo/test_asg_propagate_tag_no_instances/ec2.DeleteTags_1.json
@@ -0,0 +1,9 @@
+{
+    "status_code": 200,
+    "data": {
+        "ResponseMetadata": {
+            "HTTPStatusCode": 200,
+            "RequestId": "1b312937-e532-443a-99bd-1dab530b4a84"
+        }
+    }
+}
diff --git a/tests/test_asg.py b/tests/test_asg.py
index a4b7d75b5..a7c3d5b41 100644
--- a/tests/test_asg.py
+++ b/tests/test_asg.py
@@ -819,6 +819,28 @@ class AutoScalingTest(BaseTest):
         self.assertEqual(len(resources), 1)
         self.assertEqual(resources[0]["AutoScalingGroupName"], "c7n-asg-np-missing")
 
+    def test_asg_propagate_tag_no_instances(self):
+        factory = self.replay_flight_data("test_asg_propagate_tag_no_instances")
+        p = self.load_policy(
+            {
+                "name": "asg-tag",
+                "resource": "asg",
+                "filters": [{"tag:Platform": "ubuntu"}],
+                "actions": [
+                    {
+                        "type": "propagate-tags",
+                        "trim": True,
+                        "tags": ["CustomerId", "Platform"],
+                    },
+                ],
+            },
+            session_factory=factory,
+        )
+
+        resources = p.run()
+        self.assertEqual(len(resources), 1)
+        self.assertEqual(resources[0]["AutoScalingGroupName"], "c7n.asg.ec2.01")
+
     def test_asg_filter_capacity_delta_match(self):
         factory = self.replay_flight_data("test_asg_filter_capacity_delta_match")
         p = self.load_policy(