Repository: cloud-custodian/cloud-custodian

Issue:
ASG tagging fails when >25 ASGs matched
We have a policy very similar to the following:

```yml
- actions:
  - days: 7
    message: 'asg-inactive-mark: {op}@{action_date}'
    op: delete
    tag: c7n-asg-inactive
    type: mark-for-op
  comments: Tag empty ASGs for later termination.
  filters:
  - key: Instances
    op: less-than
    type: value
    value: 1
    value_type: size
  name: asg-inactive-mark
  resource: asg
```

As of a few days ago, this policy is now matching **143** ASGs in one of our accounts/regions, and is subsequently failing with the following exception (note line numbers are off by a few, as I've hacked in some additional logging):

```
2019-05-21 09:06:11,884: custodian.commands:ERROR Error while executing policy asg-inactive-mark, continuing
Traceback (most recent call last):
  File "/home/jantman/GIT/cloud-custodian/c7n/commands.py", line 270, in run
    policy()
  File "/home/jantman/GIT/cloud-custodian/c7n/policy.py", line 948, in __call__
    resources = mode.run()
  File "/home/jantman/GIT/cloud-custodian/c7n/policy.py", line 264, in run
    results = a.process(resources)
  File "/home/jantman/GIT/cloud-custodian/c7n/resources/asg.py", line 1438, in process
    self.process_resource_set(client, asgs, [{'Key': key, 'Value': msg}])
  File "/home/jantman/GIT/cloud-custodian/c7n/resources/asg.py", line 1152, in process_resource_set
    self.manager.retry(client.create_or_update_tags, Tags=tag_params)
  File "/home/jantman/GIT/cloud-custodian/c7n/utils.py", line 375, in _retry
    return func(*args, **kw)
  File "/home/jantman/GIT/cloud-custodian/lib/python2.7/site-packages/botocore-1.12.138-py2.7.egg/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/jantman/GIT/cloud-custodian/lib/python2.7/site-packages/botocore-1.12.138-py2.7.egg/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
ClientError: An error occurred (ValidationError) when calling the CreateOrUpdateTags operation: List of modified tags exceeds the maximum allowed size. Please retry your request with 25 or fewer tags.
```

It appears that the ASG [mark-for-op](https://github.com/cloud-custodian/cloud-custodian/blob/f139a61c8fc200af22c1e7e04e7ffe07a2f001cf/c7n/resources/asg.py#L1363) action is [calling](https://github.com/cloud-custodian/cloud-custodian/blob/f139a61c8fc200af22c1e7e04e7ffe07a2f001cf/c7n/resources/asg.py#L1431) [c7n.resources.asg.Tag.process_resource_set()](https://github.com/cloud-custodian/cloud-custodian/blob/f139a61c8fc200af22c1e7e04e7ffe07a2f001cf/c7n/resources/asg.py#L1133) with the full list of ASGs, which ends up being [passed to create_or_update_tags](https://github.com/cloud-custodian/cloud-custodian/blob/f139a61c8fc200af22c1e7e04e7ffe07a2f001cf/c7n/resources/asg.py#L1145) as one list of a length equal to the number of ASGs.

This results in sending a list with too many tags.

Interestingly, the [Tag action's process() method](https://github.com/cloud-custodian/cloud-custodian/blob/f139a61c8fc200af22c1e7e04e7ffe07a2f001cf/c7n/resources/asg.py#L1111) splits the ASGs list into chunks of size one and calls ``process_resource_set()`` once for each ASG, but that logic is bypassed when ``process_resource_set()`` is called directly from the ``MarkForOp`` class.

The end result of this is that running mark-for-op in a policy that matches more than 25 ASGs will throw an exception and not tag anything.

I'd be happy to open a PR to fix this. I'm not sure what the best implementation would be as I'm unclear on some of the internal conventions of Actions' ``process()`` and ``process_resource_set()`` methods, but it seems to me that the most performant fix without impacting the current behavior would be to modify the [create_or_update_tags call in Tag.process_resource_set()](https://github.com/cloud-custodian/cloud-custodian/blob/f139a61c8fc200af22c1e7e04e7ffe07a2f001cf/c7n/resources/asg.py#L1145) to operate on chunks of (maximum size) 25, something like changing:

```python
self.manager.retry(client.create_or_update_tags, Tags=tag_params)
```

to:

```python
for tag_set in chunks(tag_params, 25):
    self.manager.retry(client.create_or_update_tags, Tags=tag_set)
```


Discussion:
thanks for the bug report, i think respecting the  for batch size of the tag action should be in the mark-for-op for the asg, which it is for other mark-for-op implementations which are more generic. its actually unclear to me, why the asg mark-for-op is customized at this point (ie. its bit-rot), the implementation could just be removed.

Resolving patch:
diff --git a/c7n/resources/asg.py b/c7n/resources/asg.py
--- a/c7n/resources/asg.py
+++ b/c7n/resources/asg.py
@@ -18,8 +18,6 @@
 from collections import Counter
 from concurrent.futures import as_completed
 
-from datetime import datetime, timedelta
-from dateutil import tz as tzutil
 from dateutil.parser import parse
 
 import logging
@@ -29,12 +27,12 @@
 from c7n.actions import Action
 from c7n.exceptions import PolicyValidationError
 from c7n.filters import ValueFilter, AgeFilter, Filter
-from c7n.filters.offhours import OffHour, OnHour, Time
+from c7n.filters.offhours import OffHour, OnHour
 import c7n.filters.vpc as net_filters
 
 from c7n.manager import resources
 from c7n import query
-from c7n.tags import TagActionFilter, DEFAULT_TAG, TagCountFilter, TagTrim
+from c7n.tags import TagActionFilter, DEFAULT_TAG, TagCountFilter, TagTrim, TagDelayedAction
 from c7n.utils import local_session, type_schema, chunks, get_retry
 
 from .ec2 import deserialize_user_data
@@ -1360,7 +1358,7 @@ def propagate_instance_tag(self, source, destination_tag, asg):
 
 
 @ASG.action_registry.register('mark-for-op')
-class MarkForOp(Tag):
+class MarkForOp(TagDelayedAction):
     """Action to create a delayed action for a later date
 
     :example:
@@ -1387,61 +1385,25 @@ class MarkForOp(Tag):
         op={'type': 'string'},
         key={'type': 'string'},
         tag={'type': 'string'},
+        tz={'type': 'string'},
         message={'type': 'string'},
         days={'type': 'number', 'minimum': 0},
         hours={'type': 'number', 'minimum': 0})
-
+    schema_alias = False
     default_template = (
         'AutoScaleGroup does not meet org policy: {op}@{action_date}')
 
-    def validate(self):
-        self.tz = tzutil.gettz(
-            Time.TZ_ALIASES.get(self.data.get('tz', 'utc')))
-        if not self.tz:
-            raise PolicyValidationError(
-                "Invalid timezone specified %s on %s" % (self.tz, self.manager.data))
-        op = self.data.get('op')
-        if op not in self.manager.action_registry:
-            raise PolicyValidationError(
-                "Invalid op %s for asg on policy %s" % (op, self.manager.data))
-        return self
-
-    def process(self, asgs):
-        self.tz = tzutil.gettz(
-            Time.TZ_ALIASES.get(self.data.get('tz', 'utc')))
-
-        msg_tmpl = self.data.get('message', self.default_template)
-        key = self.data.get('key', self.data.get('tag', DEFAULT_TAG))
-        op = self.data.get('op', 'suspend')
-        days = self.data.get('days', 0)
-        hours = self.data.get('hours', 0)
-
-        action_date = self._generate_timestamp(days, hours)
-        try:
-            msg = msg_tmpl.format(
-                op=op, action_date=action_date)
-        except Exception:
-            self.log.warning("invalid template %s" % msg_tmpl)
-            msg = self.default_template.format(
-                op=op, action_date=action_date)
-
-        self.log.info("Tagging %d asgs for %s on %s" % (
-            len(asgs), op, action_date))
-        client = local_session(self.manager.session_factory).client('autoscaling')
-        self.process_resource_set(client, asgs, [{'Key': key, 'Value': msg}])
-
-    def _generate_timestamp(self, days, hours):
-        n = datetime.now(tz=self.tz)
-        if days == hours == 0:
-            # maintains default value of days being 4 if nothing is provided
-            days = 4
-        action_date = (n + timedelta(days=days, hours=hours))
-        if hours > 0:
-            action_date_string = action_date.strftime('%Y/%m/%d %H%M %Z')
-        else:
-            action_date_string = action_date.strftime('%Y/%m/%d')
-
-        return action_date_string
+    def get_config_values(self):
+        d = {
+            'op': self.data.get('op', 'stop'),
+            'tag': self.data.get('key', self.data.get('tag', DEFAULT_TAG)),
+            'msg': self.data.get('message', self.default_template),
+            'tz': self.data.get('tz', 'utc'),
+            'days': self.data.get('days', 0),
+            'hours': self.data.get('hours', 0)}
+        d['action_date'] = self.generate_timestamp(
+            d['days'], d['hours'])
+        return d
 
 
 @ASG.action_registry.register('suspend')
diff --git a/c7n/tags.py b/c7n/tags.py
--- a/c7n/tags.py
+++ b/c7n/tags.py
@@ -654,30 +654,30 @@ def generate_timestamp(self, days, hours):
 
         return action_date_string
 
+    def get_config_values(self):
+        d = {
+            'op': self.data.get('op', 'stop'),
+            'tag': self.data.get('tag', DEFAULT_TAG),
+            'msg': self.data.get('msg', self.default_template),
+            'tz': self.data.get('tz', 'utc'),
+            'days': self.data.get('days', 0),
+            'hours': self.data.get('hours', 0)}
+        d['action_date'] = self.generate_timestamp(
+            d['days'], d['hours'])
+        return d
+
     def process(self, resources):
-        self.tz = tzutil.gettz(
-            Time.TZ_ALIASES.get(self.data.get('tz', 'utc')))
+        cfg = self.get_config_values()
+        self.tz = tzutil.gettz(Time.TZ_ALIASES.get(cfg['tz']))
         self.id_key = self.manager.get_model().id
 
-        # Move this to policy? / no resources bypasses actions?
-        if not len(resources):
-            return
-
-        msg_tmpl = self.data.get('msg', self.default_template)
-
-        op = self.data.get('op', 'stop')
-        tag = self.data.get('tag', DEFAULT_TAG)
-        days = self.data.get('days', 0)
-        hours = self.data.get('hours', 0)
-        action_date = self.generate_timestamp(days, hours)
-
-        msg = msg_tmpl.format(
-            op=op, action_date=action_date)
+        msg = cfg['msg'].format(
+            op=cfg['op'], action_date=cfg['action_date'])
 
         self.log.info("Tagging %d resources for %s on %s" % (
-            len(resources), op, action_date))
+            len(resources), cfg['op'], cfg['action_date']))
 
-        tags = [{'Key': tag, 'Value': msg}]
+        tags = [{'Key': cfg['tag'], 'Value': msg}]
 
         # if the tag implementation has a specified batch size, it's typically
         # due to some restraint on the api so we defer to that.


Test patch: