Repository: cloud-custodian/cloud-custodian

Issue:
Variable interpolation only works on strings
**Describe the bug**

I have a c7n-org setup that has a variable that defines the grace period for the deletion of resources (EBS in this example). That variable is passed through as an string to the interpolation to make sure that yaml doesn't attempt to convert it, however that breaks the `op` action

**To Reproduce**
Steps to reproduce the behavior:

`ebs-delete-policy.yml`

```
---
policies:
  - name: ebs-mark-unattached-deletion
    resource: ebs
    comments: |
      Mark any unattached EBS volumes for deletion in 30 days.
      Volumes set to not delete on instance termination do have
      valid use cases as data drives, but 99% of the time they
      appear to be just garbage creation.
    filters:
      - Attachments: []
      - tag:Status: absent
    actions:
      - type: mark-for-op
        op: delete
        days: "{deletion_grace_days}"
        tag: Status

  - name: ebs-unmark-attached-deletion
    resource: ebs
    comments: |
      Unmark any attached EBS volumes that were scheduled for deletion
      if they are currently attached
    filters:
      - type: value
        key: Attachments[0].Device
        value: not-null
      - tag:Status: not-null
    actions:
      - type: unmark
        tags: [Status]

  - name: ebs-delete-marked
    resource: ebs
    comments: |
      Delete any attached EBS volumes that were scheduled for deletion
    filters:
      - type: marked-for-op
        tag: Status
        op: delete
    actions:
      - delete
...
```

`accounts.yml`:

```
accounts:
  - account_id: '999999999999'
    name: account1
    role: arn:aws:iam::999999999999:role/global/cloud-custodian
    regions: [all]
    vars:
        deletion_grace_days: 30
```

Fails with error:

```
c7n_org:ERROR Exception running policy:ebs-mark-unattached-deletion account:account1 region:us-east-2 error:unsupported type for timedelta days component: str
```

**Expected behavior**

I expect that I can user variable interpolation even for items that expect integers.

**Background (please complete the following information):**
 - OS: RHEL 8.2
 - Python Version: Python 3.8.0
 - Custodian Version: 
   ```
   c7n==0.9.11
   c7n-org==0.6.10
   ```
 - Tool Version: [if applicable]
 - Cloud Provider: aws

```
Please copy/paste the following info along with any bug reports:

Custodian:   0.9.11
Python:      3.8.0 (default, Mar  9 2020, 18:02:46)
             [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Platform:    posix.uname_result(sysname='Linux', nodename='ip-10-144-29-109', release='4.18.0-193.51.1.el8_2.x86_64', version='#1 SMP Thu Apr 8 13:59:36 EDT 2021', machine='x86_64')
Using venv:  True
Docker: False
Installed:

argcomplete==1.12.2
attrs==20.3.0
boto3==1.17.33
botocore==1.20.33
importlib-metadata==3.7.3
jmespath==0.10.0
jsonpickle==1.3
jsonschema==3.2.0
pyrsistent==0.17.3
python-dateutil==2.8.1
pyyaml==5.4.1
s3transfer==0.3.6
setuptools==41.6.0
six==1.15.0
tabulate==0.8.9
typing-extensions==3.7.4.3
urllib3==1.26.4
zipp==3.4.1
```



Discussion:
this is because your explicitly quoting delete days as a string in your policy, remove the quotes and it should be fine.

ie  change this 
```
        days: "{deletion_grace_days}"
```

to this

```
        days: {deletion_grace_days}
```

@kapilt I'm a little late coming back to this, but the configuration file is parsed as YAML, and the substitutions are done using `str.format()` at a later stage of the processing. Because the interpolation is happening after the YAML stage, the only way to get the `str.format()` format through is by quoting it, since `{}` is a dictionary in YAML.

Unfortunately the configuration you provided above is not valid YAML, and will not load.

https://github.com/cloud-custodian/cloud-custodian/blob/b3d4bb26c77d663c893b7c8058dded5f43ed4f42/c7n/utils.py#L567

Is the utility used to interpolate the variables into the configuration dictionary for the policy. It's called from here: https://github.com/cloud-custodian/cloud-custodian/blob/b3d4bb26c77d663c893b7c8058dded5f43ed4f42/c7n/policy.py#L1141 which is called from here: https://github.com/cloud-custodian/cloud-custodian/blob/b3d4bb26c77d663c893b7c8058dded5f43ed4f42/tools/c7n_org/c7n_org/cli.py#L596

The policy file is loaded with the `-u` flag, which is parsed as YAML here: https://github.com/cloud-custodian/cloud-custodian/blob/b3d4bb26c77d663c893b7c8058dded5f43ed4f42/tools/c7n_org/c7n_org/cli.py#L185

If variable interpolation was done BEFORE parsing the file as YAML, your suggestion would work.
@kapilt would it be possible to reopen this issue? Also seeing this behavior on our systems and confirming @archoversight's comment from last month.

Resolving patch:
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -24,6 +24,7 @@
 from c7n import deprecated, utils
 from c7n.version import version
 from c7n.query import RetryPageIterator
+from c7n.varfmt import VarFormat
 
 log = logging.getLogger('c7n.policy')
 
@@ -1231,7 +1232,9 @@ def expand_variables(self, variables):
         Updates the policy data in-place.
         """
         # format string values returns a copy
-        updated = utils.format_string_values(self.data, **variables)
+        var_fmt = VarFormat()
+        updated = utils.format_string_values(
+            self.data, formatter=var_fmt.format, **variables)
 
         # Several keys should only be expanded at runtime, perserve them.
         if 'member-role' in updated.get('mode', {}):
diff --git a/c7n/utils.py b/c7n/utils.py
--- a/c7n/utils.py
+++ b/c7n/utils.py
@@ -580,7 +580,7 @@ def set_value_from_jmespath(source, expression, value, is_first=True):
     source[current_key] = value
 
 
-def format_string_values(obj, err_fallback=(IndexError, KeyError), *args, **kwargs):
+def format_string_values(obj, err_fallback=(IndexError, KeyError), formatter=None, *args, **kwargs):
     """
     Format all string values in an object.
     Return the updated object
@@ -588,16 +588,19 @@ def format_string_values(obj, err_fallback=(IndexError, KeyError), *args, **kwar
     if isinstance(obj, dict):
         new = {}
         for key in obj.keys():
-            new[key] = format_string_values(obj[key], *args, **kwargs)
+            new[key] = format_string_values(obj[key], formatter=formatter, *args, **kwargs)
         return new
     elif isinstance(obj, list):
         new = []
         for item in obj:
-            new.append(format_string_values(item, *args, **kwargs))
+            new.append(format_string_values(item, formatter=formatter, *args, **kwargs))
         return new
     elif isinstance(obj, str):
         try:
-            return obj.format(*args, **kwargs)
+            if formatter:
+                return formatter(obj, *args, **kwargs)
+            else:
+                return obj.format(*args, **kwargs)
         except err_fallback:
             return obj
     else:
diff --git a/c7n/varfmt.py b/c7n/varfmt.py
new file mode 100644
--- /dev/null
+++ b/c7n/varfmt.py
@@ -0,0 +1,98 @@
+from string import Formatter
+
+
+class VarFormat(Formatter):
+    """Behaves exactly like the stdlib formatter, with one additional behavior.
+
+    when a string has no format_spec and only contains a single expression,
+    retain the type of the source object.
+
+    inspired by https://pypyr.io/docs/substitutions/format-string/
+    """
+
+    def _vformat(
+        self, format_string, args, kwargs, used_args, recursion_depth, auto_arg_index=0
+    ):
+        # This is mostly verbatim from stdlib format.Formatter._vformat
+        # https://github.com/python/cpython/blob/main/Lib/string.py
+        #
+        # we have to copy alot of std logic to override the str cast
+
+        if recursion_depth < 0:
+            raise ValueError('Max string recursion exceeded')
+        result = []
+        for literal_text, field_name, format_spec, conversion in self.parse(
+            format_string
+        ):
+
+            # output the literal text
+            if literal_text:
+                result.append((literal_text, True, None))
+
+            # if there's a field, output it
+            if field_name is not None:
+                # this is some markup, find the object and do
+                #  the formatting
+
+                # handle arg indexing when empty field_names are given.
+                if field_name == '':
+                    if auto_arg_index is False:
+                        raise ValueError(
+                            'cannot switch from manual field '
+                            'specification to automatic field '
+                            'numbering'
+                        )
+                    field_name = str(auto_arg_index)
+                    auto_arg_index += 1
+                elif field_name.isdigit():
+                    if auto_arg_index:
+                        raise ValueError(
+                            'cannot switch from manual field '
+                            'specification to automatic field '
+                            'numbering'
+                        )
+                    # disable auto arg incrementing, if it gets
+                    # used later on, then an exception will be raised
+                    auto_arg_index = False
+
+                # given the field_name, find the object it references
+                #  and the argument it came from
+                obj, arg_used = self.get_field(field_name, args, kwargs)
+                used_args.add(arg_used)
+
+                # do any conversion on the resulting object
+                obj = self.convert_field(obj, conversion)
+
+                # expand the format spec, if needed
+                format_spec, auto_arg_index = self._vformat(
+                    format_spec,
+                    args,
+                    kwargs,
+                    used_args,
+                    recursion_depth - 1,
+                    auto_arg_index=auto_arg_index,
+                )
+
+                # defer format
+                result.append((obj, False, format_spec))
+
+        # if input is a single expression (ie. '{expr}' don't cast
+        # source to string.
+        if len(result) == 1:
+            obj, is_literal, format_spec = result[0]
+            if is_literal:
+                return obj, auto_arg_index
+            if format_spec:
+                return self.format_field(obj, format_spec), auto_arg_index
+            else:
+                return obj, auto_arg_index
+        else:
+            return (
+                ''.join(
+                    [
+                        obj if is_literal else self.format_field(obj, format_spec)
+                        for obj, is_literal, format_spec in result
+                    ]
+                ),
+                auto_arg_index,
+            )


Test patch:
diff --git a/tests/test_varfmt.py b/tests/test_varfmt.py
new file mode 100644
--- /dev/null
+++ b/tests/test_varfmt.py
@@ -0,0 +1,78 @@
+import pytest
+
+from c7n.varfmt import VarFormat
+from c7n.utils import parse_date, format_string_values
+
+
+def test_format_mixed():
+    assert VarFormat().format("{x} abc {Y}", x=2, Y='a') == '2 abc a'
+
+
+def test_format_pass_list():
+    assert VarFormat().format("{x}", x=[1, 2, 3]) == [1, 2, 3]
+
+
+def test_format_pass_int():
+    assert VarFormat().format("{x}", x=2) == 2
+
+
+def test_format_pass_empty():
+    assert VarFormat().format("{x}", x=[]) == []
+    assert VarFormat().format("{x}", x=None) is None
+    assert VarFormat().format("{x}", x={}) == {}
+    assert VarFormat().format("{x}", x=0) == 0
+
+
+def test_format_string_values_empty():
+    formatter = VarFormat().format
+    assert format_string_values({'a': '{x}'}, x=None, formatter=formatter) == {
+        'a': None
+    }
+    assert format_string_values({'a': '{x}'}, x={}, formatter=formatter) == {'a': {}}
+    assert format_string_values({'a': '{x}'}, x=[], formatter=formatter) == {'a': []}
+    assert format_string_values({'a': '{x}'}, x=0, formatter=formatter) == {'a': 0}
+
+
+def test_format_manual_to_auto():
+    # coverage check for stdlib impl behavior
+    with pytest.raises(ValueError) as err:
+        VarFormat().format("{0} {}", 1, 2)
+    assert str(err.value) == (
+        'cannot switch from manual field specification to automatic field numbering'
+    )
+
+
+def test_format_auto_to_manual():
+    # coverage check for stdlib impl behavior
+    with pytest.raises(ValueError) as err:
+        VarFormat().format('{} {1}', 'a', 'b')
+    assert str(err.value) == (
+        'cannot switch from manual field specification to automatic field numbering'
+    )
+
+
+def test_format_date_fmt():
+    d = parse_date("2018-02-02 12:00")
+    assert VarFormat().format("{:%Y-%m-%d}", d, "2018-02-02")
+    assert VarFormat().format("{}", d) == d
+
+
+def test_load_policy_var_retain_type(test):
+    p = test.load_policy(
+        {
+            'name': 'x',
+            'resource': 'aws.sqs',
+            'filters': [
+                {'type': 'value', 'key': 'why', 'op': 'in', 'value': "{my_list}"},
+                {'type': 'value', 'key': 'why_not', 'value': "{my_int}"},
+                {'key': "{my_date:%Y-%m-%d}"},
+            ],
+        }
+    )
+
+    p.expand_variables(
+        dict(my_list=[1, 2, 3], my_int=22, my_date=parse_date('2022-02-01 12:00'))
+    )
+    test.assertJmes('filters[0].value', p.data, [1, 2, 3])
+    test.assertJmes('filters[1].value', p.data, 22)
+    test.assertJmes('filters[2].key', p.data, "2022-02-01")