internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
S3 has-statement filter is missing interpolation
When using has-statement filter with Resource parameter, using {bucket_name} in filter makes it fail. During statement parameters comparison, bucket_name is not interpolated and the filter never match. 


Discussion:
yeah we only do global interpolation on a handful of variables (account_id, etc). specific filters and actions do support additional ones. it seems useful to add in bucket name, but note that its possible to use various suffixes in statements, so exact matching is pretty limited in utility.
Yes, I saw that the bucket_name interpolation is done on set-statements action. 

Resolving patch:
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -731,6 +731,16 @@ def process(self, buckets):
             return results
 
 
+class BucketFilterBase(Filter):
+    def get_std_format_args(self, bucket):
+        return {
+            'account_id': self.manager.config.account_id,
+            'region': self.manager.config.region,
+            'bucket_name': bucket['Name'],
+            'bucket_region': get_region(bucket)
+        }
+
+
 @S3.action_registry.register("post-finding")
 class BucketFinding(PostFinding):
     def format_resource(self, r):
@@ -750,7 +760,7 @@ def format_resource(self, r):
 
 
 @filters.register('has-statement')
-class HasStatementFilter(Filter):
+class HasStatementFilter(BucketFilterBase):
     """Find buckets with set of policy statements.
 
     :example:
@@ -820,7 +830,8 @@ def process_bucket(self, b):
             if s.get('Sid') in required:
                 required.remove(s['Sid'])
 
-        required_statements = list(self.data.get('statements', []))
+        required_statements = format_string_values(list(self.data.get('statements', [])),
+                                                   **self.get_std_format_args(b))
         for required_statement in required_statements:
             for statement in statements:
                 found = 0


Test patch:
diff --git a/tests/test_s3.py b/tests/test_s3.py
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -1486,6 +1486,7 @@ def test_has_statement_policy(self):
                                 "Effect": "Deny",
                                 "Action": "s3:PutObject",
                                 "Principal": "*",
+                                "Resource": "arn:aws:s3:::{bucket_name}/*"
                             }
                         ],
                     },
Repository: cloud-custodian/cloud-custodian

Issue:
c7n-mailer-replay : Python3 - TypeError: write() argument must be str, not bytes
Using c7n-mailer-replay under Python3 gives the following trace:
```
Traceback (most recent call last):
  File "HOME/.pyenv/versions/cloud-custodian-3.6/bin/c7n-mailer-replay", line 11, in <module>
    load_entry_point('c7n-mailer', 'console_scripts', 'c7n-mailer-replay')()
  File "HOME/CLOUD_CUSTODIAN/SRC/tools/c7n_mailer/c7n_mailer/replay.py", line 134, in main
    json_dump_file=options.json_dump_file
  File "HOME/CLOUD_CUSTODIAN/SRC/tools/c7n_mailer/c7n_mailer/replay.py", line 46, in __init__
    fh.write(raw)
TypeError: write() argument must be str, not bytes
```

I had success with the following change:
```diff
diff --git a/tools/c7n_mailer/c7n_mailer/replay.py b/tools/c7n_mailer/c7n_mailer/replay.py
index b3f5456be..72f63332f 100644
--- a/tools/c7n_mailer/c7n_mailer/replay.py
+++ b/tools/c7n_mailer/c7n_mailer/replay.py
@@ -42,7 +42,7 @@ class MailerTester(object):
             logger.debug('base64-decoding and zlib decompressing message')
             raw = zlib.decompress(base64.b64decode(raw))
             if json_dump_file is not None:
-                with open(json_dump_file, 'w') as fh:
+                with open(json_dump_file, 'wb') as fh:
                     fh.write(raw)
         self.data = json.loads(raw)
         logger.debug('Loaded message JSON')
```

I believe it could be compatible with Python2 also, but it needs some testing.


Discussion:
thanks for the bug report, that looks reasonable, although, we should probably be setting the codec explicitly to utf8 as an alternative.

Resolving patch:
diff --git a/tools/c7n_mailer/c7n_mailer/replay.py b/tools/c7n_mailer/c7n_mailer/replay.py
--- a/tools/c7n_mailer/c7n_mailer/replay.py
+++ b/tools/c7n_mailer/c7n_mailer/replay.py
@@ -42,7 +42,7 @@ def __init__(self, msg_file, config, msg_plain=False, json_dump_file=None):
             logger.debug('base64-decoding and zlib decompressing message')
             raw = zlib.decompress(base64.b64decode(raw))
             if json_dump_file is not None:
-                with open(json_dump_file, 'w') as fh:
+                with open(json_dump_file, 'wb') as fh:  # pragma: no cover
                     fh.write(raw)
         self.data = json.loads(raw)
         logger.debug('Loaded message JSON')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
tools/c7n_mailer support jmespath search in mailer templates
From #3658  "By the way, I have the same need in notification templates (mails, slack, ...) Is there the same JMESPath functionality available in the Jinja2 templating solution used there ? Or should we use the Python/Jinja2 vocabulary to express the same display needs as in the CSV with "--field" ?"




Discussion:

Resolving patch:
diff --git a/tools/c7n_mailer/c7n_mailer/azure/deploy.py b/tools/c7n_mailer/c7n_mailer/azure/deploy.py
--- a/tools/c7n_mailer/c7n_mailer/azure/deploy.py
+++ b/tools/c7n_mailer/c7n_mailer/azure/deploy.py
@@ -41,7 +41,7 @@ def build_function_package(config, function_name):
 
     package.build(None,
                   modules=['c7n', 'c7n-azure', 'c7n-mailer', 'applicationinsights'],
-                  non_binary_packages=['pyyaml', 'pycparser', 'tabulate',
+                  non_binary_packages=['pyyaml', 'pycparser', 'tabulate', 'jmespath',
                                        'datadog', 'MarkupSafe', 'simplejson', 'pyrsistent'],
                   excluded_packages=['azure-cli-core', 'distlib', 'future', 'futures'])
 
diff --git a/tools/c7n_mailer/c7n_mailer/utils.py b/tools/c7n_mailer/c7n_mailer/utils.py
--- a/tools/c7n_mailer/c7n_mailer/utils.py
+++ b/tools/c7n_mailer/c7n_mailer/utils.py
@@ -19,6 +19,7 @@
 from datetime import datetime, timedelta
 
 import jinja2
+import jmespath
 from botocore.exceptions import ClientError
 from dateutil import parser
 from dateutil.tz import gettz, tzutc
@@ -35,6 +36,7 @@ def get_jinja_env(template_folders):
     env.globals['format_struct'] = format_struct
     env.globals['resource_tag'] = get_resource_tag_value
     env.globals['get_resource_tag_value'] = get_resource_tag_value
+    env.globals['search'] = jmespath.search
     env.loader = jinja2.FileSystemLoader(template_folders)
     return env
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws - ec2 key-pair not taggable
we auto-attach taggable to ec2 resources, but not all ec2 resources are taggable, the notable one in this case being key pair

Full list of  taggable ec2 resources
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html


Discussion:

Resolving patch:
diff --git a/c7n/resources/vpc.py b/c7n/resources/vpc.py
--- a/c7n/resources/vpc.py
+++ b/c7n/resources/vpc.py
@@ -1854,6 +1854,7 @@ class resource_type(object):
         filter_type = 'list'
         dimension = None
         id_prefix = "vpce-"
+        taggable = False
 
 
 @VpcEndpoint.filter_registry.register('cross-account')
@@ -1895,6 +1896,7 @@ class resource_type(object):
         name = 'KeyName'
         date = None
         dimension = None
+        taggable = False
 
 
 @Vpc.action_registry.register('set-flow-log')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
gcp serverless runtime error on implicit boto dependency
reported in gitter, gcp functions should not need to depend on boto3, looks like some of the securityhub work caused an implicit dependency on boto3.

```
textPayload: "ModuleNotFoundError: No module named 'boto3'" - Getting this error for the cloud function to stop a instance in GCP
instance-off
qte7iow5dhzi
Traceback (most recent call last): File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 346, in run_http_function result = _function_handler.invoke_user_function(flask.request) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 217, in invoke_user_function return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 210, in call_user_function return self._user_function(request_or_event) File "/user_code/main.py", line 21, in run from c7n_gcp.handler import run File "/user_code/c7n_gcp/handler.py", line 24, in <module> from c7n_gcp.entry import initialize_gcp File "/user_code/c7n_gcp/entry.py", line 18, in <module> import c7n_gcp.resources.bigquery File "/user_code/c7n_gcp/resources/bigquery.py", line 16, in <module> from c7n_gcp.query import QueryResourceManager, TypeInfo File "/user_code/c7n_gcp/query.py", line 23, in <module> from c7n.filters import FilterRegistry File "/user_code/c7n/filters/init.py", line 32, in <module> from .securityhub import SecurityHubFindingFilter File "/user_code/c7n/filters/securityhub.py", line 19, in <module> from c7n.resources import aws File "/user_code/c7n/resources/aws.py", line 31, in <module> import boto3 ModuleNotFoundError: No module named 'boto3
```


Discussion:

Resolving patch:
diff --git a/c7n/filters/securityhub.py b/c7n/filters/securityhub.py
--- a/c7n/filters/securityhub.py
+++ b/c7n/filters/securityhub.py
@@ -16,7 +16,6 @@
 from c7n.utils import local_session, type_schema
 from .core import Filter
 from c7n.manager import resources
-from c7n.resources import aws
 
 
 class SecurityHubFindingFilter(Filter):
@@ -36,6 +35,7 @@ class SecurityHubFindingFilter(Filter):
     def validate(self):
         query = self.data.get('query')
         if query:
+            from c7n.resources import aws
             aws.shape_validate(query, self.query_shape, 'securityhub')
 
     def process(self, resources, event=None):


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Azure - c7n-Mailer Errors
About 50% of the time mailer runs, the following error results and messages aren't picked up, delivered:
```
Traceback (most recent call last):
  File "/usr/local/bin/c7n-mailer", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/c7n_mailer/cli.py", line 227, in main
    processor.run()
  File "/usr/local/lib/python3.7/site-packages/c7n_mailer/azure/azure_queue_processor.py", line 62, in run
    if (self.process_azure_queue_message(queue_message) or
  File "/usr/local/lib/python3.7/site-packages/c7n_mailer/azure/azure_queue_processor.py", line 89, in process_azure_queue_message
    SendGridDelivery(self.config, self.logger))
  File "/usr/local/lib/python3.7/site-packages/c7n_mailer/azure/sendgrid_delivery.py", line 29, in __init__
    sendgrid.SendGridAPIClient(apikey=self.config.get('sendgrid_api_key', ''))
TypeError: __init__() got an unexpected keyword argument 'apikey'
```



Discussion:
hmm.. that implies a rev on the send grid api thats hosing delivery
looks like it changed a while ago to be `api_key`  per https://github.com/sendgrid/sendgrid-python/pull/557/files

Resolving patch:
diff --git a/tools/c7n_mailer/c7n_mailer/azure/sendgrid_delivery.py b/tools/c7n_mailer/c7n_mailer/azure/sendgrid_delivery.py
--- a/tools/c7n_mailer/c7n_mailer/azure/sendgrid_delivery.py
+++ b/tools/c7n_mailer/c7n_mailer/azure/sendgrid_delivery.py
@@ -26,7 +26,7 @@ def __init__(self, config, logger):
         self.config = config
         self.logger = logger
         self.sendgrid_client = \
-            sendgrid.SendGridAPIClient(apikey=self.config.get('sendgrid_api_key', ''))
+            sendgrid.SendGridAPIClient(self.config.get('sendgrid_api_key', ''))
 
     def get_to_addrs_sendgrid_messages_map(self, queue_message):
         # eg: { ('milton@initech.com', 'peter@initech.com'): [resource1, resource2, etc] }


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
tools/c7n_mailer: Support for Date/Time stamp in the email
Is there a way we can add date/time stamps in the email sent out automatically based on when the notification was created by the policy

This is because we have a huge backlog on our mailer due to many policies. 

Thank you


Discussion:

Resolving patch:
diff --git a/c7n/actions/notify.py b/c7n/actions/notify.py
--- a/c7n/actions/notify.py
+++ b/c7n/actions/notify.py
@@ -178,6 +178,8 @@ def process(self, resources, event=None):
             'account_id': self.manager.config.account_id,
             'account': alias,
             'region': self.manager.config.region,
+            'execution_id': self.manager.ctx.execution_id,
+            'execution_start': self.manager.ctx.start_time,
             'policy': self.manager.data}
         message['action'] = self.expand_variables(message)
 
diff --git a/tools/c7n_mailer/c7n_mailer/utils.py b/tools/c7n_mailer/c7n_mailer/utils.py
--- a/tools/c7n_mailer/c7n_mailer/utils.py
+++ b/tools/c7n_mailer/c7n_mailer/utils.py
@@ -14,9 +14,11 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 import base64
+from datetime import datetime, timedelta
+import functools
 import json
 import os
-from datetime import datetime, timedelta
+import time
 
 import jinja2
 import jmespath
@@ -28,7 +30,7 @@
 
 def get_jinja_env(template_folders):
     env = jinja2.Environment(trim_blocks=True, autoescape=False)
-    env.filters['yaml_safe'] = yaml.safe_dump
+    env.filters['yaml_safe'] = functools.partial(yaml.safe_dump, default_flow_style=False)
     env.filters['date_time_format'] = date_time_format
     env.filters['get_date_time_delta'] = get_date_time_delta
     env.filters['get_date_age'] = get_date_age
@@ -53,6 +55,17 @@ def get_rendered_jinja(
     except Exception as error_msg:
         logger.error("Invalid template reference %s\n%s" % (mail_template, error_msg))
         return
+
+    # recast seconds since epoch as utc iso datestring, template
+    # authors can use date_time_format helper func to convert local
+    # tz. if no execution start time was passed use current time.
+    execution_start = datetime.utcfromtimestamp(
+        sqs_message.get(
+            'execution_start',
+            time.mktime(
+                datetime.utcnow().timetuple())
+        )).isoformat()
+
     rendered_jinja = template.render(
         recipient=target,
         resources=resources,
@@ -61,6 +74,7 @@ def get_rendered_jinja(
         event=sqs_message.get('event', None),
         action=sqs_message['action'],
         policy=sqs_message['policy'],
+        execution_start=execution_start,
         region=sqs_message.get('region', ''))
     return rendered_jinja
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
AWS RDS set-snapshot-copy-tags  'dict' object is not callable
When I execute this policy

```
- name: enable-rds-snapshot-tags
  resource: aws.rds
  actions:
    - type: set-snapshot-copy-tags
      enable: True
```
I get the following error:

2019-04-25 09:02:53,231: custodian.policy:INFO policy: enable-rds-snapshot-tags resource:aws.rds region:ca-central-1 count:1 time:0.54
2019-04-25 09:02:53,686: custodian.actions:ERROR Exception updating rds CopyTagsToSnapshot
 'dict' object is not callable
2019-04-25 09:02:53,688: custodian.policy:INFO policy: enable-rds-snapshot-tags action: copysnapshottags resources: 1 execution_time: 0.46


Discussion:
fwiw its probably better to use the more generic `modify-db` action to set "CopyTagsToSnapshot"

Resolving patch:
diff --git a/c7n/resources/rds.py b/c7n/resources/rds.py
--- a/c7n/resources/rds.py
+++ b/c7n/resources/rds.py
@@ -649,6 +649,8 @@ def copy_restore_info(self, client, instance):
 class CopySnapshotTags(BaseAction):
     """Enables copying tags from rds instance to snapshot
 
+    DEPRECATED - use modify-db instead with `CopyTagsToSnapshot`
+
         .. code-block: yaml
 
             policies:
@@ -670,23 +672,29 @@ class CopySnapshotTags(BaseAction):
     permissions = ('rds:ModifyDBInstances',)
 
     def process(self, resources):
+        error = None
         with self.executor_factory(max_workers=2) as w:
-            futures = []
+            futures = {}
+            client = local_session(self.manager.session_factory).client('rds')
+            resources = [r for r in resources
+                         if r['CopyTagsToSnapshot'] != self.data.get('enable', True)]
             for r in resources:
-                futures.append(w.submit(
-                    self.set_snapshot_tags, r))
+                futures[w.submit(self.set_snapshot_tags, client, r)] = r
             for f in as_completed(futures):
                 if f.exception():
+                    error = f.exception()
                     self.log.error(
-                        'Exception updating rds CopyTagsToSnapshot  \n %s',
-                        f.exception())
+                        'error updating rds:%s CopyTagsToSnapshot \n %s',
+                        futures[f]['DBInstanceIdentifier'], error)
+        if error:
+            raise error
         return resources
 
-    def set_snapshot_tags(self, r):
-        c = local_session(self.manager.session_factory).client('rds')
-        self.manager.retry(c.modify_db_instance(
+    def set_snapshot_tags(self, client, r):
+        self.manager.retry(
+            client.modify_db_instance,
             DBInstanceIdentifier=r['DBInstanceIdentifier'],
-            CopyTagsToSnapshot=self.data.get('enable', True)))
+            CopyTagsToSnapshot=self.data.get('enable', True))
 
 
 @RDS.action_registry.register("post-finding")


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
The copy-related-tag action copies restricted aws tags
When the [copy-related-tag action](https://cloudcustodian.io/docs/generated/aws/c7n.html#c7n.tags.CopyRelatedResourceTag) is configured to copy all tags from the related resource, it does not filter out the [special tags prefixed with _aws_](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html) .  The action will fail when it attempts to add the special tags to the target resource.  For example, instances launched by an autoscaling group have the tag `aws:autoscaling:groupName` added to them automatically.  The example policy listed below will fail when it tries to copy the tags from an instance in an autoscaling group.

```yaml
policies:
    - name: copy-tags-from-ec2-instance-to-eni
      resource: eni
      actions:
        - type: copy-related-tag
          resource: ec2 
          skip_missing: True
          key: Attachment.InstanceId
          tags: '*'
```

The copy-instance-tags action for EBS resources, however, [correctly filters out the aws tags](https://github.com/cloud-custodian/cloud-custodian/blob/430357a58ac1e38504bbf4eecc27d49c5be53408/c7n/resources/ebs.py#L837).


Discussion:
Thanks for the bug report, I’ll put a fix in for this soon

Resolving patch:
diff --git a/c7n/tags.py b/c7n/tags.py
--- a/c7n/tags.py
+++ b/c7n/tags.py
@@ -1053,7 +1053,9 @@ def process(self, resources):
 
     def process_resource(self, client, r, related_tags, tag_keys, tag_action):
         tags = {}
-        resource_tags = {t['Key']: t['Value'] for t in r.get('Tags', [])}
+        resource_tags = {
+            t['Key']: t['Value'] for t in r.get('Tags', []) if not t['Key'].startswith('aws:')}
+
         if tag_keys == '*':
             tags = {k: v for k, v in related_tags.items()
                     if resource_tags.get(k) != v}


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
metrics - use a different namespace for emitting metrics
the metrics option currently outputs to the default `CloudMaid` namespace, I would like to output metrics to a specific namespace for better organization of metrics, I propose that we modify c7n.output to allow the use of an environment variable for this:

```python
import os
...
DEFAULT_NAMESPACE = os.environ.get('C7N_METRICS_NAMESPACE', 'CloudMaid')
```

Use Case:
- Policies are contributed by multiple teams that may have name conflicts--  these metrics should be put into their own namespaces such that they can be properly attributed to the right policy

This would change would preserve the current default namespace keeping backwards compatibility


Discussion:
thanks for the use case.. given we have configurable metrics via cli  uri, better would just be a cli parameter to that uri... ie these are all separate invocations.l. And it avoids a provider specific namespace grab on env vars.
fwiw, metrics namespace is already configurable via the cli

`-m aws://?namespace=foo`
Ah, gotcha thanks

I wasn't able to find that info in the cli, I'll submit a patch to add that into the help section and into the docs if it's also missing in there

Resolving patch:
diff --git a/c7n/cli.py b/c7n/cli.py
--- a/c7n/cli.py
+++ b/c7n/cli.py
@@ -240,10 +240,15 @@ def setup_parser():
         "--skip-validation",
         action="store_true",
         help="Skips validation of policies (assumes you've run the validate command seperately).")
+
+    metrics_help = ("Emit metrics to provider metrics. Specify 'aws', 'gcp', or 'azure'. "
+            "For more details on aws metrics options, see: "
+            "https://cloudcustodian.io/docs/aws/usage.html#metrics")
+
     run.add_argument(
         "-m", "--metrics-enabled",
         default=None, nargs="?", const="aws",
-        help="Emit metrics to provider metrics")
+        help=metrics_help)
     run.add_argument(
         "--trace",
         dest="tracer",


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
azure - unpinn EventGrid SDK version
We need AdvancedFilters to be added to the stable version.

https://pypi.org/project/azure-mgmt-eventgrid/


Discussion:
fwiw, i removed some existing pins on azure provider to get past the pyyaml upgrade. re adal, azure-core bits, tests all passed..

Resolving patch:
diff --git a/tools/c7n_azure/setup.py b/tools/c7n_azure/setup.py
--- a/tools/c7n_azure/setup.py
+++ b/tools/c7n_azure/setup.py
@@ -63,7 +63,7 @@
                       "azure-mgmt-web",
                       "azure-mgmt-monitor",
                       "azure-mgmt-policyinsights",
-                      "azure-mgmt-eventgrid==2.0.0rc2",  # RC2 supports AdvancedFilters
+                      "azure-mgmt-eventgrid",
                       "azure-graphrbac",
                       "azure-keyvault",
                       "azure-storage-blob",


Test patch: