internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
Running into rate-limit issues with turning on shield protection for resources.
## Running into rate-limit issues with turning on shield protection for resources.
It would be nice to have a retry logic included with turning on shield protection for resources. Currently running into api rate limit issues.


Discussion:

Resolving patch:
diff --git a/c7n/resources/shield.py b/c7n/resources/shield.py
--- a/c7n/resources/shield.py
+++ b/c7n/resources/shield.py
@@ -20,7 +20,7 @@
 from c7n.filters import Filter
 from c7n.manager import resources
 from c7n.query import QueryResourceManager
-from c7n.utils import local_session, type_schema
+from c7n.utils import local_session, type_schema, get_retry
 
 
 @resources.register('shield-protection')
@@ -103,6 +103,8 @@ class SetShieldProtection(BaseAction):
         'set-shield',
         state={'type': 'boolean'}, sync={'type': 'boolean'})
 
+    retry = staticmethod(get_retry(('ThrottlingException',)))
+
     def process(self, resources):
         client = local_session(self.manager.session_factory).client(
             'shield', region_name='us-east-1')
@@ -119,13 +121,14 @@ def process(self, resources):
             if state and arn in protected_resources:
                 continue
             if state is False and arn in protected_resources:
-                client.delete_protection(
+                self.retry(
+                    client.delete_protection,
                     ProtectionId=protected_resources[arn]['Id'])
                 continue
             try:
-                client.create_protection(
-                    Name=r[model.name],
-                    ResourceArn=arn)
+                self.retry(
+                    client.create_protection,
+                    Name=r[model.name], ResourceArn=arn)
             except ClientError as e:
                 if e.response['Error']['Code'] == 'ResourceAlreadyExistsException':
                     continue
@@ -133,7 +136,8 @@ def process(self, resources):
 
     def clear_stale(self, client, protections):
         # Get all resources unfiltered
-        resources = self.manager.get_resource_manager(self.manager.type).resources()
+        resources = self.manager.get_resource_manager(
+            self.manager.type).resources()
         resource_arns = set(map(self.manager.get_arn, resources))
 
         pmap = {}
@@ -148,5 +152,4 @@ def clear_stale(self, client, protections):
         stale = set(pmap).difference(resource_arns)
         self.log.info("clearing %d stale protections", len(stale))
         for s in stale:
-            client.delete_protection(
-                ProtectionId=pmap[s]['Id'])
+            self.retry(client.delete_protection, ProtectionId=pmap[s]['Id'])


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
cannot auto-tag-user s3 buckets with aws: tags on them
The s3 resource should use the resourcegrouptaggingapi instead of put_bucket_tagging, as the latter requires posting the full set of tags.  If you have an s3 bucket managed by cloudformation (which includes `aws:` tags), you get:
```Exception tagging bucket my-s3-bucket: An error occurred (InvalidTag) when calling the PutBucketTagging operation: System Tags cannot be removed by requester
Traceback (most recent call last):
File "/var/task/c7n/resources/s3.py", line 516, in modify_bucket_tags
Bucket=bucket['Name'], Tagging={'TagSet': tag_set})
File "/var/runtime/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (InvalidTag) when calling the PutBucketTagging operation: System Tags cannot be removed by requester
```


Discussion:
It appears that commenting out the line containing:
`not t['Key'].startswith('aws') and`

in the modify_bucket_tags function allows put_bucket_tagging to update the S3 bucket tags that contain the `aws:` tags.  The put_bucket_tags API call will replace tags on the bucket with those in the tag set, which means it will attempt to remove any tags that exist on the bucket, but aren't in the tag set.  Including existing `aws:` tags in the tag set when modifying tags will allow the put_bucket_tagging to succeed.

I don't have enough background to understand why the `aws:` tags were excluded when updating the tag set with the existing bucket tags.
wow.  AWS must have fixed their backend to allow that... we used to get permission denied problems if you were trying to use put-bucket-tagging if any of the tags included an `aws:` tag, even if you weren't changing those tags.
While it is good that their backend change now allows bucket tags to be updated in the presence of an unchanged `aws:` tag, I agree that your initial suggestion to use the resourcegrouptaggingapi would be the preferred approach.  

Let me know if you want me to put together a pull request with this change.
would be great for a pull request, else we'll look at getting to it this week, nutshell its a breaking api change on put bucket tags.

Resolving patch:
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -513,9 +513,7 @@ def modify_bucket_tags(session_factory, buckets, add_tags=(), remove_tags=()):
 
         new_tags = {t['Key']: t['Value'] for t in add_tags}
         for t in bucket.get('Tags', ()):
-            if (t['Key'] not in new_tags and
-                    not t['Key'].startswith('aws') and
-                    t['Key'] not in remove_tags):
+            if (t['Key'] not in new_tags and t['Key'] not in remove_tags):
                 new_tags[t['Key']] = t['Value']
         tag_set = [{'Key': k, 'Value': v} for k, v in new_tags.items()]
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
class MetricsOutput function select(metrics_selector) bug 
Hello,

We found the following issue after updating our cloud custodian code to the latest release: 
```
File "/var/task/c7n/output.py", line 60, in select
if k.startswith(metrics_selector):
TypeError: startswith first arg must be str, unicode, or tuple, not bool 
```

This if statement is defined in the select function in the c7n/output.py . 
This function is called in the ctx.py: 
```
metrics_enabled = getattr(options, 'metrics_enabled', None)
factory = MetricsOutput.select(metrics_enabled)
```

We define metrics_enabled in our yml file the following way: 
```
      execution-options:
          metrics_enabled: true
```
This is the same as how it is documented in docs/source/policy/lambda.rst 

as you can see this is called with a boolean variable, however, the startswith expects a string. Could you please look into this? Previously this was done the following way: 
```
    @staticmethod
    def select(metrics_enabled):
        if metrics_enabled:
            return MetricsOutput
        return NullMetricsOutput
```

Thank you in advance. 

Regards, Peter



Discussion:
yeah the latest release started to support metrics as a string to enable pushing metrics across cross providers. we handled the cli compatibility, but execution-options was missing. This also makes me wonder though that we're not defaulting to metrics output on the lambdas, which actually was an intent/goal for lambda policies, ie this explicit config shouldn't be needed, though it should be supported. i'll have a look at both, thanks for the issue report.


Resolving patch:
diff --git a/c7n/output.py b/c7n/output.py
--- a/c7n/output.py
+++ b/c7n/output.py
@@ -56,6 +56,9 @@ class MetricsOutput(object):
     def select(metrics_selector):
         if not metrics_selector:
             return NullMetricsOutput
+        # Compatibility for boolean configuration
+        if isinstance(metrics_selector, bool):
+            metrics_selector = 'aws'
         for k in metrics_outputs.keys():
             if k.startswith(metrics_selector):
                 return metrics_outputs[k]


Test patch:
diff --git a/tests/test_output.py b/tests/test_output.py
--- a/tests/test_output.py
+++ b/tests/test_output.py
@@ -21,11 +21,17 @@
 import os
 
 from c7n.ctx import ExecutionContext
-from c7n.output import S3Output, DirectoryOutput
+from c7n.output import S3Output, DirectoryOutput, MetricsOutput
 
 from .common import Bag, BaseTest, TestConfig as Config
 
 
+class MetricsTest(BaseTest):
+
+    def test_boolean_config_compatibility(self):
+        self.assertEqual(MetricsOutput.select(True), MetricsOutput)
+
+
 class DirOutputTest(BaseTest):
 
     def get_dir_output(self, location):
Repository: cloud-custodian/cloud-custodian

Issue:
always perform non schema validation for filters/actions doing lazy init

Several filters and actions do lazy initialization in validate method, we should always perform non schema validation. Additionally non schema validation was taking place pre policy expansion by the provider which meant the new policy set returned by the provider was not having validation performed on it.


Discussion:

Resolving patch:
diff --git a/c7n/version.py b/c7n/version.py
--- a/c7n/version.py
+++ b/c7n/version.py
@@ -12,4 +12,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-version = u"0.8.31.1"
+version = u"0.8.31.2"
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@ def read(fname):
 
 setup(
     name="c7n",
-    version='0.8.31.1',
+    version='0.8.31.2',
     description="Cloud Custodian - Policy Rules Engine",
     long_description=read('README.rst'),
     classifiers=[


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
c7n-org - too many values to unpack error
Generated subscriptions.yml with `python tools/c7n_org/scripts/azuresubs.py -f subscriptions.yml`

$ c7n-org run -c subscriptions.yml -s output -u policies/policy.yml
2018-10-25 23:13:11,480: c7n_org:INFO Ran account:aaros team region:global policy:delete-empty-resource-groups matched:5 time:11.35
Traceback (most recent call last):
  File "/Users/andyluong/Projects/forks/cloud-custodian/bin/c7n-org", line 11, in <module>
    load_entry_point('c7n-org', 'console_scripts', 'c7n-org')()
  File "/Users/andyluong/Projects/forks/cloud-custodian/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/andyluong/Projects/forks/cloud-custodian/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/andyluong/Projects/forks/cloud-custodian/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/andyluong/Projects/forks/cloud-custodian/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/andyluong/Projects/forks/cloud-custodian/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/andyluong/Projects/forks/cloud-custodian/tools/c7n_org/c7n_org/cli.py", line 561, in run
    for p, count in account_region_pcounts:
**ValueError: too many values to unpack (expected 2)**


Discussion:

Resolving patch:
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -558,8 +558,8 @@ def run(config, use, output_dir, accounts, tags,
                     a['name'], r, f.exception())
 
             account_region_pcounts, account_region_success = f.result()
-            for p, count in account_region_pcounts:
-                policy_counts[p] += count
+            for p in account_region_pcounts:
+                policy_counts[p] += account_region_pcounts[p]
 
             if not account_region_success:
                 success = False


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
GCP: Firewalls resource policy fails with no resource attribute 'Firewall'
When running this policy custodian fails: 

- policies:
  - name: firewall-test
    resource: gcp.firewall

The error returned is:
AttributeError: 'Resource' object has no attribute 'firewall'


Discussion:
cool, gcp users :-) i'll have a look

Resolving patch:
diff --git a/tools/c7n_gcp/c7n_gcp/resources/network.py b/tools/c7n_gcp/c7n_gcp/resources/network.py
--- a/tools/c7n_gcp/c7n_gcp/resources/network.py
+++ b/tools/c7n_gcp/c7n_gcp/resources/network.py
@@ -43,8 +43,7 @@ class Firewall(QueryResourceManager):
     class resource_type(TypeInfo):
         service = 'compute'
         version = 'v1'
-        component = 'firewall'
-        scope_template = "projects/{}/global/firewalls"
+        component = 'firewalls'
 
 
 @resources.register('router')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Marking emr cluster for termination throws exception
When marking EMR cluster for termination throws exception in version c7n-0.8.31.2. I see the tag created in cluster
`````
policies:
- name: emr-mark-clusters-for-termination
  resource: emr
  filters:
    - type: value
      key: "Id"
      op: in
      value:
        - 'abcdefghij'
  actions:
    - type: mark-for-op
      tag: 'custodian-emr-terminate'
      op: terminate
      days: 4
`````

this policy throws exception 

2018-09-27 19:20:30,262: custodian.actions:INFO Tagging 1 resources for terminate on 2018/10/01
2018-09-27 19:20:31,720: custodian.actions:ERROR Exception with tags: [{u'Value': u'Resource does not meet policy: terminate@2018/10/01', u'Key': 'custodian-emr-terminate'}] on resources: abcdefghij
 'dict' object is not callable
`
Though the EMR is marked with tag ''custodian-emr-terminate', filtering on type: marked-for-op, returns 0 resources.


Discussion:
@kapilt,
Any update on this issue?


Resolving patch:
diff --git a/c7n/resources/emr.py b/c7n/resources/emr.py
--- a/c7n/resources/emr.py
+++ b/c7n/resources/emr.py
@@ -157,7 +157,7 @@ def process_resource_set(self, resources, tags):
         client = local_session(
             self.manager.session_factory).client('emr')
         for r in resources:
-            self.retry(client.add_tags(ResourceId=r['Id'], Tags=tags))
+            self.retry(client.add_tags, ResourceId=r['Id'], Tags=tags)
 
 
 @actions.register('tag')
@@ -186,7 +186,7 @@ class TagTable(Tag):
     def process_resource_set(self, resources, tags):
         client = local_session(self.manager.session_factory).client('emr')
         for r in resources:
-            self.retry(client.add_tags(ResourceId=r['Id'], Tags=tags))
+            self.retry(client.add_tags, ResourceId=r['Id'], Tags=tags)
 
 
 @actions.register('remove-tag')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws iam user docs bugs
https://www.capitalone.io/docs/_modules/c7n/resources/iam.html references type: username when it should be type: value.


Discussion:

Resolving patch:
diff --git a/c7n/resources/iam.py b/c7n/resources/iam.py
--- a/c7n/resources/iam.py
+++ b/c7n/resources/iam.py
@@ -1123,7 +1123,7 @@ class UserDelete(BaseAction):
         - name: iam-only-whitelisted-users
           resource: iam-user
           filters:
-            - type: username
+            - type: value
               key: UserName
               op: not-in
               value:
@@ -1136,7 +1136,7 @@ class UserDelete(BaseAction):
         - name: iam-only-whitelisted-users
           resource: iam-user
           filters:
-            - type: username
+            - type: value
               key: Arn
               op: not-in
               value:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Error loading a file during `custodian run` doesn't return non-zero exit code
When a policy file can't be loaded due to a syntax error, the exit code should be non-zero to indicate an error.
```
(custodian) E:\Projects\cloud-custodian>custodian run --output-dir=. example.yml --region us-west-1
2018-11-16 13:30:00,832: custodian.utils:ERROR Error while loading yaml file example.yml
2018-11-16 13:30:00,835: custodian.utils:ERROR Skipping this file.  Error message below:
mapping values are not allowed in this context
  in "<byte string>", line 58, column 15
2018-11-16 13:30:00,835: custodian.commands:WARNING Empty policy file(s).  Nothing to do.

(custodian) E:\Projects\cloud-custodian>echo %errorlevel%
0
```

And here is a minimal `example.yml` that reproduces the issue. 
The syntax error is the missing trailing colon after `- or`:
```yml
policies:
- name: example
  filters:
  - or
    - type: value
      key: '"c7n:AccessKeys"[?Status == ''Active''].CreateDate | [0]'
      op: gt
      value_type: age
      value: 90
    - type: value
      key: '"c7n:AccessKeys"[?Status == ''Active''].CreateDate | [1]'
      op: gt
      value_type: age
      value: 90
```


Discussion:

Resolving patch:
diff --git a/c7n/commands.py b/c7n/commands.py
--- a/c7n/commands.py
+++ b/c7n/commands.py
@@ -60,16 +60,22 @@ def _load_policies(options):
 
         # for a default region for policy loading, we'll expand regions later.
         options.region = ""
-
         for fp in options.configs:
             try:
                 collection = policy_load(options, fp, validate=validate, vars=vars)
-            except IOError:
+            except IOError as e:
                 log.error('policy file does not exist ({})'.format(fp))
                 errors += 1
                 continue
+            except yaml.YAMLError as e:
+                log.error(
+                    "yaml syntax error loading policy file ({}) error:\n {}".format(
+                        fp, e))
+                errors += 1
+                continue
             except ValueError as e:
-                log.error('problem loading policy file ({})'.format(e.message))
+                log.error('problem loading policy file ({}) error: {}'.format(
+                    fp, e.message))
                 errors += 1
                 continue
 
diff --git a/c7n/utils.py b/c7n/utils.py
--- a/c7n/utils.py
+++ b/c7n/utils.py
@@ -93,12 +93,7 @@ def load_file(path, format=None, vars=None):
                 raise VarsSubstitutionError(msg)
 
         if format == 'yaml':
-            try:
-                return yaml_load(contents)
-            except yaml.YAMLError as e:
-                log.error('Error while loading yaml file %s', path)
-                log.error('Skipping this file.  Error message below:\n%s', e)
-                return None
+            return yaml_load(contents)
         elif format == 'json':
             return loads(contents)
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
getting AttributeError: 'module' object has no attribute 'get_zonefile_instance' error
after installing with pip , i get this error on very basic policy 

what am i doing wrong ?

custodian validate ebs_create_snaps.yml
Traceback (most recent call last):
  File "/usr/bin/custodian", line 9, in <module>
    load_entry_point('c7n==0.8.32.1', 'console_scripts', 'custodian')()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python2.7/site-packages/c7n/cli.py", line 36, in <module>
    from c7n.commands import schema_completer
  File "/usr/lib/python2.7/site-packages/c7n/commands.py", line 31, in <module>
    from c7n.policy import Policy, PolicyCollection, load as policy_load
  File "/usr/lib/python2.7/site-packages/c7n/policy.py", line 37, in <module>
    from c7n import query
  File "/usr/lib/python2.7/site-packages/c7n/query.py", line 35, in <module>
    from c7n.tags import register_ec2_tags, register_universal_tags
  File "/usr/lib/python2.7/site-packages/c7n/tags.py", line 36, in <module>
    from c7n.filters.offhours import Time
  File "/usr/lib/python2.7/site-packages/c7n/filters/offhours.py", line 261, in <module>
    class Time(Filter):
  File "/usr/lib/python2.7/site-packages/c7n/filters/offhours.py", line 314, in Time
    z_names = list(zoneinfo.get_zonefile_instance().zones)
AttributeError: 'module' object has no attribute 'get_zonefile_instance'



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,8 @@ def read(fname):
             'custodian = c7n.cli:main']},
     install_requires=[
         "boto3>=1.9.54",
-        "botocore>=1.12.54",
+        "botocore>=1.12.57",
+        "python-dateutil>=2.6,<3.0.0",
         "pyyaml",
         "jsonschema",
         "jsonpatch>=1.21",
diff --git a/tools/c7n_mailer/setup.py b/tools/c7n_mailer/setup.py
--- a/tools/c7n_mailer/setup.py
+++ b/tools/c7n_mailer/setup.py
@@ -22,6 +22,7 @@
     "Jinja2",
     "boto3",
     "jsonschema",
+    "python-dateutil>=2.6",
     "ruamel.yaml==0.15.42",
     "datadog",
     "slackclient",


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
mugc.py raises a TypeError if no -c option is provided
Calling mugc.py from the command line throws a TypeError if called without a `-c` option: 
```
Traceback (most recent call last):
  File "mugc.py", line 207, in <module>
    main()
  File "mugc.py", line 186, in main
    files.extend(itertools.chain(*options.config_files))
TypeError: type object argument after * must be an iterable, not NoneType
```

I feel like this should fail gracefully and print the help message along with an error message that tells the user the `-c` flag is required.

The argparse library supports this functionality with the `required` kwarg on the `ArgumentParser`'s `add_argument` function. I've cloned the repo and will have a pull request for this shortly.


Discussion:

Resolving patch:
diff --git a/tools/ops/mugc.py b/tools/ops/mugc.py
--- a/tools/ops/mugc.py
+++ b/tools/ops/mugc.py
@@ -144,7 +144,7 @@ def setup_parser():
     parser.add_argument("configs", nargs='*', help="Policy configuration file(s)")
     parser.add_argument(
         '-c', '--config', dest="config_files", nargs="*", action='append',
-        help="Policy configuration files(s)")
+        help="Policy configuration files(s)", default=[])
     parser.add_argument(
         '-r', '--region', action='append', dest='regions', metavar='REGION',
         help="AWS Region to target. Can be used multiple times, also supports `all`")
@@ -198,7 +198,12 @@ def main():
 
     # use cloud provider to initialize policies to get region expansion
     policies = AWS().initialize_policies(
-        load_policies(options, policy_config), policy_config)
+        PolicyCollection([
+            p for p in load_policies(
+                options, policy_config)
+            if p.provider_name == 'aws'],
+            policy_config),
+        policy_config)
 
     resources_gc_prefix(options, policy_config, policies)
 


Test patch: