internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
c7n-org config file requires 'tags' to be defined, unclear as to why
Leaving out tags in the config file results in the following error:

```
 c7n-org run --debug -c config/accounts.yml  -a tscloud -u policies/tscloud/tag-compliance.yml  -s /tmp/c7n --region us-west-2
Traceback (most recent call last):
  File "/home/cbouscal/.local/bin/c7n-org", line 11, in <module>
    sys.exit(cli())
  File "/usr/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/cbouscal/.local/lib/python2.7/site-packages/c7n_org/cli.py", line 387, in run
    debug)] = (a, r)
  File "/usr/lib/python2.7/site-packages/c7n/executor.py", line 63, in submit
    return MainThreadFuture(func(*args, **kw))
  File "/home/cbouscal/.local/lib/python2.7/site-packages/c7n_org/cli.py", line 334, in run_account
    with environ(**account_tags(account)):
  File "/home/cbouscal/.local/lib/python2.7/site-packages/c7n_org/utils.py", line 6, in account_tags
    for t in account.get('tags'):
TypeError: 'NoneType' object is not iterable
```


Discussion:
yeah, thats an oversight, it should be optional.

Resolving patch:
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -156,7 +156,7 @@ def report_account(account, region, policies_config, output_path, debug):
             r['policy'] = p.name
             r['region'] = p.options.region
             r['account'] = account['name']
-            for t in account['tags']:
+            for t in account.get('tags', ()):
                 if ':' in t:
                     k, v = t.split(':', 1)
                     r[k] = v
diff --git a/tools/c7n_org/c7n_org/utils.py b/tools/c7n_org/c7n_org/utils.py
--- a/tools/c7n_org/c7n_org/utils.py
+++ b/tools/c7n_org/c7n_org/utils.py
@@ -1,16 +1,18 @@
 import os
 from contextlib import contextmanager
 
+
 def account_tags(account):
     tags = {'AccountName': account['name'], 'AccountId': account['account_id']}
-    for t in account.get('tags'):
-        if not ':' in t:
+    for t in account.get('tags', ()):
+        if ':' not in t:
             continue
         k, v = t.split(':', 1)
         k = 'Account%s' % k.capitalize()
         tags[k] = v
     return tags
 
+
 @contextmanager
 def environ(**kw):
     current_env = dict(os.environ)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
The Notify action should use a deepcopy of the policy.
The Notify action should use a deepcopy to load data from `to_from` into `to` when sending messages.

Currently c7n-org emails a growing list of recipients because of a shallow copy of the policy data.


Discussion:

Resolving patch:
diff --git a/c7n/actions.py b/c7n/actions.py
--- a/c7n/actions.py
+++ b/c7n/actions.py
@@ -17,6 +17,7 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
 import base64
+import copy
 from datetime import datetime
 import jmespath
 import logging
@@ -448,7 +449,7 @@ def get_permissions(self):
     def expand_variables(self, message):
         """expand any variables in the action to_from/cc_from fields.
         """
-        p = self.data.copy()
+        p = copy.deepcopy(self.data)
         if 'to_from' in self.data:
             to_from = self.data['to_from'].copy()
             to_from['url'] = to_from['url'].format(**message)


Test patch:
diff --git a/tests/test_notify.py b/tests/test_notify.py
--- a/tests/test_notify.py
+++ b/tests/test_notify.py
@@ -44,6 +44,7 @@ def test_notify_address_from(self):
                 {'QueueUrl': queue_url}],
             'actions': [{
                 'type': 'notify',
+                'to': ['to@example.com'],
                 'to_from': {
                     'url': 'file://%s' % temp_file.name,
                     'format': 'json',
@@ -57,6 +58,7 @@ def test_notify_address_from(self):
                     'queue': queue_url}}]
             }, session_factory=session_factory)
         resources = policy.run()
+        self.assertEqual(policy.data.get('actions')[0].get('to'), ['to@example.com'])
         self.assertEqual(len(resources), 1)
         messages = client.receive_message(
             QueueUrl=queue_url,
Repository: cloud-custodian/cloud-custodian

Issue:
provisioning lambda to vpc appears to be broken still
Hi,

I've tested https://github.com/capitalone/cloud-custodian/pull/1919.

I see the validation now passes, but it looks like the vpc attributes are not being passed when the lambda is created.

Policy:
```
policies:
- name: vpc-test-sandbox
  resource: ec2
  mode:
    type: config-rule
    role: arn:aws:iam::{{ACCOUNT}}:role/service-role/tscloud_lambda_role
    timeout: 180
    security_groups: [sg-ea399290]
    subnets: [subnet-3c3d8367,subnet-a09275c6,subnet-278bb56e]
  description: |
    Testing vpc provisioning
  filters:
    - "tag:c7n_testing": present
  actions:
    - type: mark-for-op
      tag: c7n_tag_compliance
      op: terminate
      days: 1
```

Deploy:
```
$ custodian run -s /tmp/c7n --cache-period 0 vpc-test.yml 
2018-01-18 07:48:42,873: custodian.policy:INFO Provisioning policy lambda vpc-test-sandbox
2018-01-18 07:48:43,233: custodian.lambda:INFO Publishing custodian policy lambda function custodian-vpc-test-sandbox
```
 
Get function vpc config:
```
$ aws lambda get-function --function-name custodian-vpc-test-sandbox | jq '.Configuration.VpcConfig'
null
Expected result:

$ aws lambda get-function --function-name custodian-vpc-test-sandbox | jq '.Configuration.VpcConfig'
{
  "SubnetIds": [
    "subnet-3c3d8367",
    "subnet-a09275c6",
    "subnet-278bb56e"
  ],
  "SecurityGroupIds": [
    "sg-ea399290"
  ],
  "VpcId": "vpc-d3519bb5"
}
```

The role I'm using has access to vpc provisioning, as far as I can tell, using the canned AWS Policy AWSLambdaVPCAccessExecutionRole. I'm able to use the role to add vpc config manually after provisioning with c7n.


Discussion:

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -707,11 +707,11 @@ def timeout(self):
 
     @property
     def security_groups(self):
-        return None
+        return self.policy.data['mode'].get('security_groups', None)
 
     @property
     def subnets(self):
-        return None
+        return self.policy.data['mode'].get('subnets', None)
 
     @property
     def dead_letter_config(self):


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
s3 has-statement does an incomplete match between the custodian policy and bucket policy
reference: https://github.com/capitalone/cloud-custodian/pull/1988/files#r168613005


Discussion:

Resolving patch:
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -770,12 +770,11 @@ def process_bucket(self, b):
         required_statements = list(self.data.get('statements', []))
         for required_statement in required_statements:
             for statement in statements:
-                found = False
+                found = 0
                 for key, value in required_statement.items():
                     if key in statement and value == statement[key]:
-                        found = True
-                        break
-                if found:
+                        found += 1
+                if found and found == len(required_statement):
                     required_statements.remove(required_statement)
 
         if (self.data.get('statement_ids', []) and not required) or \


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Extend age filter to handle Unix epoch values
The current age filter for handling dates does not correctly parse Unix epoch integers, and it is not possible to set up an age based filter for date/time fields returning in this format. 


Discussion:

Resolving patch:
diff --git a/c7n/filters/core.py b/c7n/filters/core.py
--- a/c7n/filters/core.py
+++ b/c7n/filters/core.py
@@ -444,7 +444,11 @@ def process_value_type(self, sentinel, value, resource):
         elif self.vtype == 'age':
             if not isinstance(sentinel, datetime.datetime):
                 sentinel = datetime.datetime.now(tz=tzutc()) - timedelta(sentinel)
-
+            if isinstance(value, (int, float)):
+                try:
+                    value = datetime.datetime.fromtimestamp(value).replace(tzinfo=tzutc())
+                except ValueError:
+                    value = 0
             if not isinstance(value, datetime.datetime):
                 # EMR bug when testing ages in EMR. This is due to
                 # EMR not having more functionality.
@@ -452,7 +456,6 @@ def process_value_type(self, sentinel, value, resource):
                     value = parse(value, default=datetime.datetime.now(tz=tzutc()))
                 except (AttributeError, TypeError, ValueError):
                     value = 0
-
             # Reverse the age comparison, we want to compare the value being
             # greater than the sentinel typically. Else the syntax for age
             # comparisons is intuitively wrong.


Test patch:
diff --git a/tests/test_filters.py b/tests/test_filters.py
--- a/tests/test_filters.py
+++ b/tests/test_filters.py
@@ -13,9 +13,9 @@
 # limitations under the License.
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-from dateutil import tz
-
+import calendar
 from datetime import datetime, timedelta
+from dateutil import tz
 import unittest
 
 from c7n import filters as base_filters
@@ -98,7 +98,7 @@ def test_and(self):
 
 
 class TestNotFilter(unittest.TestCase):
-    
+
     def test_not(self):
 
         results = [
@@ -112,7 +112,7 @@ def test_not(self):
                 {'Architecture': 'x86_64'},
                 {'Color': 'green'}]})
         self.assertEqual(len(f.process(results)), 2)
-        
+
         """
         f = filters.factory({
             'not': [
@@ -154,9 +154,6 @@ def test_value_type(self):
         vf.vtype = 'size'
         res = vf.process_value_type(sentinel, value, resource)
         self.assertEqual(res, (sentinel, 0))
-        vf.vtype = 'age'
-        res = vf.process_value_type(sentinel, value, resource)
-        self.assertEqual(res, (0, sentinel))
         vf.vtype = 'cidr'
         sentinel = '10.0.0.0/16'
         value = '10.10.10.10'
@@ -316,6 +313,8 @@ def i(d):
         self.assertFilter(fdata, i(one_month), True)
         self.assertFilter(fdata, i(now), True)
         self.assertFilter(fdata, i(now.isoformat()), True)
+        self.assertFilter(fdata, i(now.isoformat()), True)
+        self.assertFilter(fdata, i(calendar.timegm(now.timetuple())), True)
 
     def test_expiration(self):
Repository: cloud-custodian/cloud-custodian

Issue:
(UnsupportedOperation) when calling the ModifyInstanceAttribute operation: Modifying 'disableApiTermination' is not supported for spot instances
When a terminate action has "force:true" attribute attempts to take action on an ec2 spot instance, the following error is received.  CloudCustodian should be modified to only attempt to call ModifyInstanceAttribute if disableApiTermination=true for the instance it should terminate.

 An error occurred (UnsupportedOperation) when calling the ModifyInstanceAttribute operation: Modifying 'disableApiTermination' is not supported for spot instances.: ClientError
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 109, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 712, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 470, in run
results = action.process(resources)
File "/var/task/c7n/resources/ec2.py", line 1049, in process
self.disable_deletion_protection(instances)
File "/var/task/c7n/resources/ec2.py", line 1076, in disable_deletion_protection
list(w.map(process_instance, instances))
File "/var/runtime/concurrent/futures/_base.py", line 641, in result_iterator
yield fs.pop().result()
File "/var/runtime/concurrent/futures/_base.py", line 462, in result
return self.__get_result()
File "/var/runtime/concurrent/futures/thread.py", line 63, in run
result = self.fn(*self.args, **self.kwargs)
File "/var/task/c7n/utils.py", line 410, in _f
return f(*args, **kw)
File "/var/task/c7n/resources/ec2.py", line 1069, in process_instance
Value='false')
File "/var/task/c7n/utils.py", line 348, in _retry
return func(*args, **kw)
File "/var/runtime/botocore/client.py", line 317, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 615, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (UnsupportedOperation) when calling the ModifyInstanceAttribute operation: Modifying 'disableApiTermination' is not supported for spot instances.


Discussion:
the issue is partly that determining if disable api termination is enabled that requires another api call per resource. we could probably just detect that its a spot instance and not bother to modify it on force

Resolving patch:
diff --git a/c7n/resources/ec2.py b/c7n/resources/ec2.py
--- a/c7n/resources/ec2.py
+++ b/c7n/resources/ec2.py
@@ -1046,7 +1046,8 @@ def process(self, instances):
             return
         if self.data.get('force'):
             self.log.info("Disabling termination protection on instances")
-            self.disable_deletion_protection(instances)
+            self.disable_deletion_protection(
+                [i for i in instances if i.get('InstanceLifecycle') != 'spot'])
         client = utils.local_session(
             self.manager.session_factory).client('ec2')
         # limit batch sizes to avoid api limits


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
sns access policy
TLDR: Custodian is able to execute a policy as is, but will fail validation. This at least fixes the issue for me.

`custodian validate` on `modify-policy` for SNS policies fails because `manager.config` is passed  w/o `account_id` or `region` when running the `validate` command. Error does not bubble up as an `AttributeError` in the console, but validation is successful when `.get()` returns `None`, so suspecting it may be the case. PR should resolve the problem.


Discussion:
also for future ref its better to start a clean branch, there's alot of commit history on this branch thats superflous to the change.

Resolving patch:
diff --git a/c7n/actions.py b/c7n/actions.py
--- a/c7n/actions.py
+++ b/c7n/actions.py
@@ -867,8 +867,8 @@ class ModifyPolicyBase(BaseAction):
     def __init__(self, data=None, manager=None):
         if manager is not None:
             config_args = {
-                'account_id': manager.config.get('account_id'),
-                'region': manager.config.get('region')
+                'account_id': getattr(manager.config, 'account_id', None),
+                'region': getattr(manager.config, 'region', None)
             }
             self.data = utils.format_string_values(data, **config_args)
         else:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
c7n_mailer lambda error after included slack integration
Hi team,

It appears that deployed lambda mailer is having some problem with the libs, after the last commit that included slack integration. 

After update c7n_mailer lambda I'm seeing this error in the logs:
```
[ERROR] 2018-05-01T00:45:26.698Z f04d1326-4cd8-11e8-9f05-670c91cd10c1 Error starting mailer MailerSqsQueueProcessor().
Error: No module named requests

Traceback (most recent call last):
File "/var/task/c7n_mailer/handle.py", line 46, in start_c7n_mailer
mailer_sqs_queue_processor.run(parallel)
File "/var/task/c7n_mailer/sqs_queue_processor.py", line 126, in run
self.process_sqs_message(sqs_message)
File "/var/task/c7n_mailer/sqs_queue_processor.py", line 170, in process_sqs_message
from .slack_delivery import SlackDelivery
File "/var/task/c7n_mailer/slack_delivery.py", line 17, in <module>
import requests
ImportError: No module named requests
```
 I've had no success even with a fresh install from the repos.


Discussion:
https://github.com/capitalone/cloud-custodian/pull/2307 I've opened a new PR to include this dependency.
Can you confirm whether this change resolves the issue you're seeing?
the pr is unrelated to the solution for the problem, pip does install recursive dependencies, I'll have another pr up shortly.

Resolving patch:
diff --git a/tools/c7n_mailer/c7n_mailer/deploy.py b/tools/c7n_mailer/c7n_mailer/deploy.py
--- a/tools/c7n_mailer/c7n_mailer/deploy.py
+++ b/tools/c7n_mailer/c7n_mailer/deploy.py
@@ -41,7 +41,7 @@ def dispatch(event, context):
 def get_archive(config):
     archive = PythonPackageArchive(
         'c7n_mailer', 'ldap3', 'pyasn1', 'jinja2', 'markupsafe', 'ruamel',
-        'redis')
+        'redis', 'datadog', 'slackclient', 'requests')
 
     template_dir = os.path.abspath(
         os.path.join(os.path.dirname(__file__), '..', 'msg-templates'))
diff --git a/tools/c7n_mailer/setup.py b/tools/c7n_mailer/setup.py
--- a/tools/c7n_mailer/setup.py
+++ b/tools/c7n_mailer/setup.py
@@ -16,7 +16,9 @@
 from setuptools import setup, find_packages
 import os
 
-requires = ["Jinja2", "boto3", "jsonschema", "ruamel.yaml<0.15", "datadog"]
+# *Any* updates here should also go into c7n_mailer/deploy.py for lambda packaging.
+requires = [
+    "Jinja2", "boto3", "jsonschema", "ruamel.yaml<0.15", "datadog", "slackclient", "ldap3"]
 try:
     from concurrent import futures
 except ImportError:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
network-addr aka EIP tagging support
Support tag actions for network-addr resource now that AWS supports it per
https://aws.amazon.com/about-aws/whats-new/2017/12/amazon-virtual-private-cloud-vpc-now-allows-customers-to-tag-their-elastic-ip-addresses/


Discussion:
hmm, this might already work, or be a one liner, we do ec2 tags generically across the service.
Addition of the `copy-instance-tags` action to the EIP resource would be awesome.
Addition of a `release` action would prove useful as well. 👍 

Resolving patch:
diff --git a/c7n/resources/vpc.py b/c7n/resources/vpc.py
--- a/c7n/resources/vpc.py
+++ b/c7n/resources/vpc.py
@@ -1460,13 +1460,13 @@ class resource_type(object):
         service = 'ec2'
         type = 'network-addr'
         enum_spec = ('describe_addresses', 'Addresses', None)
-        name = id = 'PublicIp'
+        name = 'PublicIp'
+        id = 'AllocationId'
         filter_name = 'PublicIps'
         filter_type = 'list'
         date = None
         dimension = None
         config_type = "AWS::EC2::EIP"
-        taggable = False
 
 
 @resources.register('customer-gateway')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
How to detect unused application-elbs reliably?
I'm having a tough time trying to get Cloud Custodian to flag unused application ELBs, after some Googling I'm trying this approach:
```
  - name: find-unused-app-elb
    resource: app-elb
    filters:
      - type: metrics
        name: ConsumedLCUs
        days: 1
        period: 86400
        op: eq
        value: 0
```
I've confirmed that this metric exists in cloudwatch for my test app-elb, however when run the rule gets this error:

018-05-08 13:54:42,469: custodian.filters:WARNING CW Retrieval error: Parameter validation failed:
Invalid type for parameter Dimensions[0].Name, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

I'm not sure how to correct that or even if this is the best approach, I could use some advice.



Discussion:
There's a bug here, we're not passing the dimensions for app load balancers correctly. I'll have a fix in trunk shortly. Also note default statistic for metrics filter is Average, when your looking for unused things, Sum is also a good sanity check for things that have intermittent usage.
@kapilt Thank you for the fix!

Resolving patch:
diff --git a/c7n/resources/appelb.py b/c7n/resources/appelb.py
--- a/c7n/resources/appelb.py
+++ b/c7n/resources/appelb.py
@@ -59,7 +59,7 @@ class resource_type(object):
         id = 'LoadBalancerArn'
         filter_name = None
         filter_type = None
-        dimension = None
+        dimension = "LoadBalancer"
         date = 'CreatedTime'
         config_type = 'AWS::ElasticLoadBalancingV2::LoadBalancer'
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Azure Actions - Activity log string compare needs to be case insensitive
`def get_first_operation(logs, operation_name):` needs to be case insensitive with regard to `operation_name` (c7n_azure\activity.py)

In a bizarre twist, whether the operation name in the activity json in Azure is camel cased varies randomly...


Discussion:

Resolving patch:
diff --git a/tools/c7n_azure/c7n_azure/actions.py b/tools/c7n_azure/c7n_azure/actions.py
--- a/tools/c7n_azure/c7n_azure/actions.py
+++ b/tools/c7n_azure/c7n_azure/actions.py
@@ -248,7 +248,7 @@ def process_resource(self, resource):
     def get_first_operation(logs, operation_name):
         first_operation = None
         for l in logs:
-            if l.operation_name.value == operation_name:
+            if l.operation_name.value.lower() == operation_name.lower():
                 first_operation = l
 
         return first_operation


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
S3Output should grant bucket owner full control
Currently `S3Output` misses the `ACL` option. In a cross account setup it is desirable to give the bucket owner full control.

Would you change the code like this?:
```
diff --git a/c7n/output.py b/c7n/output.py
index c3839c2f..5fb06f59 100644
--- a/c7n/output.py
+++ b/c7n/output.py
@@ -268,6 +268,7 @@ class S3Output(FSOutput):
                 self.transfer.upload_file(
                     os.path.join(root, f), self.bucket, key,
                     extra_args={
+                        'ACL': 'bucket-owner-full-control',
                         'ServerSideEncryption': 'AES256'})
 
     def use_s3(self):
```



Discussion:

Resolving patch:
diff --git a/c7n/output.py b/c7n/output.py
--- a/c7n/output.py
+++ b/c7n/output.py
@@ -284,4 +284,5 @@ def upload(self):
                 self.transfer.upload_file(
                     os.path.join(root, f), self.bucket, key,
                     extra_args={
+                        'ACL': 'bucket-owner-full-control',
                         'ServerSideEncryption': 'AES256'})


Test patch:
diff --git a/tests/test_output.py b/tests/test_output.py
--- a/tests/test_output.py
+++ b/tests/test_output.py
@@ -125,7 +125,7 @@ def test_upload(self):
             fh.name,
             "cloud-custodian",
             "policies/xyz/%s/foo.txt" % output.date_path,
-            extra_args={"ServerSideEncryption": "AES256"},
+            extra_args={"ACL": "bucket-owner-full-control", "ServerSideEncryption": "AES256"},
         )
 
     def test_sans_prefix(self):
@@ -143,5 +143,5 @@ def test_sans_prefix(self):
             fh.name,
             "cloud-custodian",
             "policies/xyz/%s/foo.txt" % output.date_path,
-            extra_args={"ServerSideEncryption": "AES256"},
+            extra_args={"ACL": "bucket-owner-full-control", "ServerSideEncryption": "AES256"},
         )