internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
 'DescribeKey' object has no attribute 'log'
**Describe the bug**
Using a KMS type resource, where access to the KMS key is denied, a log call is made that seems to not be present.

**To Reproduce**
Run a policy against an AWS KMS resource where the IAM role used does not have access to the resource.

**Background (please complete the following information):**
 - OS: Lambda (Linux)
 - Python Version: 3.x
 - Custodian Version: 0.9.6
 - Cloud Provider: aws
 - Policy: 
```yaml
policies: 
  - name: kms-key-rotation
    resource: kms-key
    filters:
      - type: key-rotation-status
        key: KeyRotationEnabled
        value: false
```

```
Exception running policy:kms-key-rotation account:<redacted> region:eu-west-1 error:'DescribeKey' object has no attribute 'log'
```

**Additional context**
I think the issue is perhaps calling `self.log` where `DescribeKey` or parent `DescribeSource` does not contain that method https://github.com/cloud-custodian/cloud-custodian/blob/master/c7n/resources/kms.py#L73-L75


Discussion:

Resolving patch:
diff --git a/c7n/resources/kms.py b/c7n/resources/kms.py
--- a/c7n/resources/kms.py
+++ b/c7n/resources/kms.py
@@ -70,7 +70,7 @@ def augment(self, resources):
                 r.update(info)
             except ClientError as e:
                 if e.response['Error']['Code'] == 'AccessDeniedException':
-                    self.log.warning(
+                    self.manager.log.warning(
                         "Access denied when describing key:%s",
                         key_id)
                 else:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws - service catalog cross-account filter
**Describe the bug**
The schema for cross-account(servicecatalog.py) has `whitelist_accounts_from` (https://github.com/cloud-custodian/cloud-custodian/blob/master/c7n/resources/servicecatalog.py#L84).
CrossAccountAccessFilter.get_accounts looks for `whitelist_from` keyword specifically (https://github.com/cloud-custodian/cloud-custodian/blob/master/c7n/filters/iamaccess.py#L298)

**To Reproduce**
Write a policy with `cross-account` filter using `whitelist_accounts_from`

**Solution**
Fix the schema to `whitelist_from`

**Additional context**
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/c7n/resources/servicecatalog.py b/c7n/resources/servicecatalog.py
--- a/c7n/resources/servicecatalog.py
+++ b/c7n/resources/servicecatalog.py
@@ -81,7 +81,7 @@ class CatalogPortfolioCrossAccount(CrossAccountAccessFilter):
 
     schema = type_schema(
         'cross-account',
-        whitelist_accounts_from={'$ref': '#/definitions/filters_common/value_from'},
+        whitelist_from={'$ref': '#/definitions/filters_common/value_from'},
         whitelist={'type': 'array', 'items': {'type': 'string'}})
 
     permissions = ('servicecatalog:ListPortfolioAccess',)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws.app-elb.actions.modify-listener add support for "protocol: TLS"
**Is your feature request related to a problem? Please describe.**
Would like to use custodian to modify network load balancer listener sslpolicy. Network load balancer's support sslpolicy when using a listener with protocol: "TLS"

**Describe the solution you'd like**
Currently supports protocols HTTP and HTTPS for app load balancers, needs TLS support aswell for NLB's

**Describe alternatives you've considered**
None

**Additional context**

```
Error on policy:elb-nlb-fix-tls-policy resource:aws.app-elb
--
TLS' is not one of ['HTTP', 'HTTPS']

Failed validating 'enum' in schema[7]['properties']['protocol']:
{'enum': ['HTTP', 'HTTPS']}

On instance['protocol']:
'TLS'
```





Discussion:
@wiredin - can you share the policy you are using to reproduce this issue?

Just some initial notes:

Using `modify_listener` to move from TCP to TLS will require a certificate and policy to also be defined in the policy.

I'm fuzzy on how this works currently, but if `c7n` is able to move from HTTP -> HTTPS, we may be accounting for that. 

To get this done we probably need patterns to...

1. Define the certificate + policy in the YAML
2. Offer some amount of fault tolerance should the defined certificate be invalid (`CertificateNotFound` error)
3. Avoid overwrites of existing configurations through `modify_listener`
   - From the docs:
   > To add an item to a list, remove an item from a list, or update an item in a list, you must provide the entire list. For example, to add an action, specify a list with the current actions plus the new action

I can poke around on this and try to confirm we are supporting these behaviors.

I'm a little fuzzy on the API - `modify_listener` takes an array of certificates, but the docs say only one can be submitted. However, more than one can be submitted through `add_listener_certificates` - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/elbv2.html#ElasticLoadBalancingv2.Client.add_listener_certificates - but the the API has to be invoked per certificate(?).

It's possible `modify_listener` is doing something "under the hood" to call `add_listener_certificates` for you as a convenience.
i think the underlying issue is that the action was written when elbv2 only did app elbs, for network elbs there's quite a few additional protocols supported.

Resolving patch:
diff --git a/c7n/resources/appelb.py b/c7n/resources/appelb.py
--- a/c7n/resources/appelb.py
+++ b/c7n/resources/appelb.py
@@ -831,7 +831,7 @@ class AppELBModifyListenerPolicy(BaseAction):
     schema = type_schema(
         'modify-listener',
         port={'type': 'integer'},
-        protocol={'enum': ['HTTP', 'HTTPS']},
+        protocol={'enum': ['HTTP', 'HTTPS', 'TCP', 'TLS', 'UDP', 'TCP_UDP', 'GENEVE']},
         sslpolicy={'type': 'string'},
         certificate={'type': 'string'}
     )


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Issues running dynamodb as a config rule
**Describe the bug**
Running a simple tag filter on dynamodb does not work when running as a config-rule, while  the same policies run locally or in lambda periodic do work

**To Reproduce**
Steps to reproduce the behavior:
Have some dynamodb table in an account, apply policy.  Failure manifests in the created lambdas logs


**Expected behavior**
Policy to run and either find a match or none


**Background (please complete the following information):**

 - Custodian Version: 0.9.10

 - Cloud Provider: aws
 - Policy: [please exclude any account/sensitive information]
```yaml
policies:
- name: dynamodb-tag-config-compliance
  resource: aws.dynamodb-table
  mode:
    type: config-rule
    role: custodian
  filters:
  - or:
    - "tag:Name": absent
    - "tag:Contact": absent
    - "tag:Environment": absent

```
 - Traceback: [if applicable, please exclude sensitive/account information]


```
[ERROR] TypeError: unsupported operand type(s) for /: 'datetime.datetime' and 'float'
Traceback (most recent call last):
  File "/var/task/custodian_policy.py", line 4, in run
    return handler.dispatch_event(event, context)
  File "/var/task/c7n/handler.py", line 165, in dispatch_event
    p.push(event, context)
  File "/var/task/c7n/policy.py", line 1140, in push
    return mode.run(event, lambda_ctx)
  File "/var/task/c7n/policy.py", line 853, in run
    resources = super(ConfigRuleMode, self).run(event, lambda_context)
  File "/var/task/c7n/policy.py", line 437, in run
    resources = self.resolve_resources(event)
  File "/var/task/c7n/policy.py", line 835, in resolve_resources
    return [source.load_resource(self.cfg_event['configurationItem'])]
  File "/var/task/c7n/resources/dynamodb.py", line 23, in load_resource
    resource['CreationDateTime'] = datetime.fromtimestamp(resource['CreationDateTime'] / 1000.0)
```




Discussion:

Resolving patch:
diff --git a/c7n/resources/dynamodb.py b/c7n/resources/dynamodb.py
--- a/c7n/resources/dynamodb.py
+++ b/c7n/resources/dynamodb.py
@@ -2,7 +2,6 @@
 # SPDX-License-Identifier: Apache-2.0
 from botocore.exceptions import ClientError
 from concurrent.futures import as_completed
-from datetime import datetime
 
 from c7n.actions import BaseAction, ModifyVpcSecurityGroupsAction
 from c7n.filters.kms import KmsRelatedFilter
@@ -20,13 +19,6 @@ class ConfigTable(query.ConfigSource):
 
     def load_resource(self, item):
         resource = super(ConfigTable, self).load_resource(item)
-        resource['CreationDateTime'] = datetime.fromtimestamp(resource['CreationDateTime'] / 1000.0)
-        if ('BillingModeSummary' in resource and
-                'LastUpdateToPayPerRequestDateTime' in resource['BillingModeSummary']):
-            resource['BillingModeSummary'][
-                'LastUpdateToPayPerRequestDateTime'] = datetime.fromtimestamp(
-                    resource['BillingModeSummary']['LastUpdateToPayPerRequestDateTime'] / 1000.0)
-
         sse_info = resource.pop('Ssedescription', None)
         if sse_info is None:
             return resource


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Event Pattern is not valid, PHD all events
**Describe the bug**

```
botocore.errorfactory.InvalidEventPatternException: An error occurred (InvalidEventPatternException) when calling the PutRule operation: Event pattern is not valid. Reason: Empty objects are not allowed at [Source: (String)"{"source": ["aws.health"], "detail": {}}"; line: 1, column: 40]
```

**To Reproduce**
Steps to reproduce the behavior:

Create a PHD alert for all events

**Expected behavior**
No Error


**Background (please complete the following information):**
  - Custodian Version: [e.g. 0.9.10.]
 
 - Policy: [please exclude any account/sensitive information]
```yaml
policies:
  - name: phd-alerts
    resource: account
    comment: PHD alerts
    mode:
      type: phd
      role: xxxxx
    description: Any PHD alert
```
**Additional context**
Add any other context about the problem here.

https://github.com/cloud-custodian/cloud-custodian/blob/master/c7n/mu.py#L1074

The empty object at the above line causes the boto3 error



Discussion:
Found additional information

https://github.com/cloud-custodian/cloud-custodian/pull/5133

https://github.com/cloud-custodian/cloud-custodian/commit/9b0bf2c1593e72fa287bdcfb77d36a40ea02d25b


explicitly enumerating the phd event types of interest would be an easy work around. there's only a handful, ie.

`issue | accountNotification | scheduledChange | investigation`

Is there a way to undo the breaking change from https://github.com/cloud-custodian/cloud-custodian/pull/6141 ?
This was working great until that release per https://github.com/cloud-custodian/cloud-custodian/pull/5133

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -1078,6 +1078,8 @@ def render_event_pattern(self):
                 })
             if self.data.get('categories', []):
                 payload['detail']['eventTypeCategory'] = self.data['categories']
+            if not payload['detail']:
+                payload.pop('detail')
         elif event_type == 'hub-finding':
             payload['source'] = ['aws.securityhub']
             payload['detail-type'] = ['Security Hub Findings - Imported']


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -208,6 +208,19 @@ def test_phd_account_mode(self):
         self.assertEqual(len(resources), 1)
         self.assertTrue('c7n:HealthEvent' in resources[0])
 
+    def test_phd_mode_sans_details(self):
+        factory = self.replay_flight_data('test_phd_event_mode')
+        p = self.load_policy(
+            {'name': 'ec2-retire',
+             'resource': 'account',
+             'mode': {'type': 'phd'}}, session_factory=factory)
+        p_lambda = PolicyLambda(p)
+        events = p_lambda.get_events(factory)
+        self.assertEqual(
+            json.loads(events[0].render_event_pattern()),
+            {'source': ['aws.health']}
+        )
+
     def test_phd_mode(self):
         factory = self.replay_flight_data('test_phd_event_mode')
         p = self.load_policy(
Repository: cloud-custodian/cloud-custodian

Issue:
ebs modify support gp3
**Is your feature request related to a problem? Please describe.**
AWS announced the availability of gp3, the next-generation general purpose SSD volumes, I would like to convert all ebs to gp3 using policies, but seem it dose not support yet. 

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

May I request modify to gp3 as below?  Thank you.
    actions:
      - type: modify
        volume-type: gp3

**Additional context**
Failed validating 'enum' in schema[9]['properties']['volume-type']:
    {'enum': ['io1', 'gp2', 'st1', 'sc1']}


Discussion:

Resolving patch:
diff --git a/c7n/resources/ebs.py b/c7n/resources/ebs.py
--- a/c7n/resources/ebs.py
+++ b/c7n/resources/ebs.py
@@ -1598,7 +1598,7 @@ class ModifyVolume(BaseAction):
 
     schema = type_schema(
         'modify',
-        **{'volume-type': {'enum': ['io1', 'gp2', 'st1', 'sc1']},
+        **{'volume-type': {'enum': ['io1', 'gp2', 'gp3', 'st1', 'sc1']},
            'shrink': False,
            'size-percent': {'type': 'number'},
            'iops-percent': {'type': 'number'}})


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
ebs-snapshot action Not able to copy snapshot in same region
Trying to copy the snapshot in the same region with encryption but custodian not able to create a snapshot in the same region.

But when I tried with the different regions it's working totally fine.

```   
 actions:
      - type: copy
        target_region: ap-south-1
        target_key: alias/aws/ebs
        encrypted: true
```

As the same region is also a valid action but still getting issue and getting this error 

`2021-05-17 15:33:33,677: custodian.actions: INFO Source and destination region are the same, skipping`

Also, I dug into the ebs.py and got to know the error `Source and destination region are the same, skipping` is coming from ebs.py and copy the snapshot in the same region is a valid option.



Discussion:

Resolving patch:
diff --git a/c7n/resources/ebs.py b/c7n/resources/ebs.py
--- a/c7n/resources/ebs.py
+++ b/c7n/resources/ebs.py
@@ -502,11 +502,6 @@ def validate(self):
         return self
 
     def process(self, resources):
-        if self.data['target_region'] == self.manager.config.region:
-            self.log.info(
-                "Source and destination region are the same, skipping")
-            return
-
         with self.executor_factory(max_workers=2) as w:
             list(w.map(self.process_resource_set, chunks(resources, 20)))
 
@@ -514,8 +509,7 @@ def process_resource_set(self, resource_set):
         client = self.manager.session_factory(
             region=self.data['target_region']).client('ec2')
 
-        if self.data['target_region'] != self.manager.config.region:
-            cross_region = True
+        cross_region = self.data['target_region'] != self.manager.config.region
 
         params = {}
         params['Encrypted'] = self.data.get('encrypted', True)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Mugc ignored Global Services policy with region and did not remove policy from other regions
**Describe the bug**
A clear and concise description of what the bug is.
According to documentation `https://cloudcustodian.io/docs/aws/examples/accountservicelimit.html` added region to one of our policies. Did redeploy and pipeline c7n-org step did update policy in specified us-east-1 region, but mugc step did not remove from us-east-2

**To Reproduce**
Steps to reproduce the behavior:
deploy policy for resource s3 in 2 regions: us-east-1 and us-east-2
add region: us-east-1 to the policy
deploy using c7n-org
run mugc

**Expected behavior**
A clear and concise description of what you expected to happen.
I would expect mugc to remove policy from all other regions

**Background (please complete the following information):**
 - Python Version: [e.g. python 3.8.1] virtual environment CPython3.8.3.final.0-64
 - Custodian Version: [e.g. 0.8.46.1] c7n 0.9.6, c7n-mailer 0.6.5, c7n-org 0.6.5
 - Tool Version: [if applicable] codebuild pipeline
 - Cloud Provider: [e.g. gcp, aws, azure] aws
 - Policy: [please exclude any account/sensitive information]
```yaml
policies: 
   - name: list-buckets
      resource: s3
      region: us-east-1
```

**Additional context**
Add any other context about the problem here.



Discussion:
the cli params here a little unclear, are you doing --region all, if you targeted to a particular region ie. --region us-east-2 does it do the right thing? 
sorry, I am using c7n-org with multiple accounts each with own set of regions. then i combine all policies in single file to make mugc command possible. in that context region is added to policy and c7n-org correctly deploys it in marked region.

adding region to a policy pins it to one region.

my previous question was i'm looking for the parameters that your using to invoke mugc.py (sans role)

your steps to reproduce note two different deploys which seems odd, mugc doesn't look at the policy region its using the regions that are passed to it on the cli. are you executing mugc with c7n-org?



Resolving patch:
diff --git a/tools/ops/mugc.py b/tools/ops/mugc.py
--- a/tools/ops/mugc.py
+++ b/tools/ops/mugc.py
@@ -43,15 +43,15 @@ def region_gc(options, region, policy_config, policies):
     client = session_factory().client('lambda')
 
     remove = []
-    current_policies = [p.name for p in policies]
     pattern = re.compile(options.policy_regex)
     for f in funcs:
         if not pattern.match(f['FunctionName']):
             continue
         match = False
-        for pn in current_policies:
-            if f['FunctionName'].endswith(pn):
-                match = True
+        for p in policies:
+            if f['FunctionName'].endswith(p.name):
+                if 'region' not in p.data or p.data['region'] == region:
+                    match = True
         if options.present:
             if match:
                 remove.append(f)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Add onhours/offhours support to sagemaker-notebook
Would be awesome to be able to define policies for onhours/offhours to sagemaker-notebook instances just like we can for EC2 instances.


Discussion:

Resolving patch:
diff --git a/c7n/resources/sagemaker.py b/c7n/resources/sagemaker.py
--- a/c7n/resources/sagemaker.py
+++ b/c7n/resources/sagemaker.py
@@ -9,6 +9,7 @@
 from c7n.tags import RemoveTag, Tag, TagActionFilter, TagDelayedAction
 from c7n.filters.vpc import SubnetFilter, SecurityGroupFilter
 from c7n.filters.kms import KmsRelatedFilter
+from c7n.filters.offhours import OffHour, OnHour
 
 
 @resources.register('sagemaker-notebook')
@@ -43,6 +44,8 @@ def _augment(r):
 
 
 NotebookInstance.filter_registry.register('marked-for-op', TagActionFilter)
+NotebookInstance.filter_registry.register('offhour', OffHour)
+NotebookInstance.filter_registry.register('onhour', OnHour)
 
 
 @resources.register('sagemaker-job')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Clarify/expand documentation for `match-operator` in `aws.security-group.filters.ingress`
### Describe the feature

The `ingress` filter for `aws.security-group` supports a `match-operator` property. It would be helpful to have an explanation of its intent and some usage examples.

### Extra information or context

A few Cloud Custodian filters support a `match-operator` property, and it behaves subtly different from one filter to another (in `aws.iam-user.access-key` it controls how to evaluate multiple subresources, where in the `guard-duty` filter it controls how to evaluate multiple attributes of the filter). The main question is "when/why would I use `match-operator: or` over the default `and`?"

This was motivated by a few discussions in Gitter, notably [this one](https://gitter.im/cloud-custodian/cloud-custodian?at=61f14921742c3d4b21b7ed51).


Discussion:

Resolving patch:
diff --git a/c7n/resources/vpc.py b/c7n/resources/vpc.py
--- a/c7n/resources/vpc.py
+++ b/c7n/resources/vpc.py
@@ -1080,6 +1080,32 @@ class SGPermission(Filter):
           value: 'default - DO NOT USE'
           op: equal
 
+    By default, this filter matches a security group rule if
+    _all_ of its keys match. Using `match-operator: or` causes a match
+    if _any_ key matches. This can help consolidate some simple
+    cases that would otherwise require multiple filters. To find
+    security groups that allow all inbound traffic over IPv4 or IPv6,
+    for example, we can use two filters inside an `or` block:
+
+    .. code-block:: yaml
+
+      - or:
+        - type: ingress
+          Cidr: "0.0.0.0/0"
+        - type: ingress
+          CidrV6: "::/0"
+
+    or combine them into a single filter:
+
+    .. code-block:: yaml
+
+      - type: ingress
+        match-operator: or
+          Cidr: "0.0.0.0/0"
+          CidrV6: "::/0"
+
+    Note that evaluating _combinations_ of factors (e.g. traffic over
+    port 22 from 0.0.0.0/0) still requires separate filters.
     """
 
     perm_attrs = {


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
NoneType Issue
I dropped a yaml file into a new OU/SubOU and its not working, though it works in other OUs just fine.   Nothing was changed in the file but I am still getting this error, not sure why.

```
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.12/bin/custodian", line 8, in <module>
    sys.exit(main())
  File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/c7n/cli.py", line 363, in main
    command(config)
  File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/c7n/commands.py", line 219, in validate
    structure.validate(data)
  File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/c7n/structure.py", line 48, in validate
    self.validate_policy(p)
  File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/c7n/structure.py", line 78, in validate_policy
    for a in p.get('actions', ()):
TypeError: 'NoneType' object is not iterable
```


Discussion:

Resolving patch:
diff --git a/c7n/structure.py b/c7n/structure.py
--- a/c7n/structure.py
+++ b/c7n/structure.py
@@ -66,7 +66,7 @@ def validate_policy(self, p):
                 'policy:%s must use a list for filters found:%s' % (
                     p['name'], type(p['filters']).__name__)))
         element_types = (dict, str)
-        for f in p.get('filters', ()):
+        for f in p.get('filters', ()) or []:
             if not isinstance(f, element_types):
                 raise PolicyValidationError((
                     'policy:%s filter must be a mapping/dict found:%s' % (
@@ -75,7 +75,7 @@ def validate_policy(self, p):
             raise PolicyValidationError((
                 'policy:%s must use a list for actions found:%s' % (
                     p.get('name', 'unknown'), type(p['actions']).__name__)))
-        for a in p.get('actions', ()):
+        for a in p.get('actions', ()) or []:
             if not isinstance(a, element_types):
                 raise PolicyValidationError((
                     'policy:%s action must be a mapping/dict found:%s' % (


Test patch:
diff --git a/tests/test_schema.py b/tests/test_schema.py
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -73,6 +73,14 @@ def test_invalid_action(self):
         self.assertTrue(str(ecm.exception).startswith(
             'policy:foo action must be a mapping/dict found:list'))
 
+    def test_null_actions(self):
+        p = StructureParser()
+        p.validate({'policies': [{'name': 'foo', 'resource': 'ec2', 'actions': None}]})
+
+    def test_null_filters(self):
+        p = StructureParser()
+        p.validate({'policies': [{'name': 'foo', 'resource': 'ec2', 'filters': None}]})
+
     def test_invalid_filter(self):
         p = StructureParser()
         with self.assertRaises(PolicyValidationError) as ecm: