internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
PyYaml conflict
So I guess the brand new PyYaml==4.2b4 is required for Python 3.7 compatibility so people are starting to require it.  Azure-CLI now requires it... 

We have a variety of conflicts.  I can pin an old version of Azure CLI now, but realistically we probably need to eliminate these old ones so we can work in Python 3.7

![image](https://user-images.githubusercontent.com/4087345/42241712-5b00dd2a-7ec0-11e8-9fef-c4795bbd438f.png)




Discussion:
Removing P0 tag as I have worked around this for now, but we should decide if we want to bump all these other dependencies.  I updated them but I'm not sure I know how to test all those other modules.
fwiw, I've been using `pipdeptree` to debug the version conflicts. its got a reverse mode dep mode as well  ie `pipdeptree -r -p pyyaml`
Yeah in this case most of these deps are on tools I've never installed/used before - not directly related to the conflict but would all need to be fixed.

So my workaround here (pinning an old azure version before they added 3.7 support) just makes it harder for us to get to Python 3.7 - assuming that is a goal we should probably go move pyyaml to latest on all those modules.   I can certainly do it without breaking the build but won't be able to test some of those tools probably.  

I'll have to do it soon anyway as we can't stuck to older Azure versions or we'll miss new API's.
so it looks like we can just fix those two reverse deps that have hard pins on 3.x to do `>=` and things should be good, and we can continue tracking azure current pkg release set.

Resolving patch:
diff --git a/c7n/sqsexec.py b/c7n/sqsexec.py
--- a/c7n/sqsexec.py
+++ b/c7n/sqsexec.py
@@ -54,6 +54,8 @@ def __init__(self, session_factory, map_queue, reduce_queue):
         self.futures = {}
 
     def submit(self, func, *args, **kwargs):
+        """Submit a function for serialized execution on sqs
+        """
         self.op_sequence += 1
         self.sqs.send_message(
             QueueUrl=self.map_queue,


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Offhours schedule doesn't accept time zone identifiers
If I add a timezone identifier that is not a part of the TZ_ALIASES list but is a valid time zone identifier I get "Could not resolve tz on resource" error.

As a test I also changed the test_onhour test tz to "America/Los_Angeles" in the offhours test file and running  the test provides the same error:

"offhours.py                417 WARNING  Could not resolve tz on resource:i-1aebf7c0 value:tz=america/los_angeles
"
In the filters/offhours.py file, we have function get_tag_values that is storing the value of the tag as lowercase. This also converts the time zone identifiers which causes the tz to become "america/los_angeles" which is not recogbized by dateutil `zoneinfo.gettz:`

def get_tag_value(self, i):
        """Get the resource's tag value specifying its schedule."""
        # Look for the tag, Normalize tag key and tag value
        found = False
        for t in i.get('Tags', ()):
            if t['Key'].lower() == self.tag_key:
                found = t['Value']
                break
        if found is False:
            return False
        # enforce utf8, or do translate tables via unicode ord mapping
        **value = found.lower().encode('utf8').decode('utf8')**
        # Some folks seem to be interpreting the docs quote marks as
        # literal for values.
        value = value.strip("'").strip('"')
        return value


Discussion:
Thanks for the bug report.

Resolving patch:
diff --git a/c7n/filters/offhours.py b/c7n/filters/offhours.py
--- a/c7n/filters/offhours.py
+++ b/c7n/filters/offhours.py
@@ -456,7 +456,10 @@ def get_tag_value(self, i):
 
     @classmethod
     def get_tz(cls, tz):
-        return zoneinfo.gettz(cls.TZ_ALIASES.get(tz, tz))
+        found = cls.TZ_ALIASES.get(tz)
+        if found:
+            tz = found
+        return zoneinfo.gettz(tz.title())
 
     def get_default_schedule(self):
         raise NotImplementedError("use subclass")


Test patch:
diff --git a/tests/test_offhours.py b/tests/test_offhours.py
--- a/tests/test_offhours.py
+++ b/tests/test_offhours.py
@@ -527,6 +527,16 @@ def test_tz_only(self):
             i = instance(Tags=[{"Key": "maid_offhours", "Value": "tz=est"}])
             self.assertEqual(OnHour({})(i), True)
 
+    def test_tz_long_form_resolve(self):
+        pacific = zoneinfo.gettz("America/Los_Angeles")
+        nzt = zoneinfo.gettz("Pacific/Auckland")
+        self.assertEqual(
+            OnHour({}).get_tz('america/los_angeles'),
+            pacific)
+        self.assertEqual(
+            OnHour({}).get_tz('pacific/auckland'),
+            nzt)
+
     def test_empty_tag(self):
         t = datetime.datetime.now(zoneinfo.gettz("America/New_York"))
         t = t.replace(year=2016, month=5, day=26, hour=7, minute=00)
Repository: cloud-custodian/cloud-custodian

Issue:
metrics filter not working for EMR
I get following warning

(py36) bash-3.2$ custodian run -c unused-emr.yml --verbose -s out
2018-08-03 10:28:21,406: custodian.commands:DEBUG Loaded file unused-emr.yml. Contains 1 policies
2018-08-03 10:28:21,418: custodian.aws:DEBUG using default region:eu-west-1 from boto
2018-08-03 10:28:21,874: custodian.output:DEBUG Storing output with <DirectoryOutput to dir:out/unused-emr-clusters>
2018-08-03 10:28:21,875: custodian.policy:DEBUG Running policy unused-emr-clusters resource: emr region:eu-west-1 c7n:0.8.28.2
2018-08-03 10:28:23,365: custodian.filters:DEBUG Querying metrics for 1
2018-08-03 10:28:23,420: custodian.filters:WARNING CW Retrieval error: 'ClusterId'
2018-08-03 10:28:23,420: custodian.resources.emrcluster:DEBUG Filtered from 1 to 0 emrcluster
2018-08-03 10:28:23,420: custodian.policy:INFO policy: unused-emr-clusters resource:emr region:eu-west-1 count:0 time:1.54
2018-08-03 10:28:23,421: custodian.output:DEBUG metric:ResourceCount Count:0 policy:unused-emr-clusters restype:emr scope:policy

when attempting to run 

```policies:
  - name: unused-emr-clusters
    description: |
      Retrieve all unused emr clusters
    resource: emr
    filters:
      - type: metrics
        name: IsIdle
        value: 1
        op: equal
```

Note that i do have a cluster thats currently idle.
Is there any other way we can query for unused EMR clusters?

Note that filtering for metrics is working just fine for other resources, EMR is the only one we've had issues with.


Discussion:

Resolving patch:
diff --git a/c7n/resources/emr.py b/c7n/resources/emr.py
--- a/c7n/resources/emr.py
+++ b/c7n/resources/emr.py
@@ -20,7 +20,7 @@
 
 from c7n.actions import ActionRegistry, BaseAction
 from c7n.exceptions import PolicyValidationError
-from c7n.filters import FilterRegistry
+from c7n.filters import FilterRegistry, MetricsFilter
 from c7n.manager import resources
 from c7n.query import QueryResourceManager
 from c7n.utils import (
@@ -47,9 +47,9 @@ class resource_type(object):
         enum_spec = ('list_clusters', 'Clusters', {'ClusterStates': cluster_states})
         name = 'Name'
         id = 'Id'
-        dimension = 'ClusterId'
         date = "Status.Timeline.CreationDateTime"
         filter_name = None
+        dimension = None
 
     action_registry = actions
     filter_registry = filters
@@ -120,6 +120,14 @@ def augment(self, resources):
         return result
 
 
+@EMRCluster.filter_registry.register('metrics')
+class EMRMetrics(MetricsFilter):
+
+    def get_dimensions(self, resource):
+        # Job flow id is legacy name for cluster id
+        return [{'Name': 'JobFlowId', 'Value': resource['Id']}]
+
+
 @actions.register('mark-for-op')
 class TagDelayedAction(TagDelayedAction):
     """Action to specify an action to occur at a later date


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
c7n_mailer throwing JSON errors on sagemaker-notebook
The `resource_format` with the j2 template is not rendering correct JSON for some resources and therefore not able to send messages.

Example:
```
policies:
- actions:
  - slack_template: slack_default
    to:
    - slack://webhook/#my-channel
    transport:
      type: sqs
      queue: https://sqs.us-east-1.amazonaws.com/12345678/my_queue
    type: notify
  filters:
  - or:
    - DirectInternetAccess: Enabled
    - KmsKeyId: absent
  resource: sagemaker-notebook
```
A section of the rendered JSON that's broken . . .
```
{
   "attachments":[
      {
         "fallback":"Cloud Custodian Policy Violation",
         "title":"Custodian",
         "color":"danger",
         "fields":[
            {
               "title":"Resources",
ERROR ---->               "value":"{
  "NetworkInterfaceId": "eni-0df98f3f169c8153f", 
  "NotebookInstanceStatus": "InService", 
  "Tags": [], 
  "Url": "public-test.notebook.us-east-1.sagemaker.aws", 
  "RoleArn": "arn:aws:iam::1234567:role/service-role/AmazonSageMaker-ExecutionRole-20180123T145367", 
  "NotebookInstanceName": "public-test", 
  "CreationTime": "2018-08-10T18:22:32.969000-04:00", 
  "c7n:MatchedFilters": [
    "DirectInternetAccess", 
    "KmsKeyId"
  ],
Traceback (most recent call last):
  File "/var/task/c7n_mailer/sqs_queue_processor.py", line 174, in process_sqs_message
    slack_delivery.slack_handler(sqs_message, slack_messages)
  File "/var/task/c7n_mailer/slack_delivery.py", line 104, in slack_handler
    json.loads(payload, strict=False)["channel"])
  File "/usr/lib64/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting ',' delimiter: line 11 column 4 (char 241)
```


Discussion:

Resolving patch:
diff --git a/tools/c7n_mailer/c7n_mailer/utils.py b/tools/c7n_mailer/c7n_mailer/utils.py
--- a/tools/c7n_mailer/c7n_mailer/utils.py
+++ b/tools/c7n_mailer/c7n_mailer/utils.py
@@ -150,6 +150,8 @@ def resource_format(resource, resource_type):
     elif resource_type == 'ami':
         return "%s %s %s" % (
             resource.get('Name'), resource['ImageId'], resource['CreationDate'])
+    elif resource_type == 'sagemaker-notebook':
+        return "%s" % (resource['NotebookInstanceName'])
     elif resource_type == 's3':
         return "%s" % (resource['Name'])
     elif resource_type == 'ebs':


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Support multiple Slack webhooks
Right now the Slack mailer assumes that a single webhook (channel) is configured in the `mailer.yaml`, which is not really practical. We need to accommodate for one or more webhooks defined in a policy.

Working on it . . .


Discussion:

Resolving patch:
diff --git a/tools/c7n_mailer/c7n_mailer/slack_delivery.py b/tools/c7n_mailer/c7n_mailer/slack_delivery.py
--- a/tools/c7n_mailer/c7n_mailer/slack_delivery.py
+++ b/tools/c7n_mailer/c7n_mailer/slack_delivery.py
@@ -11,7 +11,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import json
 import time
 
 from botocore.vendored import requests
@@ -66,6 +65,11 @@ def get_to_addrs_slack_messages_map(self, sqs_message):
                             self.logger, 'slack_template', 'slack_default')
                 self.logger.debug(
                     "Generating messages for recipient list produced by resource owner resolution.")
+            elif target.startswith('https://hooks.slack.com/'):
+                slack_messages[target] = get_rendered_jinja(
+                    target, sqs_message,
+                    resource_list,
+                    self.logger, 'slack_template', 'slack_default')
             elif target.startswith('slack://webhook/#') and self.config.get('slack_webhook'):
                 webhook_target = self.config.get('slack_webhook')
                 slack_messages[webhook_target] = get_rendered_jinja(
@@ -100,7 +104,7 @@ def slack_handler(self, sqs_message, slack_messages):
                 sqs_message['policy']['resource'],
                 str(len(sqs_message['resources'])),
                 sqs_message['action'].get('slack_template', 'slack_default'),
-                json.loads(payload, strict=False)["channel"])
+                key)
             )
 
             self.send_slack_msg(key, payload)
diff --git a/tools/c7n_mailer/c7n_mailer/sqs_queue_processor.py b/tools/c7n_mailer/c7n_mailer/sqs_queue_processor.py
--- a/tools/c7n_mailer/c7n_mailer/sqs_queue_processor.py
+++ b/tools/c7n_mailer/c7n_mailer/sqs_queue_processor.py
@@ -166,7 +166,8 @@ def process_sqs_message(self, encoded_sqs_message):
         sns_delivery.deliver_sns_messages(sns_message_packages, sqs_message)
 
         # this section sends a notification to the resource owner via Slack
-        if any(e.startswith('slack') for e in sqs_message.get('action', ()).get('to')):
+        if any(e.startswith('slack') or e.startswith('https')
+                for e in sqs_message.get('action', ()).get('to')):
             from .slack_delivery import SlackDelivery
             slack_delivery = SlackDelivery(self.config, self.session, self.logger)
             slack_messages = slack_delivery.get_to_addrs_slack_messages_map(sqs_message)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
batch-compute resource security-group filter support
Add a security-group filter to the batch-compute resource  using the securityGroupIds attribute
  


Discussion:

Resolving patch:
diff --git a/c7n/resources/batch.py b/c7n/resources/batch.py
--- a/c7n/resources/batch.py
+++ b/c7n/resources/batch.py
@@ -13,9 +13,11 @@
 # limitations under the License.
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+from c7n.actions import BaseAction
+from c7n.filters.vpc import SecurityGroupFilter, SubnetFilter
 from c7n.manager import resources
 from c7n.query import QueryResourceManager
-from c7n.actions import BaseAction
+
 from c7n.utils import local_session, type_schema
 
 
@@ -32,6 +34,18 @@ class resource_type(object):
             'describe_compute_environments', 'computeEnvironments', None)
 
 
+@ComputeEnvironment.filter_registry.register('security-group')
+class ComputeSGFilter(SecurityGroupFilter):
+
+    RelatedIdsExpression = "computeResources.securityGroupIds"
+
+
+@ComputeEnvironment.filter_registry.register('subnet')
+class ComputeSubnetFilter(SubnetFilter):
+
+    RelatedIdsExpression = "computeResources.subnets"
+
+
 @resources.register('batch-definition')
 class JobDefinition(QueryResourceManager):
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
azure - function deploy with B1 sku times out waiting for app service plan
When deploying a function policy using a B1 sku, c7n times out waiting for the app service plan to finish deploying. When this happens, we never publish the policy. This is caused by this line: https://github.com/capitalone/cloud-custodian/blob/master/tools/c7n_azure/c7n_azure/function_package.py#L187

5 retries at 15 second intervals isn't enough time for B1. B2 and above work as expected. I propose we increase the number of retries. In theory, this should not slow down performance for other skus as they should not get to the max retry limit.


Discussion:

Resolving patch:
diff --git a/tools/c7n_azure/c7n_azure/function_package.py b/tools/c7n_azure/c7n_azure/function_package.py
--- a/tools/c7n_azure/c7n_azure/function_package.py
+++ b/tools/c7n_azure/c7n_azure/function_package.py
@@ -184,12 +184,13 @@ def build(self, policy, entry_point=None, extra_modules=None):
         # cffi module needs special handling
         self._add_cffi_module()
 
-    def wait_for_status(self, app_name, retries=5, delay=15):
+    def wait_for_status(self, app_name, retries=10, delay=15):
         for r in range(retries):
             if self.status(app_name):
                 return True
             else:
-                self.log.info('Will retry Function App status check in %s seconds...' % delay)
+                self.log.info('(%s/%s) Will retry Function App status check in %s seconds...'
+                              % (r + 1, retries, delay))
                 time.sleep(delay)
         return False
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Mailer lambda throws invoke error due to AccessDeniedExceptionKMS . . . *sometimes*
We're seeing this error message once in a while during deployment testing. It can happen when no KMS key is set during the creation of a Lambda. It shows the KMS default key selected, but if we switch to another key and *back to the KMS default key*, the error goes away . . . which indicates that an empty key value is causing an error of some sort. It *SHOULD* be setting the default if it's empty, but not sure if this is happening.

![image](https://user-images.githubusercontent.com/1931732/44732430-421f1000-aab3-11e8-826e-2ec11f1659e3.png)

The reason this appears to be happening is because of `LAMBDA_EMPTY_VALUES` in `mu.py`:
```
 LAMBDA_EMPTY_VALUES = {
     'Environment': {'Variables': {}},
     'DeadLetterConfig': {},
     'TracingConfig': {'Mode': 'PassThrough'},
     'VpcConfig': {'SubnetIds': [], 'SecurityGroupIds': []},
     'KMSKeyArn': '',
 }
```
When I inspected the lambda right after deployment (and was getting the error), the lambda had `Environment': {'Variables': {}}`. Then I set the KMS key manually in the console and inspected the lambda again, but the entire `Environment` section had been _removed, which tells me that if there are no environment variables, AWS Lambda doesn't want an empty dictionary.

I"m going to submit a PR to prevent setting empty environment variables.


Discussion:

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -595,12 +595,14 @@ def get_config(self):
             'Handler': self.handler,
             'Timeout': self.timeout,
             'TracingConfig': self.tracing_config,
-            'Environment': self.environment,
             'KMSKeyArn': self.kms_key_arn,
             'DeadLetterConfig': self.dead_letter_config,
             'VpcConfig': LAMBDA_EMPTY_VALUES['VpcConfig'],
             'Tags': self.tags}
 
+        if self.environment['Variables']:
+            conf['Environment'] = self.environment
+
         if self.subnets and self.security_groups:
             conf['VpcConfig'] = {
                 'SubnetIds': self.subnets,


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -731,7 +731,6 @@ def test_config_defaults(self):
             {
                 "DeadLetterConfig": {},
                 "Description": "cloud-custodian lambda policy",
-                "Environment": {"Variables": {}},
                 "FunctionName": "custodian-hello",
                 "Handler": "custodian_policy.run",
                 "KMSKeyArn": "",
Repository: cloud-custodian/cloud-custodian

Issue:
Caught Error Allows For Continuation Instead of Exiting
Hello,

I recently ran into an issue where my policy, which is checking the tags on S3, deleted the bucket even though the tags were correct.  After consulting the logs, I found `"unable to invoke method:get_bucket_tagging error:We encountered an internal error. Please try again."`.  This in isolation is not an issue, but the code found here: https://github.com/capitalone/cloud-custodian/blob/master/c7n/resources/s3.py#L445 didn't actually stop custodian from progressing to the actions of the policy.  The actions are set to "remove-contents: true" & "delete".  Because of this, my bucket was deleted and the contents removed even though it was in-policy.  Is this going to continue to be the action going forward, or would you be open to a PR that allows it to fail hard instead of allowing the actions to occur?


Discussion:
In other words, the desired behavior is for Cloud Custodian to fail the policy execution thus not take any action if an AWS API call fails.
noted and agreed, the s3 api experienced an incident for a few hours yesterday on tag management. agreed we should just fail hard on unknown error. I'll send a pr in a few minutes.
Thank you for doing that.  Are there any other resources that have this same issue?

Resolving patch:
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -437,15 +437,23 @@ def assemble_bucket(item):
                 methods.append((m, k, default, select))
                 continue
             else:
-                if e.response['Error']['Code'] == 'AccessDenied':
-                    b.setdefault('c7n:DeniedMethods', []).append(m)
                 log.warning(
                     "Bucket:%s unable to invoke method:%s error:%s ",
                     b['Name'], m, e.response['Error']['Message'])
-                # We don't bail out, continue processing if we can.
+                # For auth failures, we don't bail out, continue processing if we can.
                 # Note this can lead to missing data, but in general is cleaner than
-                # failing hard.
-                continue
+                # failing hard, due to the common use of locked down s3 bucket policies
+                # that may cause issues fetching information across a fleet of buckets.
+
+                # This does mean s3 policies depending on augments should check denied
+                # methods annotation, generally though lacking get access to an augment means
+                # they won't have write access either.
+
+                # For other error types we raise and bail policy execution.
+                if e.response['Error']['Code'] == 'AccessDenied':
+                    b.setdefault('c7n:DeniedMethods', []).append(m)
+                    continue
+                raise
         # As soon as we learn location (which generally works)
         if k == 'Location' and v is not None:
             b_location = v.get('LocationConstraint')


Test patch: