internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
on custodian report -s if individual policy output directory given and matches -p auto correct
per report on gitter around user difficulty, by default -s is supposed to match the parent dir where individual policy output goes, report takes a single policy though, so we should auto-correct if the user specifies the individual policy output directory.


Discussion:

Resolving patch:
diff --git a/c7n/commands.py b/c7n/commands.py
--- a/c7n/commands.py
+++ b/c7n/commands.py
@@ -125,6 +125,12 @@ def run(options, policies):
 def report(options, policies):
     assert len(policies) == 1, "Only one policy report at a time"
     policy = policies.pop()
+    odir = options.output_dir.rstrip(os.path.sep)
+    if os.path.sep in odir and os.path.basename(odir) == policy.name:
+        # policy sub-directory passed - ignore
+        options.output_dir = os.path.split(odir)[0]
+        # regenerate the execution context based on new path
+        policy = Policy(policy.data, options)
     d = datetime.now()
     delta = timedelta(days=options.days)
     begin_date = d - delta


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Autoscale invalid group TargetGroupArn errors
I'm trying to leverage the ASG invalid filter. It appears that it's bailing because I do not have any ALB in  the region and the key lookup fails.
https://github.com/capitalone/cloud-custodian/blob/master/c7n/resources/asg.py#L210 

policy
```
policies:
  - name: asg-invalid-config
    resource: asg
    filters:
      - invalid
    actions:
      - type: suspend
```

```
2017-01-06 20:10:44,873: custodian.resources.keypair:INFO Filtered from 110 to 110 keypair
2017-01-06 20:10:46,097: custodian.resources.elb:INFO Filtered from 2 to 2 elb
2017-01-06 20:10:47,047: custodian.resources.appelb:INFO Filtered from 1 to 1 appelb
2017-01-06 20:10:47,048: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/src/c7n/policy.py", line 174, in run
    resources = self.policy.resource_manager.resources()
  File "/src/c7n/query.py", line 171, in resources
    return self.filter_resources(resources)
  File "/src/c7n/manager.py", line 73, in filter_resources
    resources = f.process(resources, event)
  File "/src/c7n/resources/asg.py", line 236, in process
    self.initialize(asgs)
  File "/src/c7n/resources/asg.py", line 188, in initialize
    self.appelb_target_groups = self.get_appelb_target_groups()
  File "/src/c7n/resources/asg.py", line 210, in get_appelb_target_groups
    return set([a['TargetGroupArn'] for a in manager.resources()])
KeyError: 'TargetGroupArn'
```


Discussion:
it looks like i introduced this when refactoring some of the lazy imports, this got converted from a target group resource to an app elb resource. fix upcoming.

Resolving patch:
diff --git a/c7n/resources/asg.py b/c7n/resources/asg.py
--- a/c7n/resources/asg.py
+++ b/c7n/resources/asg.py
@@ -206,7 +206,7 @@ def get_elbs(self):
         return set([e['LoadBalancerName'] for e in manager.resources()])
 
     def get_appelb_target_groups(self):
-        manager = self.manager.get_resource_manager('app-elb')   
+        manager = self.manager.get_resource_manager('app-elb-target-group')
         return set([a['TargetGroupArn'] for a in manager.resources()])
 
     def get_images(self):


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
assume role doesn't work new account id detection on trunk

account id is picked up prior to the role assume


Discussion:
@scotwk if it helps, patched local using https://github.com/cmedley/cloud-custodian/commit/e14d8190b08684ca266736d8ac7c9789c82eff92  

Resolving patch:
diff --git a/c7n/cli.py b/c7n/cli.py
--- a/c7n/cli.py
+++ b/c7n/cli.py
@@ -116,6 +116,13 @@ def _default_region(options):
 
 
 def _default_account_id(options):
+    if options.assume_role:
+        try:
+            options.account_id = options.assume_role.split(':')[4]
+            return
+        except IndexError:
+            pass
+
     profile = getattr(options, 'profile', None)
     try:
         import boto3
@@ -321,8 +328,9 @@ def main():
     if getattr(options, 'config', None) is not None:
         options.configs.append(options.config)
 
-    _default_region(options)
-    _default_account_id(options)
+    if options.subparser in ('report', 'logs', 'metrics', 'run'):
+        _default_region(options)
+        _default_account_id(options)
 
     try:
         command = options.command


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
GMT time zone wrongly set
The _filters/offhours.py_ sets alias **gmt** to **Europe/London**. This is wrong imho.

It should be set to **Etc/GMT** resp. **Etc/UTC**. London is using summer time (BST) which is GMT+1.


Discussion:

Resolving patch:
diff --git a/c7n/filters/offhours.py b/c7n/filters/offhours.py
--- a/c7n/filters/offhours.py
+++ b/c7n/filters/offhours.py
@@ -184,8 +184,8 @@ class Time(Filter):
         'cdt': 'America/Chicago',
         'ct': 'America/Chicago',
         'mt': 'America/Denver',
-        'gmt': 'Europe/London',
-        'gt': 'Europe/London',
+        'gmt': 'Etc/GMT',
+        'gt': 'Etc/GMT',
         'bst': 'Europe/London',
         'ist': 'Europe/Dublin',
         'cet': 'Europe/Berlin',


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Lambda Traceback
local variable 'resources' referenced before assignment: UnboundLocalError
Traceback (most recent call last):
File "/var/task/custodian_policy.py", line 4, in run
return handler.dispatch_event(event, context)
File "/var/task/c7n/handler.py", line 105, in dispatch_event
p.push(event, context)
File "/var/task/c7n/policy.py", line 636, in push
return mode.run(event, lambda_ctx)
File "/var/task/c7n/policy.py", line 563, in run
return resources
UnboundLocalError: local variable 'resources' referenced before assignment


Discussion:
thanks for reporting, looks like for config rules, on certain evaluations we need to set a default on resources

Resolving patch:
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -530,6 +530,7 @@ def run(self, event, lambda_context):
         self.cfg_event = json.loads(event['invokingEvent'])
         cfg_item = self.cfg_event['configurationItem']
         evaluation = None
+        resources = []
         # TODO config resource type matches policy check
         if event['eventLeftScope'] or cfg_item['configurationItemStatus'] in (
                 "ResourceDeleted",


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Feature request - tenancy
Would be nice to look for resources with `dedicated|default|host` tenancy

* [ec2](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-instance.html)
* [rds](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_VPC.WorkingWithRDSInstanceinaVPC.html) *this is done at the vpc level*
* [redshift](http://docs.aws.amazon.com/redshift/latest/mgmt/managing-clusters-vpc.html) *this is done at the vpc level*


Discussion:

Resolving patch:
diff --git a/c7n/resources/ec2.py b/c7n/resources/ec2.py
--- a/c7n/resources/ec2.py
+++ b/c7n/resources/ec2.py
@@ -1166,6 +1166,7 @@ def process(self, instances):
     'tag-key': str,
     'tag-value': str,
     'tag:': str,
+    'tenancy': ('dedicated', 'default', 'host'),
     'vpc-id': str}
 
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Fixing another problem with validate dumping a stacktrace #1642
Issue #1642 

If a policy is missing the `name` field, validate would dump a stack trace.


Discussion:
multiple one line pull requests to the same file are a bit silly. also oddly inconsistent, in one case `<EMPTY>` and in the other line modified 'unknown'.
i'm sending a separate pr with both the fixes and another one liner as well.

Resolving patch:
diff --git a/c7n/commands.py b/c7n/commands.py
--- a/c7n/commands.py
+++ b/c7n/commands.py
@@ -167,7 +167,8 @@ def validate(options):
                 raise ValueError("The config file must end in .json, .yml or .yaml.")
 
         errors += schema.validate(data, schm)
-        conf_policy_names = {p['name'] for p in data.get('policies', ())}
+        conf_policy_names = {
+            p.get('name', 'unknown') for p in data.get('policies', ())}
         dupes = conf_policy_names.intersection(used_policy_names)
         if len(dupes) >= 1:
             errors.append(ValueError(
@@ -180,8 +181,8 @@ def validate(options):
             null_config = Bag(dryrun=True, log_group=None, cache=None, assume_role="na")
             for p in data.get('policies', ()):
                 try:
-                    p = Policy(p, null_config, Bag())
-                    p.validate()
+                    policy = Policy(p, null_config, Bag())
+                    policy.validate()
                 except Exception as e:
                     msg = "Policy: %s is invalid: %s" % (
                         p.get('name', 'unknown'), e)
diff --git a/c7n/resources/__init__.py b/c7n/resources/__init__.py
--- a/c7n/resources/__init__.py
+++ b/c7n/resources/__init__.py
@@ -85,7 +85,6 @@ def load_resources():
     import c7n.resources.waf
 
     # Load external plugins (private sdks etc)
-    print("load resources")
     from c7n.manager import resources
     resources.load_plugins()
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
CreateLoadBalancer event shortcut doesn't account for app-elb
The shortcut Cloudtrail event "CreateLoadBalancer" in cwe.py is specific to elb resources and causes a "Could not find resource id" error for any cloudtrail policy created for an app-elb resource.  This is because the resource-id match of "requestParameters.loadBalancerName" is only valid for the elb type; the corresponding match for app-elb resources is "requestParameters.name".  Even when I specify the correct "ids" value in my policy it is ignored because the shortcut list is checked first, the the CreateLoadBalancer event type is matched, and the specification in the policy is never checked.


Discussion:
you'll have to clarify, what issue your seeing if you specify the full event/ids/source in the policy.

looking through the code, I see that we only lookup in shortcuts if its a string is specified as the event instead of the full dictionary specification.

My policy contains:
```
    resource: app-elb
    mode:
      type: cloudtrail
      events:
        - event: CreateLoadBalancer
          source: "elasticloadbalancing.amazonaws.com"
          ids: "requestParameters.name"
```
Event in Lambda log in Cloudwatch contains:
```
        "eventSource": "elasticloadbalancing.amazonaws.com",
```
and
```
        "requestParameters": {
            "type": "application",
            "name": "my-alb-test1"
             ...
        },
```
However, no resource is found:
```
[INFO]	2017-07-08T23:01:32.259Z	611b185e-6431-11e7-9538-11132a999bad	Found resource ids: []
```

The code looked to me like it checked for a canned event first by event name, and only if it wasn't found would it check the full event spec:
```
    def get_trail_ids(cls, event, mode):
        """extract resources ids from a cloud trail event."""
        # Check if we have a short cut / alias
        info = CloudWatchEvents.match(event)
        if info:
            return info['ids'].search(event)
```
ah.. okay I was checking in a different location (cwe target generation), thanks for the additional detail.

Resolving patch:
diff --git a/c7n/cwe.py b/c7n/cwe.py
--- a/c7n/cwe.py
+++ b/c7n/cwe.py
@@ -114,15 +114,15 @@ def match(cls, event):
     @classmethod
     def get_trail_ids(cls, event, mode):
         """extract resources ids from a cloud trail event."""
-        # Check if we have a short cut / alias
-        info = CloudWatchEvents.match(event)
-        if info:
-            return info['ids'].search(event)
         resource_ids = ()
         event_name = event['detail']['eventName']
         event_source = event['detail']['eventSource']
         for e in mode.get('events', []):
             if not isinstance(e, dict):
+                # Check if we have a short cut / alias
+                info = CloudWatchEvents.match(event)
+                if info:
+                    return info['ids'].search(event)
                 continue
             if event_name != e.get('event'):
                 continue


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
resource:redshift-snapshot  tag policy failing
```
$ custodian run --debug --cache-period 0 --cache 0   -s out x.yaml
2017-10-25 21:22:37,492: custodian.policy:INFO policy: redshift-snapshot-tag-policy resource:redshift-snapshot region:eu-west-1 count:823 time:14.91
2017-10-25 21:22:37,668: custodian.actions:ERROR Exception with tags: [{u'Value': 'bi@abc.com', u'Key': 'Owner'}] on resources: rs:cloud-dwh-2017-10-25-15-10-38, rs:cloud-dwh-2017-10-24-09-10-31, rs:cloud-dwh-2017-10-20-23-40-50 
 u'SnapshotIdentifer'
2017-10-25 21:22:37,681: custodian.actions:ERROR Exception with tags: [{u'Value': 'bi@abc.com', u'Key': 'Owner'}] on resources: rs:cloud-dwh-2017-10-25-15-10-38, rs:cloud-dwh-2017-10-24-09-10-31, rs:cloud-dwh-2017-10-20-23-40-50 
 u'SnapshotIdentifer'
```

example
```
policies:
  - name: redshift-snapshot-tag-policy
    resource: redshift-snapshot
    comments: |
      Create tags for redshift-snapshot with bi
    filters:
      - or:
        - "tag:Owner": absent
        - "tag:Department": absent
    actions:
      - type: tag
        key: Owner
        value: bi@abc.com
      - type: tag
        key: Department
        value: bi

```




Discussion:

Resolving patch:
diff --git a/c7n/resources/redshift.py b/c7n/resources/redshift.py
--- a/c7n/resources/redshift.py
+++ b/c7n/resources/redshift.py
@@ -724,7 +724,7 @@ class RedshiftSnapshotTag(tags.Tag):
     def process_resource_set(self, resources, tags):
         client = local_session(self.manager.session_factory).client('redshift')
         for r in resources:
-            arn = self.manager.generate_arn(r['SnapshotIdentifer'])
+            arn = self.manager.generate_arn(r['SnapshotIdentifier'])
             client.create_tags(ResourceName=arn, Tags=tags)
 
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Port Metrics parameter over to c7n-org
Please add metrics parameter support for c7n-org tool.  Our senior leadership teams and our monitoring team would like metrics and dashboarding but it appears it doesn't currently work in c7n-org.   This would be a great addition to the tool.  Thanks Kapil!


Discussion:

Resolving patch:
diff --git a/tools/c7n_org/c7n_org/cli.py b/tools/c7n_org/c7n_org/cli.py
--- a/tools/c7n_org/c7n_org/cli.py
+++ b/tools/c7n_org/c7n_org/cli.py
@@ -319,7 +319,7 @@ def run_script(config, output_dir, accounts, tags, region, echo, serial, script_
                     a['name'], r, " ".join(script_args))
 
 
-def run_account(account, region, policies_config, output_path, cache_period, dryrun, debug):
+def run_account(account, region, policies_config, output_path, cache_period, metrics, dryrun, debug):
     """Execute a set of policies on an account.
     """
     logging.getLogger('custodian.output').setLevel(logging.ERROR + 1)
@@ -333,7 +333,7 @@ def run_account(account, region, policies_config, output_path, cache_period, dry
     bag = Bag.empty(
         region=region, assume_role=account['role'],
         cache_period=cache_period, dryrun=dryrun, output_dir=output_path,
-        account_id=account['account_id'], metrics_enabled=False,
+        account_id=account['account_id'], metrics_enabled=metrics,
         cache=cache_path, log_group=None, profile=None, external_id=None)
 
     policies = PolicyCollection.from_data(policies_config, bag)
@@ -381,10 +381,12 @@ def run_account(account, region, policies_config, output_path, cache_period, dry
 @click.option('-r', '--region', default=['us-east-1', 'us-west-2'], multiple=True)
 @click.option('-p', '--policy', multiple=True)
 @click.option('--cache-period', default=15, type=int)
+@click.option("--metrics", default=False, is_flag=True)
 @click.option("--dryrun", default=False, is_flag=True)
 @click.option('--debug', default=False, is_flag=True)
 @click.option('-v', '--verbose', default=False, help="Verbose", is_flag=True)
-def run(config, use, output_dir, accounts, tags, region, policy, cache_period, dryrun, debug, verbose):
+def run(config, use, output_dir, accounts, tags,
+        region, policy, cache_period, metrics, dryrun, debug, verbose):
     """run a custodian policy across accounts"""
     accounts_config, custodian_config, executor = init(
         config, use, debug, verbose, accounts, tags, policy)
@@ -400,6 +402,7 @@ def run(config, use, output_dir, accounts, tags, region, policy, cache_period, d
                     custodian_config,
                     output_dir,
                     cache_period,
+                    metrics,
                     dryrun,
                     debug)] = (a, r)
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
app-elb-target-group missing metrics filter which elb has
Expected to be able to reference a "HealthyHostCount" metric filter for app-elb-target-group but the filter is not available for this resource, while it is available for elb


Discussion:

Resolving patch:
diff --git a/c7n/resources/appelb.py b/c7n/resources/appelb.py
--- a/c7n/resources/appelb.py
+++ b/c7n/resources/appelb.py
@@ -23,7 +23,8 @@
 from collections import defaultdict
 from c7n.actions import ActionRegistry, BaseAction
 from c7n.filters import (
-    Filter, FilterRegistry, FilterValidationError, DefaultVpcBase, ValueFilter)
+    Filter, FilterRegistry, FilterValidationError, DefaultVpcBase,
+    MetricsFilter, ValueFilter)
 import c7n.filters.vpc as net_filters
 from c7n import tags
 from c7n.manager import resources
@@ -138,6 +139,23 @@ def _remove_appelb_tags(albs, session_factory, tag_keys):
 actions.register('set-shield', SetShieldProtection)
 
 
+@filters.register('metrics')
+class AppElbMetrics(MetricsFilter):
+    """Filter app load balancer by metric values.
+
+    See available metrics here: https://goo.gl/TLQ9Fr
+    Custodian defaults to specifying dimensions for the app elb only.
+    Target Group dimension not supported atm.
+    """
+
+    def get_dimensions(self, resource):
+        return [{
+            'Name': self.model.dimension,
+            'Value': 'app/%s/%s' % (
+                resource[self.model.name],
+                resource[self.model.id].rsplit('/')[-1])}]
+
+
 @filters.register('security-group')
 class SecurityGroupFilter(net_filters.SecurityGroupFilter):
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Custom Metrics Filter Needed For ElasticSearch service - Currently broken
No metrics are returned from ElasticSearch service as it requires 2 dimensions to be passed in.  Please create a separate metrics filter for this similar to shield's custom metrics filter

$Dimension1.Name = 'ClientId'
$Dimension1.Value = 'AccountNumber'

$Dimension2.Name = "DomainName"
$Dimension2.Value= "ESResourceName"


Discussion:

Resolving patch:
diff --git a/c7n/resources/elasticsearch.py b/c7n/resources/elasticsearch.py
--- a/c7n/resources/elasticsearch.py
+++ b/c7n/resources/elasticsearch.py
@@ -16,8 +16,10 @@
 import functools
 import logging
 import itertools
+
 from c7n.actions import Action
 from c7n.manager import resources
+from c7n.filters import MetricsFilter
 from c7n.query import QueryResourceManager
 from c7n.utils import (
     chunks, local_session, get_retry, type_schema, generate_arn)
@@ -77,6 +79,16 @@ def _augment(resource_set):
                 *w.map(_augment, chunks(domains, 5))))
 
 
+@ElasticSearchDomain.filter_registry.register('metrics')
+class Metrics(MetricsFilter):
+
+    def get_dimensions(self, resource):
+        return [{'Name': 'ClientId',
+                 'Value': self.manager.account_id},
+                {'Name': 'DomainName',
+                 'Value': resource['DomainName']}]
+
+
 @ElasticSearchDomain.action_registry.register('delete')
 class Delete(Action):
 


Test patch:
diff --git a/tests/test_elasticsearch.py b/tests/test_elasticsearch.py
--- a/tests/test_elasticsearch.py
+++ b/tests/test_elasticsearch.py
@@ -35,6 +35,26 @@ def test_resource_manager(self):
             resources[0]['Endpoint'].startswith(
                 'search-c7n-test-ug4l2nqtnwwrktaeagxsqso'))
 
+    def test_metrics_domain(self):
+        config = Config.empty(account_id='644160558196')        
+        factory = self.replay_flight_data('test_elasticsearch_delete')
+        p = self.load_policy({
+            'name': 'es-query',
+            'resource': 'elasticsearch',
+            'filters': [{
+                'type': 'metrics',
+                'name': 'SearchableDocuments',
+                'days': 4,
+                'period': 86400,
+                'value': 1000,
+                'op': 'less-than'}]},
+            session_factory=factory, config=config)
+        self.assertEqual(
+            p.resource_manager.filters[0].get_dimensions(
+                {'DomainName': 'foo'}),
+            [{'Name': 'ClientId', 'Value': '644160558196'},
+             {'Name': 'DomainName', 'Value': 'foo'}])
+
     def test_delete_search(self):
         config = Config.empty(account_id='644160558196')        
         factory = self.replay_flight_data('test_elasticsearch_delete')
Repository: cloud-custodian/cloud-custodian

Issue:
Filters not working for datapipeline resource ?
I tried to filter datapipeline resources based on certain tags, but it's not working.

```
policies:

  - name: datapipeline
    resource: datapipeline
    filters:
        - type: value
          key: "tag:Name"
          op: not-in
          value:
          - Value1
          - Value2
```


Discussion:

Resolving patch:
diff --git a/c7n/resources/datapipeline.py b/c7n/resources/datapipeline.py
--- a/c7n/resources/datapipeline.py
+++ b/c7n/resources/datapipeline.py
@@ -67,9 +67,12 @@ def process_tags(pipe_set):
                     results = {'TagDescriptions': []}
                     break
                 continue
+
         for pipe_desc in results['pipelineDescriptionList']:
             pipe = pipe_map[pipe_desc['pipelineId']]
-            pipe['tags'] = pipe_desc['tags']
+            pipe['Tags'] = [
+                {'Key': t['key'], 'Value': t['value']}
+                for t in pipe_desc['tags']]
             for field in pipe_desc['fields']:
                 key = field['key']
                 if not key.startswith('@'):


Test patch:
diff --git a/tests/test_datapipeline.py b/tests/test_datapipeline.py
--- a/tests/test_datapipeline.py
+++ b/tests/test_datapipeline.py
@@ -55,7 +55,7 @@ def test_reporting(self):
             'name': 'datapipeline-report',
             'resource': 'datapipeline',
             'filters': [
-                {'name': 'PipelinesFTW'}],
+                {'tag:foo': 'bar'}],
             },
             config={'region': 'us-west-2'},
             session_factory=factory)
@@ -63,7 +63,7 @@ def test_reporting(self):
         self.assertEqual(len(resources), 1)
         resource = resources[0]
         self.assertEqual(resource['name'], 'PipelinesFTW')
-        self.assertEqual(resource['tags'], [{'key': 'foo', 'value': 'bar'}])
+        self.assertEqual(resource['Tags'], [{'Key': 'foo', 'Value': 'bar'}])
         self.assertEqual(resource['lastActivationTime'], '2017-03-13T11:37:36')
         self.assertEqual(resource['creationTime'], '2017-03-13T11:37:34')
         self.assertEqual(resource['sphere'], 'PIPELINE')