internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
aws - alias elastic-ip to network-addr
the naming of network-addr is a bit idiosyncratic we should use the common name from the cloud-provider nomenclature re elastic-ip.


Discussion:

Resolving patch:
diff --git a/c7n/resources/resource_map.py b/c7n/resources/resource_map.py
--- a/c7n/resources/resource_map.py
+++ b/c7n/resources/resource_map.py
@@ -103,6 +103,7 @@
     "aws.ml-model": "c7n.resources.ml.MLModel",
     "aws.nat-gateway": "c7n.resources.vpc.NATGateway",
     "aws.network-acl": "c7n.resources.vpc.NetworkAcl",
+    "aws.elastic-ip": "c7n.resources.vpc.NetworkAddress",
     "aws.network-addr": "c7n.resources.vpc.NetworkAddress",
     "aws.ops-item": "c7n.resources.ssm.OpsItem",
     "aws.opswork-cm": "c7n.resources.opsworks.OpsworksCM",
diff --git a/c7n/resources/vpc.py b/c7n/resources/vpc.py
--- a/c7n/resources/vpc.py
+++ b/c7n/resources/vpc.py
@@ -1752,7 +1752,7 @@ def process(self, resources, event=None):
         return results
 
 
-@resources.register('network-addr')
+@resources.register('elastic-ip', aliases=('network-addr',))
 class NetworkAddress(query.QueryResourceManager):
 
     class resource_type(query.TypeInfo):
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -122,6 +122,7 @@
 html_theme_options = {
   'prev_next_buttons_location': 'both',
   'style_external_links': True,
+  'analytics_id': "UA-162730326-1",
   # Toc options
   'collapse_navigation': False,
   'sticky_navigation': True,


Test patch:
diff --git a/tests/test_vpc.py b/tests/test_vpc.py
--- a/tests/test_vpc.py
+++ b/tests/test_vpc.py
@@ -867,6 +867,13 @@ def test_release_detached_vpc(self):
         self.addCleanup(self.release_if_still_present, ec2, network_addr)
         self.assert_policy_released(factory, ec2, network_addr)
 
+    def test_elasticip_alias(self):
+        try:
+            self.load_policy({'name': 'eip', 'resource': 'aws.elastic-ip'}, validate=True)
+        except PolicyValidationError:
+            raise
+            self.fail("elastic ip alias failed")
+
     def test_release_attached_ec2(self):
         factory = self.replay_flight_data("test_release_attached_ec2")
Repository: cloud-custodian/cloud-custodian

Issue:
aws - add usgs additional partitions 

iso and isob are currently missing, its unclear if boto3 has support for them out of the box, golang and nodejs sdks do.


Discussion:
verified that it does https://github.com/boto/botocore/blob/develop/botocore/data/endpoints.json

Resolving patch:
diff --git a/c7n/utils.py b/c7n/utils.py
--- a/c7n/utils.py
+++ b/c7n/utils.py
@@ -327,7 +327,9 @@ def parse_s3(s3_path):
     'us-gov-east-1': 'aws-us-gov',
     'us-gov-west-1': 'aws-us-gov',
     'cn-north-1': 'aws-cn',
-    'cn-northwest-1': 'aws-cn'
+    'cn-northwest-1': 'aws-cn',
+    'us-isob-east-1': 'aws-iso-b',
+    'us-iso-east-1': 'aws-iso'
 }
 
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
core - source get_resources should early exit on empty set
some of the service apis will return back all resources if we issue an api call for an empty set of ids. when using get_resources we should explicitly check for an empty set and return in the base describe source / or query resource manager get_resources method.


Discussion:

Resolving patch:
diff --git a/c7n/query.py b/c7n/query.py
--- a/c7n/query.py
+++ b/c7n/query.py
@@ -491,6 +491,8 @@ def _get_cached_resources(self, ids):
         return None
 
     def get_resources(self, ids, cache=True, augment=True):
+        if not ids:
+            return []
         if cache:
             resources = self._get_cached_resources(ids)
             if resources is not None:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeLogStreams operation (reached max retries: 4): Rate exceeded
**Describe the bug**
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeLogStreams operation (reached max retries: 4): Rate exceeded

**To Reproduce**
Run the policy below

**Expected behavior**
Return a list of log groups that haven't been written to in 7 days

**Background (please complete the following information):**
 - OS: Windows
 - Python Version: Python 3.6.5
 - Custodian Version: 0.8.46.1
 - Tool Version: [if applicable]
 - Cloud Provider: aws
 - Policy: [please exclude any account/sensitive information]
```yaml
policies:
  - name: cloudwatch-log-group-stale-log-groups
    resource: log-group
    description: |
      Gets all Cloudwatch Logs Groups that haven't been written to in 7 days
    comments: |
      Gets all Cloudwatch Logs Groups that haven't been written to in 7 days
    mode:
      type: periodic
      role: MY_ROLE
      schedule: "rate(1 day)"
      timeout: 300
    filters:
      - type: last-write
        days: 7
```

**Additional context**
I see this is cited before as fixed, but I also see more improvement coming to retryies and backoff in https://github.com/cloud-custodian/cloud-custodian/pull/5564

I also have this happen on PutRetentionPolicy for log groups as well.




Discussion:

Resolving patch:
diff --git a/c7n/resources/cw.py b/c7n/resources/cw.py
--- a/c7n/resources/cw.py
+++ b/c7n/resources/cw.py
@@ -258,7 +258,8 @@ def process(self, resources, event=None):
         return [r for r in resources if self.check_group(client, r)]
 
     def check_group(self, client, group):
-        streams = client.describe_log_streams(
+        streams = self.manager.retry(
+            client.describe_log_streams,
             logGroupName=group['logGroupName'],
             orderBy='LastEventTime',
             descending=True,


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
copy-related-tags issue
I was wondering if you might be able to take a look at the following code snippet to see if there is something that I am doing wrong in order to get the copy-related-tags function to work from rest-apis to rest-stages? I have tried multiple variations with no success. https://gist.github.com/khapp/51adbba0236a6bafe67cdfd5fe8d1bd5


Discussion:
thanks for reporting this, what version of custodian and boto?, ie.  please add output of `custodian version --debug` to the gist
The gist has been updated with the requested information.  Thanks!

Resolving patch:
diff --git a/c7n/tags.py b/c7n/tags.py
--- a/c7n/tags.py
+++ b/c7n/tags.py
@@ -1066,10 +1066,12 @@ def process_resource(self, client, r, related_tags, tag_keys, tag_action):
                     if k in tag_keys and resource_tags.get(k) != v}
         if not tags:
             return
+        if not isinstance(tag_action, UniversalTag):
+            tags = [{'Key': k, 'Value': v} for k, v in tags.items()]
         tag_action.process_resource_set(
             client,
             resource_set=[r],
-            tags=[{'Key': k, 'Value': v} for k, v in tags.items()])
+            tags=tags)
         return True
 
     def get_resource_tag_map(self, r_type, ids):


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
tools/c7n-org - azure subscription generation includes disabled subscriptions

per report on gitter.

ngibbondaimler - We used azuresubs.py from c7n-org to generate a list of our subscriptions, however it's picking up disabled subscriptions and c7n-org throws an exception when it tries to read from a disabled sub to apply policy. Is there a suggested workaround for this?


Stefan Gordon -
I believe the return from the subscription API list call includes a state attribute, something like "state": "Enabled" - So for your scenario perhaps you can just add a check on that value at https://github.com/cloud-custodian/cloud-custodian/blob/master/tools/c7n_org/scripts/azuresubs.py#L34
Additionally if you can file an issue with the error you are getting in c7n-org I would say that we should update it to handle this error properly. Generating a list without those is an easy workaround but it shouldn't fail on them.



Discussion:

Resolving patch:
diff --git a/tools/c7n_org/scripts/azuresubs.py b/tools/c7n_org/scripts/azuresubs.py
--- a/tools/c7n_org/scripts/azuresubs.py
+++ b/tools/c7n_org/scripts/azuresubs.py
@@ -22,7 +22,12 @@
 @click.option(
     '-f', '--output', type=click.File('w'),
     help="File to store the generated config (default stdout)")
-def main(output):
+@click.option(
+    '-s', '--state', multiple=True, type=click.Choice(
+        ['Enabled', 'Warned', 'PastDue', 'Disabled', 'Deleted']),
+    default=('Enabled',),
+    help="File to store the generated config (default stdout)")
+def main(output, state):
     """
     Generate a c7n-org subscriptions config file
     """
@@ -31,6 +36,8 @@ def main(output):
     subs = [sub.serialize(True) for sub in client.subscriptions.list()]
     results = []
     for sub in subs:
+        if state and sub['state'] not in state:
+            continue
         sub_info = {
             'subscription_id': sub['subscriptionId'],
             'name': sub['displayName']


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
pin pytest to ~5.3.5 till 5.4.2 avail
there's a few issues with pytest 5.4 for some of our plugins and usage

- xdist compatibility
- pytest-sugar compatibility
- pytest-terraform compatibility
- unit test addCleanups not being run

the fixes have been merged into the 5.4 release branch, but if we want to use them we'll need to pin pytest. digging under the hood, there's quite an impressive amount of changes in the 5.4 release. most of these plugins are broken due to various usage of internal only apis, primarily around tmpdir or terminal writer interfaces.



Discussion:

Resolving patch:
diff --git a/c7n/resources/aws.py b/c7n/resources/aws.py
--- a/c7n/resources/aws.py
+++ b/c7n/resources/aws.py
@@ -30,7 +30,6 @@
 import traceback
 
 import boto3
-import botocore.client
 
 from botocore.validate import ParamValidator
 
@@ -380,54 +379,6 @@ def get_trace_entity(self):
         return self.handle_context_missing()
 
 
-if HAVE_XRAY:  # pragma: no cover
-    # sigh, work around aws xray sdk issues, via monkey patch
-    # https://github.com/aws/aws-xray-sdk-python/issues/207
-
-    # we can't do a minimal monkey, due to namespace occlusion they
-    # do with function imports masking modules :-(
-
-    # all of this to add a PutTraceSegments to the exempted methods list.
-    import wrapt
-
-    from aws_xray_sdk.ext.boto_utils import inject_header, aws_meta_processor
-    from aws_xray_sdk.ext import botocore as xray_boto3
-
-    def _xray_traced_botocore(wrapped, instance, args, kwargs):
-        service = instance._service_model.metadata["endpointPrefix"]
-        if service == 'xray':
-            # skip tracing for SDK built-in sampling pollers
-            if ('GetSamplingRules' in args or
-                'GetSamplingTargets' in args or
-                    'PutTraceSegments' in args):
-                return wrapped(*args, **kwargs)
-        return xray_recorder.record_subsegment(
-            wrapped, instance, args, kwargs,
-            name=service,
-            namespace='aws',
-            meta_processor=aws_meta_processor,
-        )
-
-    def _xray_patch_botocore():
-        if hasattr(botocore.client, '_xray_enabled'):
-            return
-        setattr(botocore.client, '_xray_enabled', True)
-
-        wrapt.wrap_function_wrapper(
-            'botocore.client',
-            'BaseClient._make_api_call',
-            _xray_traced_botocore,
-        )
-
-        wrapt.wrap_function_wrapper(
-            'botocore.endpoint',
-            'Endpoint.prepare_request',
-            inject_header,
-        )
-
-    xray_boto3.patch = _xray_patch_botocore
-
-
 @tracer_outputs.register('xray', condition=HAVE_XRAY)
 class XrayTracer:
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
network-addr filter 
In the network-addr filter, we're passing the alloc id as public ip. we should detect the value and pass as the correct param with a fix


Discussion:
thanks for filing, per discussion on gitter here's some more details


 lpendyala_gitlab 13:50
@kapilt even passing the allocationId getting this error event ids not resolved: ['eipalloc-0a7ac776b3uyg5eeb'] error:An error occurred (InvalidParameterValue) when calling the DescribeAddresses operation: Invalid value 'eipalloc-0a7ac776b3uyg5eeb' for PublicIp. Not a valid IPv4 address

Kapilt, For the above problem, when I modified this line in the code, it fixed.  Can you please take a look at it, test and merge to the master? 
c7n/resource/vpc.py 
   class resource_type(query.TypeInfo):
        service = 'ec2'
        arn_type = 'eip-allocation'
        enum_spec = ('describe_addresses', 'Addresses', None)
        name = 'PublicIp'
        id = 'AllocationId'
        **filter_name = 'AllocationIds'**
        filter_type = 'list'
        config_type = "AWS::EC2::EIP"
Previously filter_name = 'PublicIps'---- changed to filter_name = 'AllocationIds'. I ran test and that worked for me.
yeah we need to auto detect which value we're being given and pass to the correct parameter name.

Resolving patch:
diff --git a/c7n/resources/vpc.py b/c7n/resources/vpc.py
--- a/c7n/resources/vpc.py
+++ b/c7n/resources/vpc.py
@@ -1866,7 +1866,7 @@ class resource_type(query.TypeInfo):
         enum_spec = ('describe_addresses', 'Addresses', None)
         name = 'PublicIp'
         id = 'AllocationId'
-        filter_name = 'PublicIps'
+        filter_name = 'AllocationId'
         filter_type = 'list'
         config_type = "AWS::EC2::EIP"
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
json schema export ~ stable diff
running `custodian schema --json > schema.json`. results in a jsonschema file, doing it repeatedly results in minor deltas between the files primarily around ordering. this should be normalized to produce the same output, such that running two diff on exports with the same custodian version should have zero delta.





Discussion:

Resolving patch:
diff --git a/c7n/resources/iam.py b/c7n/resources/iam.py
--- a/c7n/resources/iam.py
+++ b/c7n/resources/iam.py
@@ -492,7 +492,7 @@ class ServiceUsage(Filter):
             {'type': 'number'},
             {'type': 'null'},
             {'$ref': '#/definitions/filters/value'}]}
-        for sa in SERVICE_ATTR}
+        for sa in sorted(SERVICE_ATTR)}
     schema_attr['match-operator'] = {'enum': ['all', 'any']}
     schema_attr['poll-delay'] = {'type': 'number'}
     schema = type_schema(
diff --git a/c7n/schema.py b/c7n/schema.py
--- a/c7n/schema.py
+++ b/c7n/schema.py
@@ -279,13 +279,12 @@ def generate(resource_types=()):
     }
 
     resource_refs = []
-    for cloud_name, cloud_type in clouds.items():
-        for type_name, resource_type in cloud_type.resources.items():
+    for cloud_name, cloud_type in sorted(clouds.items()):
+        for type_name, resource_type in sorted(cloud_type.resources.items()):
             r_type_name = "%s.%s" % (cloud_name, type_name)
             if resource_types and r_type_name not in resource_types:
                 if not resource_type.type_aliases:
                     continue
-                # atm only azure is using type aliases.
                 elif not {"%s.%s" % (cloud_name, ralias) for ralias
                         in resource_type.type_aliases}.intersection(
                         resource_types):


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Add Custodian Version To Mailer Events
**Is your feature request related to a problem? Please describe.**
Not related to a problem

**Describe the solution you'd like**
It would be nice if the Custodian version number was included in the mailer messages that way I could extract it and print it on the email.  This would allow me to identify what version of custodian a policy is running as in case some old policy lambdas don't get updated or deleted.  Being able to identify if a old version of a policy took some action by looking at the email would be very helpful.

**Describe alternatives you've considered**
I don't have any other ideas on how to get the custodian version in the email dynamically





Discussion:

Resolving patch:
diff --git a/c7n/actions/notify.py b/c7n/actions/notify.py
--- a/c7n/actions/notify.py
+++ b/c7n/actions/notify.py
@@ -22,6 +22,7 @@
 from c7n.exceptions import PolicyValidationError
 from c7n.manager import resources as aws_resources
 from c7n.resolver import ValuesFrom
+from c7n.version import version
 
 
 class BaseNotify(EventAction):
@@ -178,6 +179,7 @@ def process(self, resources, event=None):
             'event': event,
             'account_id': self.manager.config.account_id,
             'account': alias,
+            'version': version,
             'region': self.manager.config.region,
             'execution_id': self.manager.ctx.execution_id,
             'execution_start': self.manager.ctx.start_time,


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Codecommit and Codepipeline missing tag filters and actions in 0.9.2.0
**Describe the bug**
Codecommit and codepipeline tagging was added per the release notes but it doesn't come up in the schema when you do a custodian schema codecommit or codepipeline

**To Reproduce**
Steps to reproduce the behavior:
      custodian schema codecommit or codepipeline
**Expected behavior**
      Expecting to see the marked-for-op filter and tag and mark-for-op actions avaialble for the 2 resources


**Background (please complete the following information):**
 - OS: [e.g. OSX 10.15] Ubuntu v20
 - Python Version: [e.g. python 3.8.1] 3.8
 - Custodian Version: [e.g. 0.8.46.1]  0.9.2.0
 - Tool Version: [if applicable]
 - Cloud Provider: [e.g. gcp, aws, azure]  aws
 - Policy: [please exclude any account/sensitive information]
```yaml
policies: 
   - name: check-buckets
      resource: aws.s3
```
 - Traceback: [if applicable, please exclude sensitive/account information]
 - `custodian version --debug` output

**Additional context**
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/c7n/resources/code.py b/c7n/resources/code.py
--- a/c7n/resources/code.py
+++ b/c7n/resources/code.py
@@ -37,7 +37,7 @@ class resource_type(TypeInfo):
         arn = "Arn"
         date = 'creationDate'
         cfn_type = 'AWS::CodeCommit::Repository'
-        universal_tagging = object()
+        universal_taggable = object()
 
     def get_resources(self, ids, cache=True):
         return universal_augment(self, self.augment([{'repositoryName': i} for i in ids]))
@@ -215,7 +215,7 @@ class resource_type(TypeInfo):
         # Note this is purposeful, codepipeline don't have a separate type specifier.
         arn_type = ""
         cfn_type = config_type = "AWS::CodePipeline::Pipeline"
-        universal_tagging = object()
+        universal_taggable = object()
 
     source_mapping = {
         'describe': DescribePipeline,


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
AWS common filters page has a link pointing to 404 link
On this page: https://cloudcustodian.io/docs/aws/resources/aws-common-filters.html#aws-common-filters-config-compliance

There is a bit.ly link: https://bit.ly/2mblVpq

The bit.ly link points to a non-existent capitalone.github.io link (throws 404): http://capitalone.github.io/cloud-custodian/docs/policy/lambda.html#config-rules

I believe the bit.ly link should be updated to point to this page in the cloud custodian docs: https://cloudcustodian.io/docs/policy/lambda.html#config-rules
AWS common filters page has a link pointing to 404 link
On this page: https://cloudcustodian.io/docs/aws/resources/aws-common-filters.html#aws-common-filters-config-compliance

There is a bit.ly link: https://bit.ly/2mblVpq

The bit.ly link points to a non-existent capitalone.github.io link (throws 404): http://capitalone.github.io/cloud-custodian/docs/policy/lambda.html#config-rules

I believe the bit.ly link should be updated to point to this page in the cloud custodian docs: https://cloudcustodian.io/docs/policy/lambda.html#config-rules


Discussion:
thanks, for the bug report 
thanks, for the bug report 

Resolving patch:
diff --git a/c7n/filters/config.py b/c7n/filters/config.py
--- a/c7n/filters/config.py
+++ b/c7n/filters/config.py
@@ -45,7 +45,7 @@ class ConfigCompliance(Filter):
                - custodian-ec2-tags-required
 
     Also note, custodian has direct support for deploying policies as config
-    rules see https://bit.ly/2mblVpq
+    rules see https://cloudcustodian.io/docs/policy/lambda.html#config-rules
     """
     permissions = ('config:DescribeComplianceByConfigRule',)
     schema = type_schema(


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws - streaming-distribution filter and action for shield enablement
**Is your feature request related to a problem? Please describe.**
We would like to enable shield for aws.streaming-distribution

**Describe the solution you'd like**
Extend shield action/filter for the resource

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.



Discussion:

Resolving patch:
diff --git a/c7n/resources/cloudfront.py b/c7n/resources/cloudfront.py
--- a/c7n/resources/cloudfront.py
+++ b/c7n/resources/cloudfront.py
@@ -661,6 +661,10 @@ def process_distribution(self, client, distribution):
             raise e
 
 
+StreamingDistribution.filter_registry.register('shield-enabled', IsShieldProtected)
+StreamingDistribution.action_registry.register('set-shield', SetShieldProtection)
+
+
 @StreamingDistribution.action_registry.register('set-attributes')
 class StreamingDistributionUpdateAction(BaseUpdateAction):
     """Action to update the attributes of a distribution


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws.Iam-Role does not have mark-for-op 
Hi, 
Functionality for performing mark-for-op is not present for iam-role. 
Kindly see if this can be added. 


Discussion:

Resolving patch:
diff --git a/c7n/resources/iam.py b/c7n/resources/iam.py
--- a/c7n/resources/iam.py
+++ b/c7n/resources/iam.py
@@ -120,6 +120,10 @@ class resource_type(TypeInfo):
     }
 
 
+Role.action_registry.register('mark-for-op', TagDelayedAction)
+Role.filter_registry.register('marked-for-op', TagActionFilter)
+
+
 @Role.action_registry.register('post-finding')
 class RolePostFinding(OtherResourcePostFinding):
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
docs - value_from examples in value filter
the value_from source docs aren't making it through to the documentation page on value filter, those examples should ideally be there as its a common integration technique with external vocabularies.




Discussion:

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -16,6 +16,7 @@
 import os
 import shlex
 import sphinx_rtd_theme
+from sphinx.ext import autodoc
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -38,6 +39,17 @@
     'sphinx_markdown_tables'
 ]
 
+# Extract only a classes docstrings
+class DocsonlyMethodDocumenter(autodoc.MethodDocumenter):
+  objtype = "doconly"
+  content_indent = ""
+
+  def format_signature(self, **kwargs):
+    return ""
+
+  def add_directive_header(self, sig: str):
+    return None
+
 # Add any paths that contain templates here, relative to this directory.
 #templates_path = ['_templates']
 
@@ -303,3 +315,4 @@
 def setup(app):
     app.add_javascript('js/expand.js')
     app.add_stylesheet('css/expand.css')
+    app.add_autodocumenter(DocsonlyMethodDocumenter)


Test patch: