internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
azure - functions increase timeout to consumption max
at the moment the default is 5m, but it can be configured up to 10m, we should default to the max supported by the intersection of plans, which is 10m

https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#timeout


Discussion:

Resolving patch:
diff --git a/tools/c7n_azure/c7n_azure/constants.py b/tools/c7n_azure/c7n_azure/constants.py
--- a/tools/c7n_azure/c7n_azure/constants.py
+++ b/tools/c7n_azure/c7n_azure/constants.py
@@ -111,9 +111,12 @@
         "healthCheckThreshold": 6,
         "counterThreshold": 0.80
     },
-    "functionTimeout": "00:05:00",
+    "functionTimeout": "00:10:00",
     "logging": {
-        "fileLoggingMode": "debugOnly"
+        "fileLoggingMode": "always",
+        "logLevel": {
+            "default": "Debug"
+        }
     },
     "extensions": {
         "http": {


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
r53domain throttling exception
I'm getting following error when trying to use r53domain -resource

`An error occurred (ThrottlingException) when calling the ListTagsForDomain operation (reached max retries: 4): Rate exceeded`

That account has a lot domains, most of our accounts works perfectly.

Any tips/help?


Discussion:
what's the policy (sanitized) look like?
I'm just listing all domains, so

```
policies:
  - name: route53-domains # List all registered domains
    resource: r53domain

```
okay, looks like we're missing a throttling retry here, though it also looks like r53domains are supported by the resource group tagging api

Resolving patch:
diff --git a/c7n/resources/route53.py b/c7n/resources/route53.py
--- a/c7n/resources/route53.py
+++ b/c7n/resources/route53.py
@@ -142,15 +142,11 @@ class resource_type(TypeInfo):
 
     def augment(self, domains):
         client = local_session(self.session_factory).client('route53domains')
-
-        def _list_tags(d):
-            tags = client.list_tags_for_domain(
+        for d in domains:
+            d['Tags'] = self.retry(
+                client.list_tags_for_domain,
                 DomainName=d['DomainName'])['TagList']
-            d['Tags'] = tags
-            return d
-
-        with self.executor_factory(max_workers=1) as w:
-            return list(filter(None, w.map(_list_tags, domains)))
+        return domains
 
 
 @Route53Domain.action_registry.register('tag')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
add python3.8 to the enum for runtime
python3.8 is supported by lambda per https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html and since their is a ci for python3.8 this should be easy


Discussion:

Resolving patch:
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -405,7 +405,8 @@ class LambdaMode(ServerlessExecutionMode):
             # Lambda passthrough config
             'layers': {'type': 'array', 'items': {'type': 'string'}},
             'concurrency': {'type': 'integer'},
-            'runtime': {'enum': ['python2.7', 'python3.6', 'python3.7']},
+            'runtime': {'enum': ['python2.7', 'python3.6',
+                                 'python3.7', 'python3.8']},
             'role': {'type': 'string'},
             'timeout': {'type': 'number'},
             'memory': {'type': 'number'},


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -55,12 +55,13 @@
 
 def test_generate_requirements():
     lines = generate_requirements(
-        'boto3', ignore=('docutils', 's3transfer'))
+        'boto3', ignore=('docutils', 's3transfer', 'six'))
     packages = []
     for l in lines.split('\n'):
         pkg_name, version = l.split('==')
         packages.append(pkg_name)
-    assert packages == ['botocore', 'jmespath']
+    assert set(packages) == set([
+        'botocore', 'jmespath', 'urllib3', 'python-dateutil'])
 
 
 class Publish(BaseTest):
Repository: cloud-custodian/cloud-custodian

Issue:
Enable the SecurityHub RecordState attribute to be configured by policy
Policies will Remediation actions should be enabled to invoke post-findings to archive the finding by specifying "RecordState=ARCHIVED" however the current code hardcodes the value to ACTIVE.


Discussion:

Resolving patch:
diff --git a/c7n/resources/securityhub.py b/c7n/resources/securityhub.py
--- a/c7n/resources/securityhub.py
+++ b/c7n/resources/securityhub.py
@@ -307,6 +307,10 @@ class PostFinding(Action):
             "type": "string",
             "enum": ["PASSED", "WARNING", "FAILED", "NOT_AVAILABLE"],
         },
+        record_state={
+            "type": "string", 'default': 'ACTIVE',
+            "enum": ["ACTIVE", "ARCHIVED"],
+        },
     )
 
     NEW_FINDING = 'New'
@@ -464,6 +468,8 @@ def get_finding(self, resources, existing_finding_id, created_at, updated_at):
             finding["Criticality"] = self.data["criticality"]
         if "compliance_status" in self.data:
             finding["Compliance"] = {"Status": self.data["compliance_status"]}
+        if "record_state" in self.data:
+            finding["RecordState"] = self.data["record_state"]
 
         fields = {
             'resource': policy.resource_type,


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
lambda assume role support AWS China partition
Currently the lambda assume role hard code with default partition name `aws`[1], it doesn't work in China partition.

[1] https://github.com/cloud-custodian/cloud-custodian/blob/master/c7n/policy.py#L953-L954


Discussion:

Resolving patch:
diff --git a/c7n/policy.py b/c7n/policy.py
--- a/c7n/policy.py
+++ b/c7n/policy.py
@@ -950,8 +950,9 @@ def get_variables(self, variables=None):
 
         if 'mode' in self.data:
             if 'role' in self.data['mode'] and not self.data['mode']['role'].startswith("arn:aws"):
-                self.data['mode']['role'] = "arn:aws:iam::%s:role/%s" % \
-                                            (self.options.account_id, self.data['mode']['role'])
+                partition = utils.get_partition(self.options.region)
+                self.data['mode']['role'] = "arn:%s:iam::%s:role/%s" % \
+                    (partition, self.options.account_id, self.data['mode']['role'])
 
         variables.update({
             # standard runtime variables for interpolation


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Issue - Incorrect release version
https://github.com/cloud-custodian/cloud-custodian/releases/tag/0.8.46.0

```
pip show c7n
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Name: c7n
Version: 0.8.46.0
Summary: Cloud Custodian - Policy Rules Engine
Home-page: https://github.com/cloud-custodian/cloud-custodian
Author: None
Author-email: None
License: Apache-2.0
Location: [REDACTED]
Requires: python-dateutil, jsonpatch, PyYAML, botocore, boto3, urllib3, argcomplete, jsonschema, certifi, tabulate
Required-by: c7n-org
```

However when you run  

> custodian version

 it still shows as **0.8.45.4**



Discussion:

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -34,8 +34,13 @@
 from concurrent.futures import ThreadPoolExecutor
 
 # We use this for freezing dependencies for serverless environments
-# that support service side building.
-# Its also used for release engineering on our pypi uploads
+# that support service side building. Its also used for release
+# engineering on our pypi uploads
+try:
+    FileNotFoundError
+except NameError:
+    FileNotFoundError = IOError
+
 try:
     import importlib_metadata as pkgmd
 except (ImportError, FileNotFoundError):
diff --git a/c7n/version.py b/c7n/version.py
--- a/c7n/version.py
+++ b/c7n/version.py
@@ -12,4 +12,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-version = u"0.8.45.4"
+version = u"0.8.46.1"
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@ def read(fname):
 
 setup(
     name="c7n",
-    version='0.8.46.0',
+    version='0.8.46.1',
     description="Cloud Custodian - Policy Rules Engine",
     long_description=read('README.md'),
     long_description_content_type='text/markdown',


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
docker build issues with setuptools_scm
````
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-rf3t8j_o/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-rf3t8j_o/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-req-build-rf3t8j_o/pip-egg-info
cwd: /tmp/pip-req-build-rf3t8j_o/
Complete output (29 lines):
/usr/local/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: 'extra_requires'
warnings.warn(msg)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-rf3t8j_o/setup.py", line 46, in <module>
":python_version<'3.3'": ["backports.functools-lru-cache"]
File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/local/lib/python3.7/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/local/lib/python3.7/site-packages/setuptools/dist.py", line 448, in __init__
k: v for k, v in attrs.items()
File "/usr/local/lib/python3.7/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/usr/local/lib/python3.7/site-packages/setuptools/dist.py", line 740, in finalize_options
ep.load()(self)
File "/usr/local/lib/python3.7/site-packages/setuptools/dist.py", line 747, in _finalize_setup_keywords
ep.load()(self, ep.name, value)
File "/tmp/pip-req-build-rf3t8j_o/.eggs/setuptools_scm-3.4.3-py3.7.egg/setuptools_scm/integration.py", line 17, in version_keyword
dist.metadata.version = _get_version(config)
File "/tmp/pip-req-build-rf3t8j_o/.eggs/setuptools_scm-3.4.3-py3.7.egg/setuptools_scm/__init__.py", line 147, in _get_version
parsed_version = _do_parse(config)
File "/tmp/pip-req-build-rf3t8j_o/.eggs/setuptools_scm-3.4.3-py3.7.egg/setuptools_scm/__init__.py", line 118, in _do_parse
"use git+https://github.com/user/proj.git#egg=proj" % config.absolute_root
LookupError: setuptools-scm was unable to detect version for '/tmp/pip-req-build-rf3t8j_o'.
Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.
For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
```


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@ def read(fname):
 
 setup(
     name="c7n",
-    use_scm_version={'write_to': 'c7n/version.py'},
+    use_scm_version={'write_to': 'c7n/version.py', 'fallback_version': '0.9.0-unreleased'},
     setup_requires=['setuptools_scm'],
     description="Cloud Custodian - Policy Rules Engine",
     long_description=read('README.md'),


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
custodian validate failed with latest docker image
We are using cloudcustodian/c7n:latest, and getting errors below:

> 09:39:41 + custodian validate policy.yaml
> 09:39:41 Traceback (most recent call last):
> 09:39:41   File "/usr/local/bin/custodian", line 8, in <module>
> 09:39:41     sys.exit(main())
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n/cli.py", line 374, in main
> 09:39:41     command(config)
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n/commands.py", line 204, in validate
> 09:39:41     load_resources()
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n/resources/__init__.py", line 113, in load_resources
> 09:39:41     initialize_azure()
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n_azure/entry.py", line 21, in initialize_azure
> 09:39:41     import c7n_azure.policy
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n_azure/policy.py", line 28, in <module>
> 09:39:41     from c7n_azure.resources.arm import ArmResourceManager
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n_azure/resources/arm.py", line 24, in <module>
> 09:39:41     from c7n_azure.query import QueryResourceManager, QueryMeta, ChildResourceManager, TypeInfo, \
> 09:39:41   File "/usr/local/lib/python3.7/site-packages/c7n_azure/query.py", line 397, in <module>
> 09:39:41     resources.subscribe(QueryResourceManager.register_actions_and_filters)
> 09:39:41 TypeError: subscribe() missing 1 required positional argument: 'func'


Discussion:
i'd recommend using a release tag for the moment (ie. cloudcustodian/c7n:0.8.46.1)

debugging this is showing some oddities in the docker build, in that the custodian version in latest image seems to be the last version published to pypi instead of the local version. 
the fallback version we're using c7n setup.py is not being recognized (0.9.0-unreleased), which is causing the setuptools to pickup the last pypi published version
```
  Found existing installation: c7n 0.9.0-unreleased                                                                                  
    Uninstalling c7n-0.9.0-unreleased:                                                                                                  
      Successfully uninstalled c7n-0.9.0-unreleased     
```
simply changing that value to `0.9.0dev` seems to do the trick.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -9,7 +9,7 @@ def read(fname):
 
 setup(
     name="c7n",
-    use_scm_version={'write_to': 'c7n/version.py', 'fallback_version': '0.9.0-unreleased'},
+    use_scm_version={'write_to': 'c7n/version.py', 'fallback_version': '0.9.0dev'},
     setup_requires=['setuptools_scm'],
     description="Cloud Custodian - Policy Rules Engine",
     long_description=read('README.md'),


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
gcp - cscc - post finding failing in c7n version 0.8.46.1
c7n version: 0.8.46.1


```yaml
policies:
  - name: buckets
     description: custodian
     resource: gcp.bucket
     actions:
       - type: post-finding
          org-domain: mydomainname.com
          org-id: 1234567891234
          category: mynewcategory
```

Company policy prevents me doing copy/paste, but the stack calls are:
```
c7n/cli.py 374
c7n/commands.py 141
c7n/commands.py 283
c7n/policy.py 1049
c7n/policy.py 320
c7n_gcp/actions/cscc.py 80
c7n_gcp/actions/core.py 73
c7n_gcp/actions/core.py 81
c7n_gcp/actions/core.py 87
c7n_gcp/client 392
<snip> ... </snip>
googleapiclient/http.py 856

googleapiclient.errors.HttpError:
<HttpError 400 when requesting https://securitycentre.googleapis.com/v1beta/organizations/1234567891234/sources/22222/findings?33333=id&alt=json returned "Source property is invalid. All names must start with a letter; contain only letters, numbers and underscores; and be at most 256 characters long">
```

Where orgid, sourceid and findingId are obfuscated here.








Discussion:
it looks like the service, broke compatibility on their api and changed to tighten up the allowed characters in a source property name, we have a `-` hypens in our names, I'll update to underscores.

Resolving patch:
diff --git a/tools/c7n_gcp/c7n_gcp/actions/cscc.py b/tools/c7n_gcp/c7n_gcp/actions/cscc.py
--- a/tools/c7n_gcp/c7n_gcp/actions/cscc.py
+++ b/tools/c7n_gcp/c7n_gcp/actions/cscc.py
@@ -155,9 +155,9 @@ def get_finding(self, resource):
             'category': self.data.get('category', self.DefaultCategory),
             'eventTime': datetime.datetime.utcnow().isoformat('T') + 'Z',
             'sourceProperties': {
-                'resource-type': self.manager.type,
+                'resource_type': self.manager.type,
                 'title': policy.data.get('title', policy.name),
-                'policy-name': policy.name,
+                'policy_name': policy.name,
                 'policy': json.dumps(policy.data)
             }
         }


Test patch:
diff --git a/tools/c7n_gcp/tests/gcp_common.py b/tools/c7n_gcp/tests/gcp_common.py
--- a/tools/c7n_gcp/tests/gcp_common.py
+++ b/tools/c7n_gcp/tests/gcp_common.py
@@ -19,13 +19,10 @@
 
 from c7n.testing import TestUtils
 from c7n.schema import generate
-from c7n.resources import load_resources
-
 from c7n_gcp.client import Session, LOCAL_THREAD
 
 from recorder import HttpRecorder, HttpReplay
 
-load_resources()
 
 PROJECT_ID = ""
Repository: cloud-custodian/cloud-custodian

Issue:
Put retention days on Cloud Watch logs. with the TRaceback
Here is the Policy that I'm running periodically. I'm getting throttling Errors.

policies:
  - name: Custodian-loggroup-retention
    resource: log-group
    description: |
        Checks log groups weekely and sets the log retention for log groups that doesn't have log retention set.
    mode:
       type: periodic
       schedule: "cron(0 12 ? * 2 *)"
       role: CustodianLambdaExecutionRole
       packages:
          - boto3
          - botocore
    filters:
      - type: value
         key: retentionInDays
         value: absent
    actions:
       - type: retention
         days: 400

**Here is the traceback.**

[DEBUG] 2020-02-10T16:00:25.670Z fc012410-e0d7-49a5-a52f-c8a996ffb8e8 metric:ResourceCount Count:820 policy:custodian-loggroup-retention restype:log-group scope:policy
[DEBUG] 2020-02-10T16:00:25.670Z fc012410-e0d7-49a5-a52f-c8a996ffb8e8 metric:PolicyException Count:1 policy:custodian-loggroup-retention restype:log-group
[DEBUG] 2020-02-10T16:00:25.670Z fc012410-e0d7-49a5-a52f-c8a996ffb8e8 metric:ApiCalls Count:110 policy:custodian-loggroup-retention restype:log-group
[ERROR] 2020-02-10T16:00:25.670Z fc012410-e0d7-49a5-a52f-c8a996ffb8e8 Error while executing policy
Traceback (most recent call last):
File "/var/task/c7n/policy.py", line 320, in run
results = a.process(resources)
File "/var/task/c7n/resources/cw.py", line 201, in process
retentionInDays=days)
File "/var/task/botocore/client.py", line 276, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/task/botocore/client.py", line 586, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the PutRetentionPolicy operation (reached max retries: 4): Rate exceeded




Discussion:

Resolving patch:
diff --git a/c7n/resources/cw.py b/c7n/resources/cw.py
--- a/c7n/resources/cw.py
+++ b/c7n/resources/cw.py
@@ -196,7 +196,8 @@ def process(self, resources):
         client = local_session(self.manager.session_factory).client('logs')
         days = self.data['days']
         for r in resources:
-            client.put_retention_policy(
+            self.manager.retry(
+                client.put_retention_policy,
                 logGroupName=r['logGroupName'],
                 retentionInDays=days)
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
core - AWS Notify action is registered for all providers
AWS Notify action is registered for all providers while it has a lot of AWS specifics.. 

We need to think about the good strategy for actions that share the name, but differ in their schemas for different providers..


Discussion:

Resolving patch:
diff --git a/c7n/actions/core.py b/c7n/actions/core.py
--- a/c7n/actions/core.py
+++ b/c7n/actions/core.py
@@ -28,9 +28,7 @@ class ActionRegistry(PluginRegistry):
     def __init__(self, *args, **kw):
         super(ActionRegistry, self).__init__(*args, **kw)
         # Defer to provider initialization of registry
-        from .notify import Notify
         from .webhook import Webhook
-        self.register('notify', Notify)
         self.register('webhook', Webhook)
 
     def parse(self, data, manager):
diff --git a/c7n/actions/notify.py b/c7n/actions/notify.py
--- a/c7n/actions/notify.py
+++ b/c7n/actions/notify.py
@@ -20,6 +20,7 @@
 from .core import EventAction
 from c7n import utils
 from c7n.exceptions import PolicyValidationError
+from c7n.manager import resources as aws_resources
 from c7n.resolver import ValuesFrom
 
 
@@ -299,3 +300,13 @@ def send_sqs(self, message):
             MessageBody=self.pack(message),
             MessageAttributes=attrs)
         return result['MessageId']
+
+    @classmethod
+    def register_resource(cls, registry, resource_class):
+        if 'notify' in resource_class.action_registry:
+            return
+
+        resource_class.action_registry.register('notify', cls)
+
+
+aws_resources.subscribe(Notify.register_resource)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
IAM filtering potential fix for 'all' operator
Filtering iam-roles for all usage greater than a certain date did not function as I believed it should.  To my understanding this filter should show roles with no usage for the last 5 days, but was showing roles that had been used.

```yaml
resource: iam-role
    filters:
      - type: usage
        match-operator: all
        LastAuthenticated:
          type: value
          value_type: age
          op: greater-than
          value: 5
```

There were 2 changes that enabled this filter to function per my understanding.

c7n/resources/iam.py:469 - Add an additional "and match_operator!= 'all'" to the condition so that this will not fire for 'all' operators that only returned a partial number of true responses.

c7n/filters/core.py:616 - change 0 to "parse_date('1970-01-01 00:00:00.000000+00:00')" so that comparisons to other datetime objects will compare against a pre-cloud date for null values rather than error out for non-comparable types.



Discussion:
fwiw, iam roles have native attributes for last usage date now which post-date the apis that usage filter users, ie. you can simply use

```
type: value
key: RoleLastUsed.LastUsedDate
value_type: age
op: greater-than
value: 5
```
which will use significantly less api calls, as a usage filter has to do two api calls per resource.


regarding usage filter.

iam - I don't see what you mean.

 -  a) please provide details/data of the issue, rather than writing code in prose.
 -  b) rather than list line numbers that that will be broken in a day,. 
         send a pr or link to a particular line and use actual code.

core - converting nulls to default arbitrary date values is highly implicit, if you want to distinguish be explicit. This is exactly why the filter example has `TotalAuthenticatedEntities: 1`

please use an bug issue template (versions, valid policy) and supply a representative sample of the usage report of the role in question that you feel like is being matched erroneously. also note per the filter docs the underlying data from aws that this based on may be hours out of date.


Resolving patch:
diff --git a/c7n/resources/iam.py b/c7n/resources/iam.py
--- a/c7n/resources/iam.py
+++ b/c7n/resources/iam.py
@@ -466,7 +466,7 @@ def process(self, resources, event=None):
                 saf_matches = saf.process(saf_results)
                 if match_operator == 'all' and len(saf_matches) == len(saf_results):
                     results.append(r)
-                elif saf_matches:
+                elif match_operator != 'all' and saf_matches:
                     results.append(r)
 
             time.sleep(self.data.get('poll-delay', 2))


Test patch: