internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
code coverage issues for master/trunk commits
Nutshell master commits aren't being uploaded to coverage which is causing some drift on the prs, as there isn't a proper ref to baseline coverage from.

Example master build
https://dev.azure.com/cloud-custodian/cloud-custodian/_build/results?buildId=110


from the coverage upload attempt, it looks like we end up inserting some magic string literal value for system.pullRequest.pullRequestNumber ideally it would be empty. we may need to have a separate front end script to invoke codecov

```
2018-12-26T15:01:42.7646941Z     ‌+‌ TOXENV‌
2018-12-26T15:01:42.7647543Z ‌==>‌ Uploading‌
2018-12-26T15:01:42.7648015Z     .url https://codecov.io
2018-12-26T15:01:42.7648206Z     .query pr=%24%28system.pullRequest.pullRequestNumber%29&package=py2.0.15&yaml=.codecov.yml&token=<secret>&build=20181226.3&branch=%24%28system.pullRequest.sourceBranch%29&commit=%24%28system.pullRequest.sourceCommitId%29
2018-12-26T15:01:42.7648414Z     Pinging Codecov...
2018-12-26T15:01:42.7648799Z ‌Error:‌ HTTP 400‌
2018-12-26T15:01:42.7648997Z pr must match pattern ^(\d+|false|null|undefined|true)$
```


Discussion:

Resolving patch:
diff --git a/tools/dev/cicov.py b/tools/dev/cicov.py
new file mode 100755
--- /dev/null
+++ b/tools/dev/cicov.py
@@ -0,0 +1,49 @@
+#!/usr/bin/env python2
+
+import subprocess
+import os
+import logging
+
+log = logging.getLogger('cicov')
+
+# https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts
+
+
+config = {
+    'branch': os.environ.get('BRANCH'),
+    'pr': os.environ.get('PR'),
+    'build': os.environ.get('BUILD_ID'),
+    'commit': os.environ.get('BUILD_COMMIT')}
+
+UNSET = "system.pullRequest"
+
+
+def main():
+    logging.basicConfig(level=logging.INFO)
+
+    # Foresenics on the variables set in a context
+    for k in ('BUILD_BRANCH', 'PR', 'BUILD_ID', 'BUILD_COMMIT', 'COMMIT', 'BRANCH'):
+        v = os.environ.get(k)
+        log.info("Env var %s=%s" % (k, v))
+
+    # Defer to pull request branch (system.pullRequest), if thats not set
+    # then use Build.SourceBranchName
+    if UNSET in config['branch']:
+        config['branch'] = os.environ['BUILD_BRANCH']
+
+    args = ['codecov']
+    # Assemble cli args, skip all unset values.
+    for k, v in config.items():
+        if not v:
+            continue
+        # value not set
+        if UNSET in v:
+            continue
+        args.append("--%s" % k)
+        args.append(v)
+    log.info("Uploading CodeCoverage: %r" % (' '.join(args)))
+    subprocess.check_call(args)
+
+
+if __name__ == '__main__':
+    main()


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Azure - duplicate packages causing failed download
**Issue**

When putting together package lists, duplicates exist in the list causing the following error message during deployment:

```
c7n_mailer.azure.deploy - INFO - Building function package for mailer1-mailer-76d56b36
custodian.azure.function_package - INFO - Cached packages not found or requirements were changed.
custodian.azure.function_package - INFO - Preparing non binary wheels...
custodian.azure.function_package - INFO - Downloading wheels...
c7n_azure.dependency_manager - ERROR - Failed to download wheels!
```

**Root Cause**

Static method get_dependency_packages_list was de-duping the packaeg list but not taking into consideration the version #

**Resolution**

Using regex to match on package name and ignore version. If duplicate package is found it is removed from the list. 

PR shortly to resolve issue

@aluong @logachev
Azure - duplicate packages causing failed download
**Issue**

When putting together package lists, duplicates exist in the list causing the following error message during deployment:

```
c7n_mailer.azure.deploy - INFO - Building function package for mailer1-mailer-76d56b36
custodian.azure.function_package - INFO - Cached packages not found or requirements were changed.
custodian.azure.function_package - INFO - Preparing non binary wheels...
custodian.azure.function_package - INFO - Downloading wheels...
c7n_azure.dependency_manager - ERROR - Failed to download wheels!
```

**Root Cause**

Static method get_dependency_packages_list was de-duping the packaeg list but not taking into consideration the version #

**Resolution**

Using regex to match on package name and ignore version. If duplicate package is found it is removed from the list. 

PR shortly to resolve issue

@aluong @logachev


Discussion:

Resolving patch:
diff --git a/tools/c7n_azure/c7n_azure/dependency_manager.py b/tools/c7n_azure/c7n_azure/dependency_manager.py
--- a/tools/c7n_azure/c7n_azure/dependency_manager.py
+++ b/tools/c7n_azure/c7n_azure/dependency_manager.py
@@ -42,7 +42,13 @@ def get_dependency_packages_list(packages, excluded_packages):
 
         # boto3 is a dependency for both c7n and c7n_mailer.. Remove the duplicate from the list
         # because not all versions of pip can handle this.
-        res = [t for t in res if not any((e != t and t in e) for e in res)]
+        # use regex to get rid of duplicates excluding version number
+        regex = "^[^<>~=]*"
+        for i, val in enumerate(res):
+            pname = re.match(regex, val)
+            if sum(pname[0].lower() in e.lower() for e in res) > 1:
+                logger.debug("removing duplicate dependency:" + val)
+                res.pop(i)
         return res
 
     @staticmethod


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Wrong retry code in rds.py at describe_db_instances
In rds.py, class RDS(QueryResourceManager), line 115 the following is coded:
retry = staticmethod(get_retry(('Throttled',)))

This should be :
retry = staticmethod(get_retry(('Throttling',)))
or:
retry = staticmethod(get_retry(('Throttled','Throttling')))


In utils.py this text is compared to ClientError.response['Error']['Code']
The error code that is returned from the boto call is 'Throttling'.


Evidence:

import boto3
from datetime import datetime
from botocore.exceptions import ClientError

rdsclient = boto3.client('rds', region_name='eu-west-1')

bool_retry  = True
while bool_retry:
    try:
        dbinstances = rdsclient.describe_db_instances()
    except ClientError as ce:
        bool_retry = False
        print ('%s - ClientError - %s' % (datetime.utcnow() , ce ))
        print ('%s - ClientError [Error][Code] - %s' % (datetime.utcnow() , ce.response['Error']['Code'] ))

2019-01-10 15:25:09.445489 - ClientError - An error occurred (Throttling) when calling the DescribeDBInstances operation (reached max retries: 4): Rate exceeded
2019-01-10 15:25:09.445539 - ClientError [Error][Code] - Throttling




Discussion:
we can probably remove that one, there's one in the base class thats a bit more comprehensive wrt to error codes signaling throttle and retries.
https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/CommonErrors.html

Resolving patch:
diff --git a/c7n/query.py b/c7n/query.py
--- a/c7n/query.py
+++ b/c7n/query.py
@@ -350,6 +350,7 @@ class QueryResourceManager(ResourceManager):
             'ThrottlingException',
             'RequestLimitExceeded',
             'Throttled',
+            'Throttling',
             'Client.RequestLimitExceeded')))
 
     def __init__(self, data, options):
diff --git a/c7n/resources/rds.py b/c7n/resources/rds.py
--- a/c7n/resources/rds.py
+++ b/c7n/resources/rds.py
@@ -114,7 +114,6 @@ class resource_type(object):
     filter_registry = filters
     action_registry = actions
     _generate_arn = None
-    retry = staticmethod(get_retry(('Throttled',)))
 
     def __init__(self, data, options):
         super(RDS, self).__init__(data, options)


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
aws.support-case resource returns zero resources even with empty filter
The aws.support-case resource is returning zero results, both for a policy with nothing in the filter section as well as when filter is just 
```
   filters:
      - type: value
        key: "status"
        value: not-null
```
noticed the c7n/resources/support.py code is very minimal and there is no unit test


Discussion:

Resolving patch:
diff --git a/c7n/resources/support.py b/c7n/resources/support.py
--- a/c7n/resources/support.py
+++ b/c7n/resources/support.py
@@ -22,7 +22,7 @@ class SupportCase(QueryResourceManager):
 
     class resource_type(object):
         service = 'support'
-        enum_spec = ('describe_cases', 'q', None)
+        enum_spec = ('describe_cases', 'cases', None)
         filter_name = 'caseIdList'
         filter_type = 'list'
         id = 'caseId'


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
ci - failure around mismatched versions of tabulate
Per current ci failures we're getting 0.8.3 of tabulate installed even though azure-cli-core calls out a pin to under 0.8.2.

This mirrors the issue we had with fakeredis, where it properly declared a dependency for six == 0.12.0 and we picked up the version pin in requirements.txt.

digging around a bit more, pip released a new 19 release series in the last 72hrs, that i'm currently examining for regressions that allowed for installs that ignore package dependencies, when given requirements.



Discussion:
verified same issue presents with 18.1, checking release history on tabulate shows a release in the last 24hrs so seems like a coincident, due to lack of proper pin. adding a pin for now to requirements.txt and setup.py of c7n, with and followup issue to look at/revisit some of the azure pins (adal in c7n_azure and tabulate in azure-cli-core), also custodian pins on flake8, and sphinx

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -30,6 +30,6 @@ def read(fname):
         "jsonschema",
         "jsonpatch>=1.21",
         "argcomplete",
-        "tabulate"
+        "tabulate==0.8.2"
     ],
 )


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Govcloud SNS prefix incorrect
Hello,

It seems that cloud custodian is putting the SNS arn prefix incorrectly for govcloud. This is the error I'm getting:

botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameter) when calling the Publish operation: Invalid parameter: TopicArn Reason: A us-gov-west-1 ARN must begin with arn:aws-us-gov, not arn:aws:sns:us-gov-west-1:xxxxxxxxxxx:snsname


policies:
  - name: custodian-tags
    resource: ec2
    filters:
      - or:
        - "tag:CustodianTest": present
        - "tag:CustodianTest2": present
    actions:
      - type: notify
        to:
          - my@email.com
        template: default
        transport:
          type: sns
          region: us-gov-west-1
          topic: snsname



Discussion:
thanks for the bug report, we need to take into account partition when constructing sns arns.
the easy work around here is to specify the full arn in your notify policy, account_id interpolation still works for using policies across accounts.
hmmm.. actually it looks like that won't work, the arn check is qualified with partition.
Correct, when I use the full ARN is still adds the prefix. 

Resolving patch:
diff --git a/c7n/actions/notify.py b/c7n/actions/notify.py
--- a/c7n/actions/notify.py
+++ b/c7n/actions/notify.py
@@ -232,7 +232,7 @@ def send_data_message(self, message):
     def send_sns(self, message):
         topic = self.data['transport']['topic'].format(**message)
         user_attributes = self.data['transport'].get('attributes')
-        if topic.startswith('arn:aws:sns'):
+        if topic.startswith('arn:'):
             region = region = topic.split(':', 5)[3]
             topic_arn = topic
         else:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
jsonschema policy mode helper
originally we had just a handful of modes, so an error on the mode block was still readable/interpretable, now that we have a dozen plus execution modes, thats not the case, so in the same manner that we do helpers for actions/filters we should also do one for mode to give the user a clearer error message.


Discussion:

Resolving patch:
diff --git a/c7n/schema.py b/c7n/schema.py
--- a/c7n/schema.py
+++ b/c7n/schema.py
@@ -91,6 +91,7 @@ def specific_error(error):
         return error
 
     r = t = None
+
     if isinstance(error.instance, dict):
         t = error.instance.get('type')
         r = error.instance.get('resource')
@@ -118,6 +119,9 @@ def specific_error(error):
             if '$ref' in v and v['$ref'].rsplit('/', 2)[-1] == t:
                 found = idx
                 break
+            elif 'type' in v and t in v['properties']['type']['enum']:
+                found = idx
+                break
 
         if found is not None:
             for e in error.context:


Test patch:
diff --git a/tests/test_schema.py b/tests/test_schema.py
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -87,6 +87,23 @@ def test_py3_policy_error(self):
         self.assertTrue("'asdf' is not of type 'boolean'" in str(err).replace("u'", "'"))
         self.assertEqual(policy, 'policy-ec2')
 
+    def test_semantic_mode_error(self):
+        data = {
+            'policies': [{
+                'name': 'test',
+                'resource': 'ec2',
+                'mode': {
+                    'type': 'periodic',
+                    'scheduled': 'oops'}}]}
+        errors = list(self.validator.iter_errors(data))
+        self.assertEqual(len(errors), 1)
+        error = specific_error(errors[0])
+        self.assertTrue(
+            len(errors[0].absolute_schema_path) < len(error.absolute_schema_path)
+        )
+        self.assertTrue("'scheduled' was unexpected" in str(error))
+        self.assertTrue(len(str(error)) < 2000)
+
     def test_semantic_error(self):
         data = {
             "policies": [
Repository: cloud-custodian/cloud-custodian

Issue:
asg resource doesn't follow the standard pattern for `remove-tag` action
All resources support the action `remove-tag` via the following syntax:
```
actions:
  - type: remove-tag
    tags:
      - tag1
      - tag2
```
Except for asg which expects the following syntax:
```
actions:
  - type: remove-tag
    key: tag1
```
It would be nice to have consistency around the `remove-tag` action.


Discussion:

Resolving patch:
diff --git a/c7n/resources/asg.py b/c7n/resources/asg.py
--- a/c7n/resources/asg.py
+++ b/c7n/resources/asg.py
@@ -1009,20 +1009,24 @@ class RemoveTag(Action):
     schema = type_schema(
         'remove-tag',
         aliases=('untag', 'unmark'),
+        tags={'type': 'array', 'items': {'type': 'string'}},
         key={'type': 'string'})
+
     permissions = ('autoscaling:DeleteTags',)
     batch_size = 1
 
     def process(self, asgs):
         error = False
-        key = self.data.get('key', DEFAULT_TAG)
+        tags = self.data.get('tags', [])
+        if not tags:
+            tags = [self.data.get('key', DEFAULT_TAG)]
         client = local_session(self.manager.session_factory).client('autoscaling')
 
         with self.executor_factory(max_workers=2) as w:
             futures = {}
             for asg_set in chunks(asgs, self.batch_size):
                 futures[w.submit(
-                    self.process_asg_set, client, asg_set, key)] = asg_set
+                    self.process_asg_set, client, asg_set, tags)] = asg_set
             for f in as_completed(futures):
                 asg_set = futures[f]
                 if f.exception():
@@ -1036,11 +1040,14 @@ def process(self, asgs):
         if error:
             raise error
 
-    def process_asg_set(self, client, asgs, key):
-        tags = [dict(
-            Key=key, ResourceType='auto-scaling-group',
-            ResourceId=a['AutoScalingGroupName']) for a in asgs]
-        self.manager.retry(client.delete_tags, Tags=tags)
+    def process_asg_set(self, client, asgs, tags):
+        tag_set = []
+        for a in asgs:
+            for t in tags:
+                tag_set.append(dict(
+                    Key=t, ResourceType='auto-scaling-group',
+                    ResourceId=a['AutoScalingGroupName']))
+        self.manager.retry(client.delete_tags, Tags=tag_set)
 
 
 @ASG.action_registry.register('tag')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
post-finding failed action: maximum of 10 IPv4 addresses
Hi,

The finding format in Security Hub defines a maximum of 10 IPv4 addresses in `Resource.Details.AwsEc2Instance.IpV4Addresses` but the resource **ec2** (/c7n/resources/ec2.py) doesn't limit the list in the class **InstanceFinding**

See https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-findings-format.html

A dirty workaround is to change the code as below. I ignore if jmespath.search can limit the result:

```
            "IpV4Addresses": jmespath.search(
                "NetworkInterfaces[].PrivateIpAddresses[].PrivateIpAddress", r
            )[0:9]
```

```
FailedFindings': [{'Id': 'eu-central-1/897816849202/61d1691981c34026571dfa7b356f679b/f33f01673d01baefed3cce37deb0b59a', 'ErrorCode': 'InvalidInput', 'ErrorMessage': 'Finding does not adhere to Amazon Finding Format. data.Resources[0].Details.AwsEc2Instance.IpV4Addresses should NOT have more than 10 items'}]
```


Discussion:
thanks for the bug report, `[:10]` should do it, although i'm not a huge fan of arbitrary data omission,  given the service limitation and the relative infrequency of the condition, its unclear that we can do anything else that would be better.

Resolving patch:
diff --git a/c7n/resources/ec2.py b/c7n/resources/ec2.py
--- a/c7n/resources/ec2.py
+++ b/c7n/resources/ec2.py
@@ -867,15 +867,19 @@ def process_resource_set(self, client, resources):
 @EC2.action_registry.register("post-finding")
 class InstanceFinding(PostFinding):
     def format_resource(self, r):
+        ip_addresses = jmespath.search(
+            "NetworkInterfaces[].PrivateIpAddresses[].PrivateIpAddress", r)
+
+        # limit to max 10 ip addresses, per security hub service limits
+        ip_addresses = ip_addresses and ip_addresses[:10] or ip_addresses
         details = {
             "Type": r["InstanceType"],
             "ImageId": r["ImageId"],
-            "IpV4Addresses": jmespath.search(
-                "NetworkInterfaces[].PrivateIpAddresses[].PrivateIpAddress", r
-            ),
+            "IpV4Addresses": ip_addresses,
             "KeyName": r.get("KeyName"),
-            "LaunchedAt": r["LaunchTime"].isoformat(),
+            "LaunchedAt": r["LaunchTime"].isoformat()
         }
+
         if "VpcId" in r:
             details["VpcId"] = r["VpcId"]
         if "SubnetId" in r:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Guard-duty filter is not returning master information
Version = 0.8.42.0
Issue = When using the following guardduty filter it for a member account it is not returning the master account information.  

```
    filters:
      - type: guard-duty
        Detector.Status: ENABLED
        Master.AccountId: "xxx"
        Master.RelationshipStatus: "Enabled"
```
Example resource.json which should show master information
```
  {
    "c7n:guard-duty": {
      "Detector": {
        "Status": "ENABLED",
        "ServiceRole": "arn:aws:iam::xxx:role/aws-service-role/guardduty.amazonaws.com/AWSServiceRoleForAmazonGuardDuty",
        "UpdatedAt": "2018-11-02T10:16:15.132Z",
        "FindingPublishingFrequency": "SIX_HOURS",
        "CreatedAt": "2018-11-02T10:16:15.132Z"
      },
      "Master": null
    },
    "account_id": "xxx",
    "account_name": "xxx"
  }
]
```
Using AWS CLI and the ```get-master-account``` I can see ```Master``` information for this account
```
{
    "Master": {
        "InvitationId": "xxxx", 
        "InvitedAt": "2018-11-02T10:16:21.134Z", 
        "RelationshipStatus": "Enabled", 
        "AccountId": "xxx"
    }
}
```


Discussion:

Resolving patch:
diff --git a/c7n/resources/account.py b/c7n/resources/account.py
--- a/c7n/resources/account.py
+++ b/c7n/resources/account.py
@@ -236,7 +236,7 @@ def get_target(self, resource):
 
         detector = client.get_detector(DetectorId=detector_id)
         detector.pop('ResponseMetadata', None)
-        master = client.get_master_account(DetectorId=detector_id).get('master')
+        master = client.get_master_account(DetectorId=detector_id).get('Master')
         resource[self.annotation] = r = {'Detector': detector, 'Master': master}
         return r
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
invoke lambda action should take into account sdk issues
we default to async, but for folks doing sync invocation, we should guard against multiple invocations by setting config explicitly to have a longer connect timeout.

https://github.com/boto/boto3/issues/883




Discussion:

Resolving patch:
diff --git a/c7n/actions/invoke.py b/c7n/actions/invoke.py
--- a/c7n/actions/invoke.py
+++ b/c7n/actions/invoke.py
@@ -11,6 +11,12 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
+try:
+    from botocore.config import Config
+except ImportError:
+    from c7n.config import Bag as Config  # pragma: no cover
+
 from .core import EventAction
 from c7n import utils
 from c7n.manager import resources
@@ -18,7 +24,7 @@
 
 
 class LambdaInvoke(EventAction):
-    """ Invoke an arbitrary lambda
+    """Invoke an arbitrary lambda
 
     serialized invocation parameters
 
@@ -37,6 +43,13 @@ class LambdaInvoke(EventAction):
 
      - type: invoke-lambda
        function: my-function
+
+    Note if your synchronously invoking the lambda, you may also need
+    to configure the timeout, to avoid multiple invokes. The default
+    is 90s, if the lambda doesn't respond within that time the boto
+    sdk will invoke the lambda again with the same
+    arguments. Alternatively use async: true
+
     """
     schema_alias = True
     schema = {
@@ -47,21 +60,14 @@ class LambdaInvoke(EventAction):
             'function': {'type': 'string'},
             'async': {'type': 'boolean'},
             'qualifier': {'type': 'string'},
-            'batch_size': {'type': 'integer'}
+            'batch_size': {'type': 'integer'},
+            'timeout': {'type': 'integer'},
         }
     }
 
-    def get_permissions(self):
-        if self.data.get('async', True):
-            return ('lambda:InvokeAsync',)
-        return ('lambda:Invoke',)
-
     permissions = ('lambda:InvokeFunction',)
 
     def process(self, resources, event=None):
-        client = utils.local_session(
-            self.manager.session_factory).client('lambda')
-
         params = dict(FunctionName=self.data['function'])
         if self.data.get('qualifier'):
             params['Qualifier'] = self.data['Qualifier']
@@ -69,6 +75,10 @@ def process(self, resources, event=None):
         if self.data.get('async', True):
             params['InvocationType'] = 'Event'
 
+        config = Config(read_timeout=self.data.get('timeout', 90))
+        client = utils.local_session(
+            self.manager.session_factory).client('lambda', config=config)
+
         payload = {
             'version': VERSION,
             'event': event,


Test patch: