internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freqtrade/freqtrade

Issue:
Allow freqai to pull prediction_models from user_data
Currently, only classes present in `freqai/prediction_models` are available for backtesting/trading.
Allowing the user to define a custom model to be used with `--freqaimodel` would allow more flexibility.


Discussion:
that's not really true.

Models placed in `user_data/freqaimodels` will be loading just fine.
It's however missing from the docs ... 

Resolving patch:
diff --git a/freqtrade/configuration/directory_operations.py b/freqtrade/configuration/directory_operations.py
--- a/freqtrade/configuration/directory_operations.py
+++ b/freqtrade/configuration/directory_operations.py
@@ -3,7 +3,8 @@
 from pathlib import Path
 from typing import Optional
 
-from freqtrade.constants import USER_DATA_FILES, Config
+from freqtrade.constants import (USER_DATA_FILES, USERPATH_FREQAIMODELS, USERPATH_HYPEROPTS,
+                                 USERPATH_NOTEBOOKS, USERPATH_STRATEGIES, Config)
 from freqtrade.exceptions import OperationalException
 
 
@@ -49,8 +50,8 @@ def create_userdata_dir(directory: str, create_dir: bool = False) -> Path:
     :param create_dir: Create directory if it does not exist.
     :return: Path object containing the directory
     """
-    sub_dirs = ["backtest_results", "data", "hyperopts", "hyperopt_results", "logs",
-                "notebooks", "plot", "strategies", ]
+    sub_dirs = ["backtest_results", "data", USERPATH_HYPEROPTS, "hyperopt_results", "logs",
+                USERPATH_NOTEBOOKS, "plot", USERPATH_STRATEGIES, USERPATH_FREQAIMODELS]
     folder = Path(directory)
     chown_user_directory(folder)
     if not folder.is_dir():


Test patch:
diff --git a/tests/test_directory_operations.py b/tests/test_directory_operations.py
--- a/tests/test_directory_operations.py
+++ b/tests/test_directory_operations.py
@@ -25,7 +25,7 @@ def test_create_userdata_dir(mocker, default_conf, caplog) -> None:
     md = mocker.patch.object(Path, 'mkdir', MagicMock())
 
     x = create_userdata_dir('/tmp/bar', create_dir=True)
-    assert md.call_count == 9
+    assert md.call_count == 10
     assert md.call_args[1]['parents'] is False
     assert log_has(f'Created user-data directory: {Path("/tmp/bar")}', caplog)
     assert isinstance(x, Path)
Repository: cloud-custodian/cloud-custodian

Issue:
Fix Requirements in Pypi Package 
The requirements in setup.py need to be updated and a new pypi package needs to be delpoyed. When you run `pip install c7n`, the dependencies are not installed.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -3,7 +3,7 @@
 
 setup(
     name="c7n",
-    version='0.8.2',
+    version='0.8.3',
     description="Cloud Custodian - Policy Rules Engine",
     long_description_markdown_filename='README.md',
     classifiers=[
@@ -16,6 +16,6 @@
     entry_points={
         'console_scripts': [
             'custodian = c7n.cli:main']},
-    requires=["boto3", "pyyaml"],
+    install_requires=["boto3", "pyyaml", "jsonschema"],
 )
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Handle Auto Scaling Groups without Launch Configurations
The LaunchConfigurationName is not a required parameter when creating an Auto Scaling Groups so there may be ASGs without LaunchConfigurations. It seems that Custodian cannot handle this:

```
2016-05-25 14:29:13,342: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/src/c7n/policy.py", line 182, in poll
    resources = self.resource_manager.resources()
  File "/src/c7n/resources/asg.py", line 60, in resources
    return self.filter_resources(asgs)
  File "/src/c7n/manager.py", line 66, in filter_resources
    resources = f.process(resources, event)
  File "/src/c7n/resources/asg.py", line 118, in process
    self.initialize(asgs)
  File "/src/c7n/resources/asg.py", line 79, in initialize
    config_names.add(a['LaunchConfigurationName'])
KeyError: 'LaunchConfigurationName'
```



Discussion:
The relevant policy is something like

```
  - name: asg-public-ip-compliance
    resource: asg
    filters:
      - type: launch-config
        key: "AssociatePublicIpAddress"
        value: true
    actions:
      - type: delete
        force: true
```

Hmm, that is probably incorrect. You must specify either a LaunchConfigurationName or Instance ID when creating the ASG. Let me see what happens if I create an ASG with an Instance ID.

In that case the LaunchConfigurationName is the name of the Auto Scaling Group.

aha, thanks! also there's a case i've seen where the launch config gets deleted separate then the asg and leaves an orphan asg.

i'm wondering if this is just a transient issue around asg teardown


Resolving patch:
diff --git a/c7n/resources/asg.py b/c7n/resources/asg.py
--- a/c7n/resources/asg.py
+++ b/c7n/resources/asg.py
@@ -60,9 +60,18 @@ class LaunchConfigBase(object):
     def initialize(self, asgs):
         """Get launch configs for the set of asgs"""
         config_names = set()
+        skip = []
+
         for a in asgs:
+            # Per https://github.com/capitalone/cloud-custodian/issues/143
+            if 'LaunchConfigurationName' not in a:
+                skip.append(a)
+                continue
             config_names.add(a['LaunchConfigurationName'])
 
+        for a in skip:
+            asgs.remove(a)
+
         session = local_session(self.manager.session_factory)
         client = session.client('autoscaling')
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
key error on deleting ebs snapshots
This error can occur when deleting ebs snapshots

```
ERROR - custodian.output - Error while executing policy Traceback (most recent call last):
File "/usr/local/custodian/lib/python2.7/site-packages/c7n/policy.py", line 207, in poll results = a.process(resources)
File "/usr/local/custodian/lib/python2.7/site-packages/c7n/resources/ebs.py", line 70, in process snaps.add(dev['Ebs']['SnapshotId'])
KeyError: 'SnapshotId'`
```

I believe this is happening when there are non-EBS backed AMIs and a check needs to be made for the presence of the `SnapshotId` key



Discussion:
I can take a look at this.

Here's an example of an **Image** that causes the problem:

``` json
{
    "Architecture": "x86_64",
    "BlockDeviceMappings": [
        {
            "DeviceName": "/dev/sda1",
            "Ebs": {
                "DeleteOnTermination": true,
                "Encrypted": false,
                "SnapshotId": "snap-12345678",
                "VolumeSize": 20,
                "VolumeType": "gp2"
            }
        },
        {
            "DeviceName": "/dev/sdb",
            "Ebs": {
                "DeleteOnTermination": true,
                "Encrypted": true,
                "VolumeSize": 20,
                "VolumeType": "gp2"
            }
        },
        {
            "DeviceName": "/dev/sdc",
            "Ebs": {
                "DeleteOnTermination": true,
                "Encrypted": true,
                "VolumeSize": 200,
                "VolumeType": "gp2"
            }
        }
    ],
    "CreationDate": "2016-05-04T15:38:50.000Z",
    "Description": "Blah blah blah",
    "Hypervisor": "xen",
    "ImageId": "ami-12345678",
    "ImageType": "machine",
    "Public": false,
    "RootDeviceName": "/dev/sda1",
    "RootDeviceType": "ebs",
    "SriovNetSupport": "simple",
    "State": "available",
    "VirtualizationType": "hvm"
}
```


Resolving patch:
diff --git a/c7n/resources/ebs.py b/c7n/resources/ebs.py
--- a/c7n/resources/ebs.py
+++ b/c7n/resources/ebs.py
@@ -66,7 +66,7 @@ def process(self, snapshots):
             c = local_session(self.manager.session_factory).client('ec2')
             for i in c.describe_images(Owners=['self'])['Images']:
                 for dev in i.get('BlockDeviceMappings'):
-                    if 'Ebs' in dev:
+                    if 'Ebs' in dev and 'SnapshotId' in dev['Ebs']:
                         snaps.add(dev['Ebs']['SnapshotId'])
         log.info("Deleting %d snapshots", len(snapshots))
         with self.executor_factory(max_workers=3) as w:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Occasional 'PolicyNotFound' errors when setting ELB security policy
Occasionally we see `PolicyNotFound` not found errors when setting ELB security policy:

```
2016-06-13 15:56:51,525: custodian.filters:ERROR Exception processing elb policies 
 An error occurred (PolicyNotFound) when calling the DescribeLoadBalancerPolicies operation: There is no policy with name ELBSecurityPolicy-2015-05 for load balancer xyz-elb
```

When I look in the AWS console, ELB **xyz-elb** does not exist.
So when the list of ELBs is built the ELB exists but when policies are being applied the ELB no longer exists.



Discussion:
I suggest just ignoring this error.


Resolving patch:
diff --git a/c7n/resources/elb.py b/c7n/resources/elb.py
--- a/c7n/resources/elb.py
+++ b/c7n/resources/elb.py
@@ -306,7 +306,7 @@ def process_elb_policy_set(self, elb_policy_set):
                     LoadBalancerName=elb_name,
                     PolicyNames=policy_names)['PolicyDescriptions']
             except ClientError as e:
-                if e.response['Error']['Code'] == "LoadBalancerNotFound":
+                if e.response['Error']['Code'] in ['LoadBalancerNotFound', 'PolicyNotFound']:
                     continue
                 raise
             active_lb_policies = []


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
errlog context gathering can miss original



Discussion:

Resolving patch:
diff --git a/c7n/ufuncs/logsub.py b/c7n/ufuncs/logsub.py
--- a/c7n/ufuncs/logsub.py
+++ b/c7n/ufuncs/logsub.py
@@ -14,6 +14,7 @@
 """Ops feedback via log subscription
 """
 import boto3
+from botocore.vendored import requests
 
 import base64
 from datetime import datetime
@@ -70,6 +71,11 @@ def process_log_event(event, context):
         "Log Contents",
         ""]
 
+    # We may get things delivered from log sub that are not in log events
+    for evt in data['logEvents']:
+        if evt not in events:
+            events.append(evt)
+
     for evt in events:
         message.append(message_event(evt))
         message.append("")


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
support cloud watch events with account pseduo resource
use cases, re-enable cloud-config, cloud trail realtime if attempt is made to disable.



Discussion:

Resolving patch:
diff --git a/c7n/resources/account.py b/c7n/resources/account.py
--- a/c7n/resources/account.py
+++ b/c7n/resources/account.py
@@ -42,6 +42,14 @@ def resources(self):
               'account_name': client.list_account_aliases(
               ).get('AccountAliases', ('',))[0]}])
 
+    def get_resources(self, resource_ids):
+        session = local_session(self.session_factory)
+        client = session.client('iam')
+        return [
+            {'account_id': get_account_id(session),
+             'account_name': client.list_account_aliases().get(
+                 'AccountAliases', ('',))[0]}]
+
 
 @filters.register('check-cloudtrail')
 class CloudTrailEnabled(Filter):


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Monitoring endpoint does not exist
When running the s3stats in custodian tools, it attempts to grab the bucket region: 

`bucket_region = s3.get_bucket_location(
                Bucket=b['Name'])['LocationConstraint']` 

However, if the bucket is in Ireland, the value returned is "EU". So it would fail when it attempted to reach monitoring.EU.amazonaws.com since that domain doesn't exist.
I added a quick fix within s3stat.py but should be possible to just map the monitoring endpoint with the existing boto data file or another work around.



Discussion:
hmm.. looks like an s3 inconsistency that needs client handling, for all other values of LocationConstraint this maps 1 to 1 to region per 
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html

we'll need to add this to resources/s3.py as well which uses the same location constraint to region logic.


Resolving patch:
diff --git a/c7n/resources/account.py b/c7n/resources/account.py
--- a/c7n/resources/account.py
+++ b/c7n/resources/account.py
@@ -183,7 +183,7 @@ class IAMSummary(ValueFilter):
             "UserPolicySizeQuota": 2048
         }
 
-q    """
+    """
     schema = type_schema('iam-summary', rinherit=ValueFilter.schema)
 
     def process(self, resources, event=None):
diff --git a/c7n/resources/s3.py b/c7n/resources/s3.py
--- a/c7n/resources/s3.py
+++ b/c7n/resources/s3.py
@@ -157,6 +157,9 @@ def assemble_bucket(item):
         # As soon as we learn location (which generally works)
         if k == 'Location' and v is not None:
             b_location = v.get('LocationConstraint')
+            # Location == region for all cases but EU per https://goo.gl/iXdpnl
+            if b_location == 'EU':
+                b_location = "eu-west-1"
             if v and v != c_location:
                 c = s.client('s3', region_name=b_location)
             elif c_location != location:
diff --git a/tools/s3stat.py b/tools/s3stat.py
--- a/tools/s3stat.py
+++ b/tools/s3stat.py
@@ -86,6 +86,11 @@ def main():
         try:
             bucket_region = s3.get_bucket_location(
                 Bucket=b['Name'])['LocationConstraint']
+            if bucket_region is None:
+                bucket_region = "us-east-1"
+            # special case per https://goo.gl/iXdpnl
+            elif bucket_region == "EU":
+                bucket_region = "eu-west-1"
         except ClientError as e:
             # We don't have permission to the bucket, try us-east-1
             bucket_region = "us-east-1"


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Temp File Issue
I'm getting a ton binaries showing up in the `/tmp` space. please help! k thx bye



Discussion:
sigh windows support, time to yank. we used to do ops on files which windows didn't like, got merged in #118 to be reverted. Primarily its around lambda uploads.

> sigh windows support, time to yank. we used to do ops on files which windows didn't like

Does this mean we’re no longer opening the file a second time and those changes should just be reversed, or is there more to it than that?


Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -86,7 +86,7 @@ def filter_files(self, files):
 
     def create(self):
         assert not self._temp_archive_file, "Archive already created"
-        self._temp_archive_file = tempfile.NamedTemporaryFile(delete=False)
+        self._temp_archive_file = tempfile.NamedTemporaryFile()
         self._zip_file = zipfile.ZipFile(
             self._temp_archive_file, mode='w',
             compression=zipfile.ZIP_DEFLATED)
@@ -144,8 +144,7 @@ def close(self):
         return self
 
     def remove(self):
-        # dispose of the temp file for garbag collection
-        os.remove(self._temp_archive_file.name)
+        # dispose of the temp file for garbage collection
         if self._temp_archive_file:
             self._temp_archive_file = None
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Security Group filter "default-vpc" does not function correctly.
Using the following policy:

```
policies:
  - name: default-sg-allows-all-traffic
    description: |
      Find whether the default security group allows all traffic.
    resource: security-group
    filters:
       - type: default-vpc
```

Comes up with no results, even when I have confirmed that my account has a default vpc and a default security group associated with it.



Discussion:

Resolving patch:
diff --git a/c7n/filters/vpc.py b/c7n/filters/vpc.py
--- a/c7n/filters/vpc.py
+++ b/c7n/filters/vpc.py
@@ -52,8 +52,6 @@ def match(self, vpc_id):
             vpcs = [v['VpcId'] for v
                     in client.describe_vpcs(VpcIds=[vpc_id])['Vpcs']
                     if v['IsDefault']]
-            if not vpcs:
-                self.default_vpc = ""
-            else:
+            if vpcs:
                 self.default_vpc = vpcs.pop()
         return vpc_id == self.default_vpc and True or False


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
mu zip assembler umask fixes
We run into issues a few times where a restrictive umask on a server can cause issues with lambda deployments. Effectively to run correctly we need to ensure read only permission for all, so at least a 444 permission set on all added files.


Discussion:

Resolving patch:
diff --git a/c7n/mu.py b/c7n/mu.py
--- a/c7n/mu.py
+++ b/c7n/mu.py
@@ -19,6 +19,7 @@
 
 import abc
 import base64
+from datetime import datetime
 import inspect
 import fnmatch
 import hashlib
@@ -105,9 +106,9 @@ def create(self):
                     self.src_filter(root, dirs, files)
                 files = self.filter_files(files)
                 for f in files:
-                    self._zip_file.write(
-                        os.path.join(root, f),
-                        os.path.join(arc_prefix, f))
+                    f_path = os.path.join(root, f)
+                    dest_path = os.path.join(arc_prefix, f)
+                    self.add_file(f_path, dest_path)
 
         # Library Source
         venv_lib_path = os.path.join(
@@ -119,18 +120,22 @@ def create(self):
             arc_prefix = os.path.relpath(root, venv_lib_path)
             files = self.filter_files(files)
             for f in files:
-                self._zip_file.write(
-                    os.path.join(root, f),
-                    os.path.join(arc_prefix, f))
+                f_path = os.path.join(root, f)
+                dest_path = os.path.join(arc_prefix, f)
+                self.add_file(f_path, dest_path)
 
     def add_file(self, src, dest):
-        self._zip_file.write(src, dest)
+        info = zipfile.ZipInfo(dest)
+        with open(src, 'rb') as fp:
+            contents = fp.read()
+            self.add_contents(info, contents)
 
     def add_contents(self, dest, contents):
         if not isinstance(dest, zipfile.ZipInfo):
             dest = zinfo(dest)
         # see zinfo function for some caveats
         assert not self._closed, "Archive closed"
+        dest.external_attr = 0444 << 16L
         self._zip_file.writestr(dest, contents)
 
     def close(self):
@@ -603,7 +608,6 @@ def zinfo(fname):
     """
     info = zipfile.ZipInfo(fname)
     # Grant other users permissions to read
-    info.external_attr = 0o644 << 16
     info.compress_type = zipfile.ZIP_DEFLATED
     return info
 


Test patch:
diff --git a/tests/test_mu.py b/tests/test_mu.py
--- a/tests/test_mu.py
+++ b/tests/test_mu.py
@@ -336,3 +336,15 @@ def test_archive_skip(self):
                       'c7n/resources/s3.pyc',
                       'boto3/__init__.py']:
                 self.assertFalse(i in fileset)
+
+    def test_archive_permissions(self):
+        # files should all be readable
+        self.archive = custodian_archive("*.pyc")
+        self.archive.create()
+        self.addCleanup(self.archive.remove)
+        self.archive.close()
+        readable = 0444 << 16L
+        with open(self.archive.path) as fh:
+            reader = zipfile.ZipFile(fh, mode='r')
+            for i in reader.infolist():
+                self.assertGreaterEqual(i.external_attr, readable)
Repository: cloud-custodian/cloud-custodian

Issue:
related resource filter can depend on a side-effect
Given this policy based largely on the examples from #541 (and the code that fixed it), an exception will be triggered.

```
policies:
  - name: related-rds-test
    description: |
      If databse is using default security group, adjust the retention.
    resource: rds
    filters:
      - type: security-group
        match-resource: true
        key: "GroupName"
        value: "default"
    actions:
      - type: retention
        days: 14
```

Specifically, this will happen *any* time `match-resource` is `True` and `key` doesn't start with “tag:”.

```
Traceback (most recent call last):
  File "cloud-custodian/c7n/policy.py", line 167, in run
    resources = self.policy.resource_manager.resources()
  File "cloud-custodian/c7n/query.py", line 152, in resources
    return self.filter_resources(resources)
  File "cloud-custodian/c7n/manager.py", line 63, in filter_resources
    resources = f.process(resources, event)
  File "cloud-custodian/c7n/filters/related.py", line 96, in process
    return [r for r in resources if self.process_resource(r, related)]
  File "cloud-custodian/c7n/filters/related.py", line 70, in process_resource
    self.data['value'] = self.get_resource_value(
  File "cloud-custodian/c7n/filters/core.py", line 274, in get_resource_value
    self.expr = jmespath.compile(self.k)
AttributeError: 'SecurityGroupFilter' object has no attribute 'k'
```

The policy passes validation.

The problem is

https://github.com/capitalone/cloud-custodian/blob/master/c7n/filters/related.py#L69

`self.get_resource_value()` can depend on `self.k` in some cases, but that’s only set up as a side-effect of calling `self.match()`.

Just calling `self.match(None)` first will set up the properties then bail, which seems to work, but also seems like a hack.

Moving lines 68-70 down below the `for` loop also seems to work, but I'm unsure of the impact. (The calls to `self.match()` could be influenced by the change to `self.data['value']`.)



Discussion:

Resolving patch:
diff --git a/c7n/filters/core.py b/c7n/filters/core.py
--- a/c7n/filters/core.py
+++ b/c7n/filters/core.py
@@ -271,7 +271,7 @@ def get_resource_value(self, k, i):
         elif self.expr:
             r = self.expr.search(i)
         else:
-            self.expr = jmespath.compile(self.k)
+            self.expr = jmespath.compile(k)
             r = self.expr.search(i)
         return r
 


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
ELB set-ssl-listener-policy fails after first run when same policy name used
The ELB set-ssl-listener-policy command runs successfully the first time it's run against an ELB with a given SSL policy name, but after that it fails when the same policy name is used.  

The use case for needing multiple runs with the same policy name is that if someone updates the ELB policy to something out of compliance after Custodian cleaned it up, the subsequent daily/hourly run will fail to revert those changes unless a unique policy name is provided with every call.

/opinion
If you want to keep it simple with a single set-ssl-listener-policy command instead of create/delete/set commands, if a duplicate policy name is passed either ignore the attributes and issue a set command or do a deletion of the old policy and create and set the new one with the new attributes.




Discussion:
afaicr, the same policy updates was handled a few months ago, are you seeing traceback, errors or just failure to effect the goal.
Thank you for the response Kapil.  I deleted Custodian and Python and did a full reinstall of both to make sure I wasn't using any old code or utilities, and I'm still getting the error "Policy Name [SSLPolicyTest] already exists.

When I look at elb.py in c7n/resources/ I see the following code which makes me think that this is the section that's blowing up.

```python
try:
client.create_load_balancer_policy(
          LoadBalancerName=lb_name,
          PolicyName=policy_name,
          PolicyTypeName='SSLNegotiationPolicyType',
          PolicyAttributes=policy_attributes)
except ClientError as e:
          if e.response['Error']['Code'] in (
                      'DuplicatePolicyName', 'DuplicationPolicyNameException'):
              raise
```
yup, definitely a bug, i was looking for the traceback, but i got a copy.

Resolving patch:
diff --git a/c7n/resources/elb.py b/c7n/resources/elb.py
--- a/c7n/resources/elb.py
+++ b/c7n/resources/elb.py
@@ -275,7 +275,7 @@ def process_elb(self, elb):
                 PolicyTypeName='SSLNegotiationPolicyType',
                 PolicyAttributes=policy_attributes)
         except ClientError as e:
-            if e.response['Error']['Code'] in (
+            if e.response['Error']['Code'] not in (
                     'DuplicatePolicyName', 'DuplicationPolicyNameException'):
                 raise
 


Test patch: