internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: cloud-custodian/cloud-custodian

Issue:
releng - add tecentcloud to tools/dev/dockerpkg and the docker images
### Describe the feature

The dockerfiles dont install the tencentcloud provider currently

### Extra information or context

_No response_


Discussion:

Resolving patch:
diff --git a/tools/dev/dockerpkg.py b/tools/dev/dockerpkg.py
--- a/tools/dev/dockerpkg.py
+++ b/tools/dev/dockerpkg.py
@@ -59,9 +59,11 @@
 RUN rm -R tools/c7n_kube/tests
 ADD tools/c7n_openstack /src/tools/c7n_openstack
 RUN rm -R tools/c7n_openstack/tests
+ADD tools/c7n_tencentcloud /src/tools/c7n_tencentcloud
+RUN rm -R tools/c7n_tencentcloud/tests
 
 # Install requested providers
-ARG providers="gcp kube openstack azure"
+ARG providers="gcp kube openstack tencentcloud azure"
 RUN . /usr/local/bin/activate && \\
     for pkg in $providers; do cd tools/c7n_$pkg && \\
     poetry install && cd ../../; done


Test patch:
diff --git a/tests/test_docker.py b/tests/test_docker.py
--- a/tests/test_docker.py
+++ b/tests/test_docker.py
@@ -153,7 +153,7 @@ def test_image_metadata(image_name):
 def test_cli_providers_available():
     providers = os.environ.get("CUSTODIAN_PROVIDERS", None)
     if providers is None:
-        providers = {"aws", "azure", "gcp", "k8s", "openstack"}
+        providers = {"aws", "azure", "gcp", "k8s", "openstack", "tencentcloud"}
     elif providers == "":
         providers = {"aws"}
     else:
Repository: cloud-custodian/cloud-custodian

Issue:
Use case-insensitive checks for allowed conditions in `cross-account` filters
### Describe the feature

Today the `cross-account` filter has a `whitelist_conditions` parameter that skips matching resources if certain condition keys exist (by default, this list includes `aws:username` and `aws:userid`).

When evaluating condition keys against the allow list, we [lowercase resource data as part of a normalization pass](https://github.com/cloud-custodian/cloud-custodian/blob/772f5461ac851a1b4ad4b1e8e82068d50c3270bc/c7n/filters/iamaccess.py#L206). But we don't also lowercase the `whitelist_conditions` items in the policy. Normalizing both sides (possibly as part of the property defined [here](https://github.com/cloud-custodian/cloud-custodian/blob/772f5461ac851a1b4ad4b1e8e82068d50c3270bc/c7n/filters/iamaccess.py#L73-L75)) would avoid some usability bumps when matching on mixed-case condition keys.

### Extra information or context

While we're at it, it's probably worth adding aliases like `allow*` for the existing `whitelist*` parameters (opening the door for deprecating `whitelist*` going forward). _This is possibly/probably best left as a separate work item, but the idea does naturally come up when you find yourself typing whitelist so many times into an issue description!_


Discussion:
that makes sense, alternatively doc string to specify conditions on lower, but auto convert does make it non dependent on user/author so net win.
Hello, can I work on this issue?
Sure @Rajath-55 - go for it! A couple resources that might be helpful along the way:

- [Contributor Guide](https://cloudcustodian.io/docs/contribute.html)
- [Community Slack](https://communityinviter.com/apps/cloud-custodian/c7n-chat)
  - The #development channel would be a good place to ask questions if you hit any bumps getting set up or making changes

Resolving patch:
diff --git a/c7n/filters/iamaccess.py b/c7n/filters/iamaccess.py
--- a/c7n/filters/iamaccess.py
+++ b/c7n/filters/iamaccess.py
@@ -72,7 +72,7 @@ def check_actions(self):
 
     @property
     def whitelist_conditions(self):
-        return self.checker_config.get('whitelist_conditions', ())
+        return set(v.lower() for v in self.checker_config.get('whitelist_conditions', ()))
 
     @property
     def allowed_vpce(self):


Test patch:
diff --git a/tests/test_iam.py b/tests/test_iam.py
--- a/tests/test_iam.py
+++ b/tests/test_iam.py
@@ -1950,6 +1950,7 @@ def test_sns_cross_account_endpoint_condition(self):
                     {
                         "type": "cross-account",
                         "whitelist_endpoints": ["@whitelist.com"],
+                        "whitelist_conditions": ["aws:UserName"]
                     },
                 ],
             },
Repository: cloud-custodian/cloud-custodian

Issue:
route 53 resources are all global
### Describe the bug

only domains are listed as global but all resources in route53 should be considered global

### What did you expect to happen?

only collect r53 resources in us-east-1

### Cloud Provider

Amazon Web Services (AWS)

### Cloud Custodian version and dependency information

_No response_

### Policy

_No response_

### Relevant log/traceback output

_No response_

### Extra information or context

_No response_


Discussion:

Resolving patch:
diff --git a/c7n/resources/route53.py b/c7n/resources/route53.py
--- a/c7n/resources/route53.py
+++ b/c7n/resources/route53.py
@@ -104,6 +104,7 @@ class resource_type(TypeInfo):
         name = id = 'Id'
         universal_taggable = True
         cfn_type = 'AWS::Route53::HealthCheck'
+        global_resource = True
 
 
 @resources.register('rrset')
@@ -116,6 +117,7 @@ class resource_type(TypeInfo):
         enum_spec = ('list_resource_record_sets', 'ResourceRecordSets', None)
         name = id = 'Name'
         cfn_type = 'AWS::Route53::RecordSet'
+        global_resource = True
 
 
 @resources.register('r53domain')


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
releng - pin linter
we've had ruff updates break ci twice over the last month, let's install pinned versions and deal with the upgrades as part of our regular dependency upgrade cycles.


Discussion:

Resolving patch:
diff --git a/tools/dev/lint_version.py b/tools/dev/lint_version.py
new file mode 100644
--- /dev/null
+++ b/tools/dev/lint_version.py
@@ -0,0 +1,28 @@
+"""
+extract from poetry lock on a package version.
+
+used in ci to ensure we used locked tool versions, but
+also can install fast.
+"""
+
+# only use stdlib
+
+import tomllib
+import argparse
+from pathlib import Path
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('package')
+    parser.add_argument('lock_file')
+    args = parser.parse_args()
+    lock_data = tomllib.loads(Path(args.lock_file).read_text())
+    for pkg in lock_data['package']:
+        if pkg['name'] == args.package:
+            print(pkg['version'], end='')
+
+
+if __name__ == '__main__':
+    main()
+


Test patch:
diff --git a/tests/test_tags.py b/tests/test_tags.py
--- a/tests/test_tags.py
+++ b/tests/test_tags.py
@@ -47,6 +47,8 @@ def test_tag_action_filter_call(test, tag_action_filter_call):
     test.assertEqual(len(resources), 1)
 
     stopped_ec2_instance_id = tag_action_filter_call['aws_instance.past_stop.id']
+    if test.recording:
+       time.sleep(0.25)
     ec2 = session_factory().resource('ec2')
     instance = ec2.Instance(stopped_ec2_instance_id)
     test.assertEqual(instance.state['Name'], 'stopping')
Repository: cloud-custodian/cloud-custodian

Issue:
RDS instance delete not working
### Describe the bug

I'm not sure after 0.9.30 only it happened or not.  Earlier we had the working process of RDS instance deletion feature, but currently it seems not working.

### What did you expect to happen?

When I checked in general @ajkerrigan replied as follows 
#8713 aws - rds - delete - handle  InvalidParameterValue exception
Not sure this impact or anything I missed out.

### Cloud Provider

Amazon Web Services (AWS)

### Cloud Custodian version and dependency information

```shell
0.9.30
```


### Policy

```shell
- name: rds-delete
    resource: aws.rds
    actions:
      - type: modify-db
        update:
          - property: 'DeletionProtection'
            value: false
          - property: 'PubliclyAccessible'
            value: false
        immediate: true
      - type: delete
        skip-snapshot: true
```


### Relevant log/traceback output

```shell
2023-08-28 14:32:17,822: c7n_org:DEBUG Running policy:rds-delete account:acct_id region:us-east-1
2023-08-28 14:32:20,408: custodian.actions:WARNING delete implicitly filtered 0 of 1 resources key:DBClusterIdentifier on None
2023-08-28 14:32:20,420: c7n_org:INFO Ran account:acct_id region:us-east-1 policy:rds-delete matched:1 time:2.60
2023-08-28 14:32:20,446: c7n_org:INFO Policy resource counts Counter({'rds-delete': 1})
```


### Extra information or context

_No response_


Discussion:
@kapilt  @ajkerrigan I tested and confirmed that 0.9.29 works for rds deleting.
thanks for filing the report, treating this as a release blocker for the next release.

Resolving patch:
diff --git a/c7n/resources/rds.py b/c7n/resources/rds.py
--- a/c7n/resources/rds.py
+++ b/c7n/resources/rds.py
@@ -565,8 +565,7 @@ def validate(self):
     def process(self, dbs):
         skip = self.data.get('skip-snapshot', False)
         # Can't delete an instance in an aurora cluster, use a policy on the cluster
-        dbs = self.filter_resources(dbs, "DBClusterIdentifier", [None])
-
+        dbs = [r for r in dbs if not r.get('DBClusterIdentifier')]
         # Concurrency feels like overkill here.
         client = local_session(self.manager.session_factory).client('rds')
         for db in dbs:


Test patch:
Repository: cloud-custodian/cloud-custodian

Issue:
Handle aws-iso region
### Describe the feature

Currently, c7n is hard coded to only handle 'aws', 'aws-us-gov', and 'aws-cn' partitions. My team works in the 'aws-iso' partition. To support working in the 'aws-iso' region, a small change to the [list of hardcoded regions](https://github.com/cloud-custodian/cloud-custodian/blob/e8859d5569d68ebe19ee3cc710149f76e146ac92/c7n/resources/aws.py#L861C39-L861C39) would allow it to work for our usecase.

We have confirmed (by editing the downloaded code) that this change would work.

### Extra information or context

_No response_


Discussion:

Resolving patch:
diff --git a/c7n/resources/aws.py b/c7n/resources/aws.py
--- a/c7n/resources/aws.py
+++ b/c7n/resources/aws.py
@@ -855,10 +855,10 @@ def get_service_region_map(regions, resource_types, provider='aws'):
     resource_service_map = {
         r: clouds[provider].resources.get(r).resource_type.service
         for r in normalized_types if r != 'account'}
-    # support for govcloud and china, we only utilize these regions if they
+    # support for govcloud, china, and iso. We only utilize these regions if they
     # are explicitly passed in on the cli.
     partition_regions = {}
-    for p in ('aws-cn', 'aws-us-gov'):
+    for p in ('aws-cn', 'aws-us-gov', 'aws-iso'):
         for r in session.get_available_regions('s3', partition_name=p):
             partition_regions[r] = p
 


Test patch:
Repository: Kaggle/docker-python

Issue:
NameError: name 'io' is not defined
## 🐛 Bug

I am trying to run my scripts on GPU notebook, and I keep getting the following error.

```shell
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1172, in init
    getcaller()
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 846, in getcaller
    src, line, func, stack = logger.findCaller(stack_info=True)
  File "/root/.local/lib/python3.10/site-packages/log.py", line 42, in findCaller
    sio = io.StringIO()
NameError: name 'io' is not defined
```

In addition, I found that there is no import `io` package in [this](https://github.com/Kaggle/docker-python/blob/main/patches/log.py) code.

### To Reproduce 

### Expected behavior

### Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Please add more context on how to reproduce the problem.

Resolving patch:
diff --git a/patches/log.py b/patches/log.py
--- a/patches/log.py
+++ b/patches/log.py
@@ -1,3 +1,4 @@
+import io
 import logging
 import os
 
@@ -129,4 +130,4 @@ def _static_init():
         logging.basicConfig(level=logging.INFO, handlers=[handler])
         Log._initialized = True
 
-Log._static_init()
\ No newline at end of file
+Log._static_init()


Test patch:
Repository: privacyidea/privacyidea

Issue:
LDAP/TLS: Use system cert store on Debian/Ubuntu and RedHat/CentOS by default
That patch seems to work:
```
--- privacyidea/lib/resolvers/LDAPIdResolver.py.orig    2017-03-05 17:50:35.132519060 +0100
+++ privacyidea/lib/resolvers/LDAPIdResolver.py 2017-03-05 18:04:42.268135183 +0100
@@ -60,6 +60,8 @@
 from ldap3 import Server, Tls, Connection
 import ssl
 
+import os.path
+
 import traceback
 
 import hashlib
@@ -87,7 +89,14 @@
 MS_AD_MULTIPLYER = 10 ** 7
 MS_AD_START = datetime.datetime(1601, 1, 1)
 
-DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
+if os.path.isfile("/etc/privacyidea/ldap-ca.crt"):
+    DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
+elif os.path.isfile("/etc/ssl/certs/ca-certificates.crt"):
+    DEFAULT_CA_FILE = "/etc/ssl/certs/ca-certificates.crt"
+elif os.path.isfile("/etc/ssl/certs/ca-bundle.crt"):
+    DEFAULT_CA_FILE = "/etc/ssl/certs/ca-bundle.crt"
+else:
+    DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
 
 
 def get_ad_timestamp_now():
```

If you like the approach I can update the documentation and prepare a pull request.


Discussion:
Basically this is nice, but I am not sure about it.
Usually privacyIDEA follows the explicit principle. And If we use a default value it is the value which is denoted as placeholder in the input field. This would not work if searching for different files.

Well, the first file you are searching is in fact /etc/privacyidea/Ldap-ca.crt, which would be the placeholder.

I would appreciate a pull request. Which would make #639 obsolete I think.

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -60,6 +60,8 @@
 from ldap3 import Server, Tls, Connection
 import ssl
 
+import os.path
+
 import traceback
 
 import hashlib
@@ -87,7 +89,14 @@
 MS_AD_MULTIPLYER = 10 ** 7
 MS_AD_START = datetime.datetime(1601, 1, 1)
 
-DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
+if os.path.isfile("/etc/privacyidea/ldap-ca.crt"):
+    DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
+elif os.path.isfile("/etc/ssl/certs/ca-certificates.crt"):
+    DEFAULT_CA_FILE = "/etc/ssl/certs/ca-certificates.crt"
+elif os.path.isfile("/etc/ssl/certs/ca-bundle.crt"):
+    DEFAULT_CA_FILE = "/etc/ssl/certs/ca-bundle.crt"
+else:
+    DEFAULT_CA_FILE = "/etc/privacyidea/ldap-ca.crt"
 
 
 def get_ad_timestamp_now():


Test patch:
Repository: privacyidea/privacyidea

Issue:
LDAP resolver does not respect some boolean configuration options
Please think about and cover the following questions:

# Versions

privacyIDEA: git revision 1888c8a

Installation method: virtualenv

## more details: 

OS: Arch Linux

Webserver: Flask integrated webserver 

Tokendatabase: sqlite3

# What did you try to do?

Create LDAP resolver, *uncheck* STARTTLS check box and *uncheck* "No anonymous referral chasing"

# What outcome did you expect?

STARTTLS should not be used. The configuration is written to the database correctly:
<pre>
$ sqlite3 data.sqlite "SELECT * FROM resolverconfig WHERE Key='START_TLS' OR Key='NOREFERRALS'"
6|2|START_TLS|0|bool|
16|2|NOREFERRALS|0|bool|
</pre>

# What outcome did you experience?

According to the logfile, STARTTLS is established regardless:

<pre>$ grep tls privacyidea.log                                                
[2017-03-16 14:48:55,682][22582][140512990626880][DEBUG][privacyidea.lib.resolvers.LDAPIdResolver:1042] Doing start_tls
[2017-03-16 14:48:56,417][22582][140512990626880][DEBUG][privacyidea.lib.resolvers.LDAPIdResolver:1042] Doing start_tls
[2017-03-16 14:48:57,123][22582][140512990626880][DEBUG][privacyidea.lib.resolvers.LDAPIdResolver:1042] Doing start_tls
[2017-03-16 14:48:57,570][22582][140512990626880][DEBUG][privacyidea.lib.resolvers.LDAPIdResolver:1042] Doing start_tls
[2017-03-16 14:49:11,092][22582][140512990626880][DEBUG][privacyidea.lib.resolvers.LDAPIdResolver:1042] Doing start_tls
</pre>

By setting a breakpoint at ``ldap3.core.connection.Connection.__init__``, we can see that ``auto_referrals=False`` is passed even though we did enable referral chasing.


Discussion:

Resolving patch:
diff --git a/privacyidea/lib/resolvers/LDAPIdResolver.py b/privacyidea/lib/resolvers/LDAPIdResolver.py
--- a/privacyidea/lib/resolvers/LDAPIdResolver.py
+++ b/privacyidea/lib/resolvers/LDAPIdResolver.py
@@ -589,8 +589,8 @@ def loadConfig(self, config):
         self.userinfo["username"] = self.loginname_attribute
         self.map = yaml.safe_load(userinfo)
         self.uidtype = config.get("UIDTYPE", "DN")
-        self.noreferrals = config.get("NOREFERRALS", False)
-        self.start_tls = config.get("START_TLS", False)
+        self.noreferrals = is_true(config.get("NOREFERRALS", False))
+        self.start_tls = is_true(config.get("START_TLS", False))
         self._editable = config.get("EDITABLE", False)
         self.scope = config.get("SCOPE") or ldap3.SUBTREE
         self.resolverId = self.uri


Test patch:
Repository: privacyidea/privacyidea

Issue:
Latency for Flask Endpoints
Please think about and cover the following questions:

# Versions
privacyIDEA: 2.18
Installation method:
- from source / github ?
## more details: 
OS: Amazon Linux
Webserver: uwsgi/nginx
Tokendatabase: PostgreSQL


Hello Cornelius,

We are noticing latency when there is a large amount of tokens in the database. I believe it is a long runtime when getting the tokentype for auditing events. Some events have the serial number as **, and the logic to get the tokentype runs too long.

https://github.com/privacyidea/privacyidea/blob/branch-2.18/privacyidea/api/before_after.py#L127

The following function call contains two timely for-loops in the event of large amounts of tokens in the database.
- Query all tokens in the database to create token instances
  - https://github.com/privacyidea/privacyidea/blob/branch-2.18/privacyidea/lib/token.py#L327
- Getting the tokentype of the last token object
  - https://github.com/privacyidea/privacyidea/blob/branch-2.18/privacyidea/lib/token.py#L448

The following line create a SQL Query of the Token database, and loops through all the serial numbers create token objects. Once the token objects are created, another loop initiates to go through all the tokens and get the last objects tokentype.

Here are some results of time taken to load  I did.
3,000  tokens => 8-9s
5,000  tokens => 12-14s
10,000 tokens => 30-34s

I replaced the function call with a string for to observe the performance and loading the token/list took less then 2s for 10 tokens.
`tokentype = 'hotp'`

I am looking forward to your input.

Thank you.


Discussion:
A cleaner fix I have for this issue is change the if-condition

`if serial and "**" not in serial:`
This sounds good to me. Thanks a lot for pointing this out. The "**" comes from the audit log or searches for tokens.

    if serial and not (serial.startswith("*") and serial.endswith("*"))

would make sense. But a search like "\*A\*" could in fact return one single token.

So I think your suggestion sounds very good. Would you commit a PR?
Alright, so we agree that the change will be to
`if serial and "**" not in serial`

I will have a PR soon.

Thank you.

Resolving patch:
diff --git a/privacyidea/api/before_after.py b/privacyidea/api/before_after.py
--- a/privacyidea/api/before_after.py
+++ b/privacyidea/api/before_after.py
@@ -123,7 +123,7 @@ def before_request():
                          request.host
     # Already get some typical parameters to log
     serial = getParam(request.all_data, "serial")
-    if serial:
+    if serial and "**" not in serial:
         tokentype = get_token_type(serial)
     else:
         tokentype = None


Test patch:
Repository: privacyidea/privacyidea

Issue:
can not filter audit log for date with postgresql
Filtering audit log for date causes an error with postgres database (works fine with mysql)

When i try to insert any character or number on date->filter in audit log

(psycopg2.ProgrammingError) operator does not exist: timestamp without time zone ~~ unknown LINE 4: WHERE pidea_audit.date LIKE ‘%21%’) AS anon_1 ^ HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts. [SQL: ‘SELECT count(*) AS count_1 \nFROM (SELECT pidea_audit.id AS pidea_audit_id \nFROM pidea_audit \nWHERE pidea_audit.date LIKE %(date_1)s) AS anon_1’] [parameters: {‘date_1’: u’%21%’}]

This happens on every version of privacyidea from 2.11 to 2.19.



Discussion:
In the case of Postgres, we apparently need to cast the timestamp to string in order to compare it using "LIKE" ([see here](https://stackoverflow.com/a/4799730)). I guess we can do that!
But maybe the cooler solution in the long term would be to be able to specify a date range.

Resolving patch:
diff --git a/privacyidea/lib/auditmodules/sqlaudit.py b/privacyidea/lib/auditmodules/sqlaudit.py
--- a/privacyidea/lib/auditmodules/sqlaudit.py
+++ b/privacyidea/lib/auditmodules/sqlaudit.py
@@ -41,7 +41,7 @@
 import logging
 from privacyidea.lib.auditmodules.base import (Audit as AuditBase, Paginate)
 from privacyidea.lib.crypto import Sign
-from sqlalchemy import MetaData
+from sqlalchemy import MetaData, cast, String
 from sqlalchemy import asc, desc, and_, or_
 import datetime
 import traceback
@@ -144,14 +144,16 @@ def _create_filter(param, timelimit=None):
                         conditions.append(getattr(LogEntry, search_key) ==
                                           int(search_value))
                     else:
-                        # All other keys are strings
+                        # All other keys are compared as strings
+                        column = getattr(LogEntry, search_key)
+                        if search_key == "date":
+                            # but we cast "date" to a string first (required on postgresql)
+                            column = cast(column, String)
                         search_value = search_value.replace('*', '%')
                         if '%' in search_value:
-                            conditions.append(getattr(LogEntry,
-                                                      search_key).like(search_value))
+                            conditions.append(column.like(search_value))
                         else:
-                            conditions.append(getattr(LogEntry, search_key) ==
-                                              search_value)
+                            conditions.append(column == search_value)
                 except Exception as exx:
                     # The search_key was no search key but some
                     # bullshit stuff in the param


Test patch: