mteb/LocBenchRR huggingface dataset:
snippet performance issue
**CKAN version**
all

**Describe the bug**
Heavy use of template `{% snippet %}`  slows page rendering times compared to alternatives like macros or `{% include %}`

**Steps to reproduce**
Compare the rendering time of a page using snippets with one that uses macros or `{% include %}` to display the same information. The activity feed, the search results page and customized dataset forms when using ckanext-scheming are examples of pages with lots of snippets.

**Expected behavior**
Render time for `{% snippet %}` should be similar to macros and `{% include %}`

**Additional details**
`{% snippet %}` is a great tag worth saving, not just for compatibility.

Unlike `{% include %}` snippets support passing parameters, and unlike macros snippet parameters can be added over time without causing errors. Snippets are easier for developers to use than deeply extended templates with blocks selectively overridden.

The performance issue may be caused by the way jinja2 converts templates into byte code - a template passed to `render()` needs to be interpreted and can't use the byte code cache, unlike extended templates or imported macros. Our `{% snippet %}` tag calls `render()` returning the resulting string so snippets are re-interpreted every time they are used, even if it's the exact same snippet used in a loop. 

A possible fix is to have the snippet tag automatically convert snippets into a macro with the parameters passed by the caller. The macro-ified snippets could be generated with a dynamic template loader instead of creating them on the filesystem.  The snippet tag would convert itself into an `{% import %}` and a call to the macro generated.
[adag] Avoid deserialization during CompiledDAGRef's deallocation
### What happened + What you expected to happen

Although we don't call ray.get, ray.get is called and deserialization still happens when the dag ref is deallocated because of the following code. https://github.com/ray-project/ray/blob/4aea49f5301ac083aed5d016919482afd2f54f83/python/ray/experimental/compiled_dag_ref.py#L80


### Versions / Dependencies

ray master

### Reproduction script

```
a = Actor.remote(0)
with InputNode() as inp:
    dag = a.inc.bind(inp)

compiled_dag = dag.experimental_compile()
ref = compiled_dag.execute(1)
compiled_dag.teardown()

# ref.get() is called upon ref's deallocation -- deserialization still happens
```

### Issue Severity

None
Add sparse matrix output to SplineTransformer
### Describe the workflow you want to enable

As B-splines naturally have a sparse structure, I'd like to have the option that `SplineTransformer` returns a sparse matrix instead of always an ndarray.
```python
import numpy as np
from sklearn.preprocessing import SplineTransformer

X = np.arange(6).reshape(6, 1)
spline = SplineTransformer(degree=2, n_knots=3, sparse=True)
spline.fit_transform(X)
```

### Describe your proposed solution

With scipy >= 1.8 (yet to be released), we can use `design_matrix` from https://github.com/scipy/scipy/pull/14344.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_
Increase parsing speed
This task contains several steps:

1. Search different ways that might increase parsing speed. Parsing is done right now by the `pandas.read_xml` method [here](https://github.com/OpenEnergyPlatform/open-MaStR/blob/d17a92c17912f7f96814d309859dc495f942c07a/open_mastr/xml_download/utils_write_to_database.py#L158). Several alternatives are:
  * `polars`, `duckdb`, `pyspark` might have xml parsers and might be faster
  * use plain xml parsing from python (without pandas)
  * ...

2. Writing to sqlite database right now is done by `pandas.to_sql` [here](https://github.com/OpenEnergyPlatform/open-MaStR/blob/d17a92c17912f7f96814d309859dc495f942c07a/open_mastr/xml_download/utils_write_to_database.py#L182C5-L182C26). There might be other faster methods depending on step 1.
3. Construct a benchmark in an own repository. Use a benchmark xml file from the Marktstammdatenregister and test different implementations for parsing them.
4. Decide for a best method and implement it in `open-mastr`
Support Constant Learning Rate with Cooldown
### Feature request

In `transformers.optimization` support `constant learning rate with cooldown` functions.


### Motivation

This method will implement that scaling experiments can be performed with significantly reduced compute and GPU hours by utilizing fewer but reusable training runs.
[SmolLM](https://huggingface.co/blog/smollm) had used this method to train a series of SOTA small language models.
Paper: [https://arxiv.org/pdf/2405.18392](https://arxiv.org/pdf/2405.18392)

### Your contribution

I've created a branch, I'm finishing the implementation of these functions, and intend to submit a PR.
Make control naming consistent
**Is your feature request related to a problem? Please describe.**
The variables that are optimized (controls in Everest nomenclature) are specified in a nested fashion: We have controls, which defines groups of variables, which in turn can have multiple components labeled with an integer index. For example (from `config_advanced.yml`):

```yml
controls:
  - name: point
    max: 1.0
    min: -1.0
    initial_guess: 0.25
    perturbation_magnitude: 0.005
    type: generic_control
    variables:
      - name: x
        index: 0
      - name: x
        index: 1
      - name: x
        index: 2
```
 This defines three controls, with the following `control`, `variable`, and `index` fields:

- `point`, `x`, `0`
- `point`, `x`, `1`
- `point`, `x`, `2`

When reporting results, control names are constructed from the three fields as follows `control_variable-index`, so we get:
- `point_x-0`
- `point_x-1`
- `point_x-2`

When specifying input constraints, weights need to be set for a sub-set of controls, so controls are needed, but in this case they follow the form `control.variable-index`. For instance (from `config_advanced.yml`):

```yml
input_constraints:
  - weights:
      point.x-0: 0
      point.x-1: 0
      point.x-2: 1
    upper_bound: 0.4
```

This is inconsistent and potentially confusing for the user.

**Describe the solution you'd like**
When a string representation of controls is shown, or required as an input, it should always have the same form. There should be a singe function/method responsible for formatting the string from `control`, `variable`, `index` fields. Preferably this transformation should be invertible, although we should not rely on that in code. If possible, the delimiters used should not be allowed in control and variable names. We should discuss the delimiters to use, but using `.` and `-` would be backwards compatible for specifying input constraints and it would leave the underscore available for control and variable naming.
Prowler CSV Output no longer outputs Failed Critical Findings
### Steps to Reproduce

Running prowler 4.5.3
Execute prowler aws against an AWS account. 
CSV Outputs PASS critical findings but does output FAIL findings.

Prowler Dashboard does not display any cirtical findings

It was working with 4.3.5 but after I upgraded to 4.5.3 this now fails.

json-ocsf files are fine. These do contain Critical Fail


### Expected behavior

Failed critical findings to be contained in CSV file to use with prowler dashboard

### Actual Result with Screenshots or Logs

prowler aws --output-bucket-no-assume prowler-output-bucket-XXXXXXXXXXXX --output-directory output --slack --role arn:aws:iam::XXXXXXXXXXXX:role/XXXXXXXXXXX --role-session-name XXXXX_XXXXXX

### How did you install Prowler?

From pip package (pip install prowler)

### Environment Resource

2. Fargate Task running python:alpine with python 3.12

### OS used

3. Alpine Linux

### Prowler version

4.5.3

### Pip version

3.12

### Context

_No response_
Weekly CI run with NVidia GPU hardware
Now that #22554 was merged in `main`, it would be great to find a a way  to run a weekly scheduled job to run the scikit-learn `main` test on a CI worker with an NVidia GPU and CuPy.

In case of failure, it could create a report as [dedicated issues](https://github.com/scikit-learn/scikit-learn/issues?q=is%3Aissue+%22%E2%9A%A0%EF%B8%8F+ci+failed+on%22+) as we do for other scheduled jobs:

- https://github.com/scikit-learn/scikit-learn/blob/main/maint_tools/update_tracking_issue.py

Maybe @betatim has a plan and connections to do that? ;)
Reduce CreateModel + AlterModelTable to CreateModel
Description
	 
Like #33572, migration optimization can reduce CreateModel and AlterModelTable for the same model down to just CreateModel with db_table in options.
[Bug][Performance] Significant overhead when `record_stream` for subgraph in dataloading
## 🐛 Bug

It seems there are many overheads observed in dataloader during feature prefetching when `use_alternate_streams=True` . When it calls `_record_stream` for every subgraph (batch), like the following: 
https://github.com/dmlc/dgl/blob/829ce109f1604ee59ac039331a3609896301e8de/python/dgl/dataloading/dataloader.py#L339
many unnecessary h2d copies (or gpu feature slicing if use_uva) are invoked from L339, due to materializing the column data(feature) within its `record_stream` method, i.e., within its `data` method, for example from here (when uva is disabled)
https://github.com/dmlc/dgl/blob/829ce109f1604ee59ac039331a3609896301e8de/python/dgl/frame.py#L246-L251
called by `record_stream` method for each column:
https://github.com/dmlc/dgl/blob/829ce109f1604ee59ac039331a3609896301e8de/python/dgl/frame.py#L551

It poses significant slow-down for both cases: `use_uva=True` and `use_uva=False`, for example, when running `example/graphsage/node_classification.py`, with uva on, `use_alternate_streams=True` causes **1.56x** more time for each epoch, while with uva off, `use_alternate_streams=True` (default) causes **1.55x** more time for each epoch.


## To Reproduce

set `use_alternate_streams=True` and run

```
python node_classification.py
```

## Environment

 - DGL Version (e.g., 1.0): 1.0
 - Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): Pytorch 1.13
 - How you installed DGL (`conda`, `pip`, source): source
 - GPU models and configuration (e.g. V100): A5000
flow.validate_parameters regression
### Bug summary

Certain flow deployments that worked fine before cannot be executed anymore after updating from Prefect 2.20.9 to any newer Prefect version.
This is due to a change in the parameter validation.

MRE:
```python
from prefect import flow

@flow
def my_flow(param: dict[str, str]):
    pass

flow_run_params = {
    "param": {'foo': 'bar'}
}

my_flow.validate_parameters(flow_run_params)
```
runs fine for Prefect <= 2.20.9.

But raises
```python
Traceback (most recent call last):
  File "mre.py", line 13, in <module>
    my_flow.validate_parameters(flow_run_params)
  File "mre.py", line 532, in validate_parameters
    has_v1_models = any(
  File "mre.py", line 533, in <genexpr>
    issubclass(param.annotation, V1BaseModel)
  File "/usr/lib/python3.10/abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
```
for Prefect >= 2.20.10


### Version info

```Text
Version:             2.20.14
API version:         0.8.4
Python version:      3.10.12
Git commit:          fb919c67
Built:               Mon, Nov 18, 2024 4:41 PM
OS/Arch:             linux/x86_64
Server type:         cloud
```


### Additional context

related to https://github.com/PrefectHQ/prefect/pull/15608
Minimum TLS 1.2 fails for Azure Web App when TLS 1.3 is enabled
### Steps to Reproduce

1. prowler azure --subscription-ids xx-88ae-4fe8-901a-16e33871e7c7 xx-5c28-4e32-94df-591a5baedf69 --az-cli-auth
2. Azure Web App with TLS 1.3 enabled

### Expected behavior

in file prowler/providers/azure/services/app/app_minimum_tls_version_12/app_minimum_tls_version_12.py

When checking for minimum TLS 1.2, please don't fail on TLS 1.3

### Actual Result with Screenshots or Logs

"Minimum TLS version is not set to 1.2 for app"

```
                if (
                    app.configurations
                    and getattr(app.configurations, "min_tls_version", "") == "1.2"
                ):
                    report.status = "PASS"
                    report.status_extended = f"Minimum TLS version is set to 1.2 for app '{app_name}' in subscription '{subscription_name}'."

```

### How did you install Prowler?

From pip package (pip install prowler)

### Environment Resource

Azure Cloud Shell

### OS used

Azure Cloud Shell

### Prowler version

4.5.3 

### Pip version

pip 23.0.1 from /usr/lib/python3.9/site-packages/pip (python 3.9)

### Context

_No response_
Support Request: RF01 rule
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### Description

It seems sqlfluff can’t handle the dot notation we use for Redshift SUPER data type. RF01 rule (https://docs.sqlfluff.com/en/stable/rules.html#rule-references.from).

Usage example:

```
w_extract as (
    select
        pj.hash_key,
        pj.snapshot_date,
        trim(cast(pj.metadata_super."amount" as numeric))                                       as amount,
        trim(cast(pj.metadata_super."leadAge" as numeric))                                      as lead_age,
        trim(cast(pj.metadata_super."leadType" as varchar))                                     as lead_type,
        trim(cast(pj.metadata_super."leadValue" as numeric))                                    as lead_value,
        trim(cast(pj.metadata_super."randValue" as numeric))                                    as rand_value,
        trim(cast(pj.metadata_super."callReason" as varchar))                                   as call_reason
    from w_parse_json as pj
)
```
Failures:
```
L:  49 | P:  22 | RF01 | Reference 'pj.metadata_super."amount"' refers to
                       | table/view not found in the FROM clause or found in
                       | ancestor statement. [references.from]
L:  50 | P:  22 | RF01 | Reference 'pj.metadata_super."leadAge"' refers to
                       | table/view not found in the FROM clause or found in
                       | ancestor statement. [references.from]
L:  51 | P:  22 | RF01 | Reference 'pj.metadata_super."leadType"' refers to
                       | table/view not found in the FROM clause or found in
                       | ancestor statement. [references.from]
L:  52 | P:  22 | RF01 | Reference 'pj.metadata_super."leadValue"' refers to
                       | table/view not found in the FROM clause or found in
                       | ancestor statement. [references.from]
L:  53 | P:  22 | RF01 | Reference 'pj.metadata_super."randValue"' refers to
                       | table/view not found in the FROM clause or found in
                       | ancestor statement. [references.from]
L:  54 | P:  22 | RF01 | Reference 'pj.metadata_super."callReason"' refers to
                       | table/view not found in the FROM clause or found in
                       | ancestor statement. [references.from]
```

Can we improve this so this is automatically handled via ```sqlfluff fix```? Currently, this is an unfixable violation and we simply have to ignore it. 

### Use case

_No response_

### Dialect

-

### Are you willing to work on and submit a PR to address the issue?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)
Deprecate Series.view
Same as the others, that's deprecated in pandas for 3.0 so we can move ahead with it already
Addition of /docs endpoint
Creation of /docs endpoint in order to have a place to view easily all the avilable endpoints with:

- a basic endpoint description.
- its development status/maturity
- requiered and optional parameters
- example of request body
- example of a succesful response
- custem errors explanation(if exist)
Poor Performance on TPC-H Queries
In a benchmark project ([link](https://github.com/hesamshahrokhi/modin_tpch_benchmark)), I tried to compare the performance of Pandas and Modin (on Ray and Dask) on simple TPC-H queries (Q1 and Q6) written in Pandas. The results ([link](https://github.com/hesamshahrokhi/modin_tpch_benchmark/blob/main/Results_TPCH_SF1.txt)) show that the ```read_table``` API is 2x faster than its Pandas equivalent on Ray. However, the same API on Dask, and also the individual query run times on both backends are much slower than Pandas. Moreover, the results of Q1 are not correct.
 
Could you please help me to understand why this is the case? I tried to use the suggested configurations and imported the data using ```read_table``` which is officially supported by Modin.
`CrossEncoder` is not pushed to cuda until predict is called, even if cuda is specified as device.
Hi, this is more like a question rather than a bug or issue.

When I specify the target device during initialization of any CrossEncoder, the model is not pushed to that device until the `predict` or the `fit` method is called, until then the model is kept in cpu.

```python
from sentence_transformers import CrossEncoder

model2 = CrossEncoder("mixedbread-ai/mxbai-rerank-large-v1", device="cuda:0")
print(model2.model.device)
# cpu
```

I mean I expect the model to be pushed to the specified device during initialization and until I am calling predict it is taking up my system Ram.
Is there any high level reason(s) why this is the case?
Allow comma-separated values for `CREATE SEQUENCE` properties (Snowflake)
Allow comma-separated values for `CREATE SEQUENCE` properties in Snowflake queries.

This query can be parsed successfully (results is an instance of `exp.Create`):
```
    query = "CREATE SEQUENCE seq1 WITH START=1 INCREMENT=1 ORDER"
    result = parse_one(query, dialect="snowflake")
    print(type(result))

    --> <class 'sqlglot.expressions.Create'>
```

... whereas the following query cannot be fully parsed and is returned as a generic `exp.Command` as fallback:
```
    query = "CREATE SEQUENCE seq1 WITH START=1, INCREMENT=1 ORDER"
    result = parse_one(query, dialect="snowflake")
    print(type(result))

    --> <class 'sqlglot.expressions.Command'>
```

(note - the only subtle difference is the comma `,` between the `START` and `INCREMENT` properties)

Note that the second query syntax may not be covered in the Snowflake docs, but it is indeed a valid syntax that can be executed against Snowflake (and some Snowflake client apps/SDKs are generating queries that are using this syntax, unfortunately).
SAM Dump for accounts without secrets
I realised that some defaults Windows accounts, like for example WDAGUtilityAccount, throw the following error:

![image](https://github.com/user-attachments/assets/cb6a1d68-d50a-4a0b-8689-41bf878a9ca2)

However there is no error here. WDAGUtilisatyAccount does not have a NT hash in the SAM database because this is a virtual account used to contain applications in a sandbox (for example browsers) and these featuers are not used on windows servers. Considering I never saw secretsdump failing in dumping SAM database, I believe it is possible to switch the following liens from impacket/impacket/examples/secretsdump.py:

```python
if userAccount['NTHashLength'] == 0:
    logging.error('SAM hashes extraction for user %s failed. The account doesn\'t have hash information.' % userName)
    continue
```

to 

```python
if userAccount['NTHashLength'] == 0:
    logging.debug('SAM hashes extraction for user %s failed. The account doesn\'t have hash information.' % userName)
    continue
```

That way most of tools using impacket secretsdump won't have a messed up output.

Let me know what you think about this :)
Support the "Trusted Publishing" flow directly in `twine`
Last month PyPI added support for "Trusted Publishing": https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/. This uses an credential exchange flow to swap OIDC credentials with a short-lived PyPI API token, removing the need to maintain long-lived credentials/secrets. (More technical details on the exchange are here: https://docs.pypi.org/trusted-publishers/internals/)

We were able to do launch this without modifying `twine` directly because we instead updated https://github.com/pypa/gh-action-pypi-publish to do the token exchange prior to invoking `twine upload`. 

That said, we're actively working on expanding support to other providers for which we don't have "canonical" workflows (such as https://github.com/pypi/warehouse/issues/13551, https://github.com/pypi/warehouse/issues/13575, and https://github.com/pypi/warehouse/issues/13888).

For these providers (as well as for GitHub users who prefer not to use the `pypi-publish` GitHub action for whatever reason) it would be preferable to have `twine` be able to support the OIDC/API token exchange directly.

This would ideally include:

1. detecting whether an "ambient" OIDC token is available at the time `twine upload` is run (this can be handled by https://pypi.org/p/id, so `twine` doesn't need to understand how to do this for N different providers)
2. verifying that the OIDC token is acceptable for use with PyPI (this just requires that the `aud` claim of the token is set to the audience of the index that `twine` is attempting to upload to)
3. exchanging the OIDC token for a PyPI API token (this is essentially the same as what the `pypi-publish` workflow does here: https://github.com/pypa/gh-action-pypi-publish/blob/110f54a3871763056757c3e203635d4c5711439f/oidc-exchange.py)

The alternative to `twine` supporting this directly is that the end user has to perform a fairly manual token exchange themselves directly (https://docs.pypi.org/trusted-publishers/using-a-publisher/#the-manual-way) which would be pretty bad UX overall.

Opening this issue to make sure maintainers are on board with the general idea before any implementation is started, please let me know if you have any thoughts.

(cc @woodruffw)
[Bug Report] `refactor_factored_attn_matrices` uses lots of GPU memory
**Describe the bug**
When loading models with `refactor_factored_attn_matrices=True` and `device="cuda"`, they take up a lot of space on GPU. Thanks to @callummcdougall for noticing this. See example code at bottom for snippets and statistics.

**System Info**
vast.ai 2*3090 devbox, on branch main.

**Additional context**
Doing `import gc; gc.collect(); import torch; torch.cuda.empty_cache()` after the refactor stuff would help but if someone could look into further whether we can keep the permanent refactor objects on GPU (or not) that would be great and save some more trouble! If nothing else I can add a warning if we enter the refactor function with device equal to GPU.

### Checklist

- [ :white_check_mark:  ] I have checked that there is no similar [issue](https://github.com/neelnanda-io/TransformerLens/issues) in the repo (**required**)

Example code

On my machine with 3090s, this uses 46% of GPU memory:

```
model = HookedTransformer.from_pretrained(
    "solu-10l", # "NeelNanda/SoLU_10L1280W_C4_Code"
    device="cuda",    
    refactor_factored_attn_matrices=True,
)
```

I check memory with 

```
_r = torch.cuda.memory_reserved(0)
_a = torch.cuda.memory_allocated(0)
_p = 100*(_r-_a)/_r 
print(f"Percentage of reserved memory that is allocated: {_p:.2f}%")
```

After `import gc; gc.collect(); import torch; torch.cuda.empty_cache()` this is down to 6%

However, with this code

```
model = HookedTransformer.from_pretrained(
    "solu-10l", # "NeelNanda/SoLU_10L1280W_C4_Code"
    device="cpu",
    refactor_factored_attn_matrices=True,
)
model.to("cuda")
```

just 0.16% of my memory is used