mteb/LocBenchRR huggingface dataset:
shard_map should support static_argnums
We've started using (a type-based derivative of) shard_map as a function decorator. Ideally we'd like to annotate our top-level training function with `@jax.jit @typed_shard_map` decorator on a single top-level function. However, `shard_map` doesn't support `static_argnums`. We're forced to add a level of nested functions, in order to capture the static arguments in a closure, see this example:

https://github.com/MatX-inc/seqax/blob/871bd91efc004e569aa91228ef0a66931ed16986/train.py#L292-L295

Would it be possible to add `static_argnums` support to `shard_map`? That would allow us to simplify the above to:

```
@partial(jax.jit, static_argnums=(2, 3), donate_argnums=(0,))
@typed_shard_map
def training_step(state: State, step: u32[b''], h: Hparams, hparams: TrainingHparams, batch: TokenBatch) -> Tuple[Any, Metrics]:
  ...
```
`utils.sanitize_path` behavioral differences <3.12
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

https://github.com/python/cpython/pull/100351 implements proper splitting for UNC paths.

We should implement compat for Python <3.12 path parsing and sanitization.

See this comparison:
```xonsh
@ py -3.11 -c r'import os.path; print(os.path.splitdrive("\\\\?\\UNC\\ComputerName\\Share\\path"))'
('\\\\?\\UNC', '\\ComputerName\\Share\\path')

@ py -3.12 -c r'import os.path; print(os.path.splitdrive("\\\\?\\UNC\\ComputerName\\Share\\path"))'
('\\\\?\\UNC\\ComputerName\\Share', '\\path')
```
The underlying `nt` function works consistently:
```xonsh
@ py -3.11 -c r'import nt; print(nt._path_splitroot("\\\\?\\UNC\\ComputerName\\Share\\path"))'
('\\\\?\\UNC\\ComputerName\\Share\\', 'path')

@ py -3.12 -c r'import nt; print(nt._path_splitroot("\\\\?\\UNC\\ComputerName\\Share\\path"))'
('\\\\?\\UNC\\ComputerName\\Share\\', 'path')
```
As is also documented [here](https://learn.microsoft.com/en-us/dotnet/standard/io/file-path-formats#evaluate-relative-components).

### Provide verbose output that clearly demonstrates the problem

- [ ] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [ ] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
```
Feature Request: Add Functionality to Apply Constraints to Predictions
I am proposing the addition of a new method to our model class, designed to apply constraints to predictions to ensure that the values fall within specified bounds. This functionality would be useful for maintaining the integrity of our model's predictions in scenarios where certain variables have predefined limits.

Proposed Method:

```
def apply_constraints(self, prediction):
    """
    Apply constraints to prediction to ensure values are within the
    specified bounds
    """
    for param, (min_val, max_val) in constants.PARAM_CONSTRAINTS.items():
        indices = self.variable_indices[param]
        for index in indices:
            # Apply clamping to ensure values are within the specified
            # bounds
            prediction[:, :, :, index] = torch.clamp(
                prediction[:, :, :, index],
                min=min_val,
                max=max_val if max_val is not None else float("inf"),
            )
    return prediction
```
Rationale:

Data Integrity: Ensuring that predictions adhere to real-world constraints is essential for the reliability of our model's outputs. This method would allow us to enforce these constraints directly on the prediction tensor.
Flexibility: By dynamically applying constraints based on the variable indices, we can maintain a high degree of flexibility in how we handle different variables with varying constraints.

The method could be added to the ARModel class, which is our primary model class.
The constants.PARAM_CONSTRAINTS dictionary, which maps variable names to their minimum and maximum values, should be used to determine the constraints for each variable.

```
PARAM_CONSTRAINTS = {
    "RELHUM": (0, 100),
    "CLCT": (0, 100),
    "TOT_PREC": (0, None),
}
```

This feature is closely related to #18
rds_instance_no_public_access reports incorrect SG
### Steps to Reproduce

1. prowler aws -c rds_instance_no_public_access
2. AWS
3. N/A
4. See finding's `status_detail`

### Expected behavior

The status detail should correctly identify the Security Group(s) that contain the rule(s)

### Actual Result with Screenshots or Logs

Only one group is in the detail, and it may not be the group that actually has the rule(s). Result may change between runs
![rds_check_correct](https://github.com/user-attachments/assets/743e8e8f-cd97-40f1-8e7e-e85f94c1eb1d)
![rds_check_incorrect](https://github.com/user-attachments/assets/fe7dd840-a636-4ff5-b19f-ef8a6eac5bfc)


### How did you install Prowler?

From pip package (pip install prowler)

### Environment Resource

ECS/Locally

### OS used

Amazon Linux 2/WSL

### Prowler version

Prowler 4.5.3 (You are running the latest version, yay!)

### Pip version

pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

### Context

No changes made between scan runs.
`Default` SG is attached but has no rules.
Abstract `_run` as much as possible
## Description
The `_run()` method implementation in the various runners is very similar. We can refactor this to be even more similar so we can move all common code into the `_run()` in the `AbstractRunner`.

## Context
This will make it easier for people to implement a custom runner, because the `_run()` method will have more in place to get started from.
Move Trainer's tokens per second metric into the inner training loop
### Feature request

Right now `include_tokens_per_second=True` in `Trainer` only reports the tokens per second metric at [the end of training](https://github.com/huggingface/transformers/blob/c1753436dbb8bcbcee183cdd6eba9f08a90d602a/src/transformers/trainer.py#L2610). It would be very useful to have this metric reported continuously inside the training loop so we can monitor it during training. 

### Motivation

The current behavior is counter-intuitive, doesn't align with other convenient trainers (like torchtune), and it's undocumented, so I had to RTFC to figure out why the metric wasn't showing up.

### Your contribution

I probably don't have time to contribute it myself.
Hide navigation items when user lacks permission
API v2 provides the navigation structure for the UI to build the navigation tree. This needs to be filtered.


## Details

If a user lacks permission for the location of the nav path, filter it out so it's not rendered within the interface.


## Tasks

- [x] Filter navigation based off of user permissions


## Requirements

-  [ ] Test to ensure that items the user has no access to, is provided.

Hide global organization from organization dropdown
If a global organization has been set, some items are automagically moved to this organization.


## Details

As the global organization is managed it should not be selectable within the organization field by anyone.

For items that are set as global, the `organization` field should be removed as should the `is_global`.


## Tasks

- [ ] #419
Add support for CompositePrimaryKey in inspectdb
Description
	 
We can now replace the ​half-measure created when introspecting a composite primary key with the actual field.
​PR
[Bug]: supported params are out of date for fireworks AI
### What happened?

when calling fireworks models, litellm is complianing: logprobs is not supproted but it's actually supported by fireworks ai.
ref: https://docs.fireworks.ai/api-reference/post-completions

### Relevant log output

_No response_

### Twitter / LinkedIn details

_No response_
generate_preset_pass_manager should take initial_layout as a list
### What should we add?

Now that https://github.com/Qiskit/qiskit/pull/10344 is merged, we should make the `generate_preset_pass_manager` function take an integer list for layout. This is a user entry point and should be easy to use. The Layout object is a pain to use all around (I would love to completely eliminate it, since it doesn't contain any more information than a list of int. The circuit qubits have an order).
makemessages: is_valid_locale regex fails to validate locales with numeric region codes
Description
		
(last modified by Juan Pablo Mallarino)		
The is_valid_locale function in the makemessages management command uses a regular expression that doesn't account for locales with numeric region codes, such as es_419. The current regex only allows uppercase letters after the underscore. This excludes valid locale codes that use numbers in the region subtag.
This can cause issues when trying to generate message files for these locales. For example, running makemessages with es_419 will not generate the expected message files, potentially leading to missing translations.
Proposed Solution
Modify the regular expression in is_valid_locale to include numeric characters in the region subtag validation. The suggested change is:
From:
r"^[a-z]+_[A-Z].*$"
To:
r"^[a-z]+_[A-Z0-9].*$"
This change would allow the validation of locales while still maintaining the expected format for other locale codes. This simple modification would ensure broader compatibility and avoid unexpected behavior when working with valid locales containing numeric region codes.
BUG: signal: Confusing error when giving an invalid mode to `correlation_lags`
### Describe your issue.

When correlation_lags is given a mode that is not "valid", "same", or "full", the `lags` array is never defined, and the function fails with `UnboundLocalError: local variable 'lags' referenced before assignment`.
The fix is very simple, I will submit a PR soon.

### Reproducing Code Example

```python
signal.correlation_lags(10, 10, "some_invalid_format")
```


### Error message

```shell
Traceback (most recent call last)

<ipython-input-25-202bff163744> in <cell line: 1>()
----> 1 signal.correlation_lags(10, 10, "some_invalid_format")

/usr/local/lib/python3.10/dist-packages/scipy/signal/_signaltools.py in correlation_lags(in1_len, in2_len, mode)
    384         else:
    385             lags = np.arange(lag_bound, 1)
--> 386     return lags
    387 
    388 

UnboundLocalError: local variable 'lags' referenced before assignment
```


### SciPy/NumPy/Python version and system information

```shell
1.13.1 1.26.4 sys.version_info(major=3, minor=10, micro=12, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /usr/local/include
    lib directory: /usr/local/lib
    name: openblas
    openblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= ZEN MAX_THREADS=64
    pc file directory: /usr/local/lib/pkgconfig
    version: 0.3.27
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /usr/local/include
    lib directory: /usr/local/lib
    name: openblas
    openblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS=
      NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= ZEN MAX_THREADS=64
    pc file directory: /usr/local/lib/pkgconfig
    version: 0.3.27
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 2.12.0
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.10
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 10.2.1
  pythran:
    include directory: ../../tmp/pip-build-env-mnl4e8vy/overlay/lib/python3.10/site-packages/pythran
    version: 0.15.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /opt/python/cp310-cp310/bin/python
  version: '3.10'

```
```
CLI for empty DB raises `ValueError`
### Expected behavior

CLI for empty DB should output empty result, but the current implementation raises `ValueError`.

### Environment

- Optuna version:4.2.0.dev
- Python version:3.13.0
- OS:macOS-15.1-x86_64-i386-64bit-Mach-O
- (Optional) Other libraries and their versions:

### Error messages, stack traces, or logs

```shell
See below.
```


### Steps to reproduce

For empty DB (`tmp.db` does not exist before the command), the `optuna studies` command raises `ValueError`.

```bash
$ optuna --storage sqlite:///tmp.db studies
Traceback (most recent call last):
  File "/Users/naotomizuno/.pyenv/versions/optuna-3.13.0/bin/optuna", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/Users/naotomizuno/optuna/optuna/cli.py", line 991, in main
    return args.handler(args)
           ~~~~~~~~~~~~^^^^^^
  File "/Users/naotomizuno/optuna/optuna/cli.py", line 463, in take_action
    _format_output(
    ~~~~~~~~~~~~~~^
        records, self._study_list_header, parsed_args.format, parsed_args.flatten
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/naotomizuno/optuna/optuna/cli.py", line 258, in _format_output
    return _dump_table(values, header).strip()
           ~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/Users/naotomizuno/optuna/optuna/cli.py", line 222, in _dump_table
    max_width = max(len(header[column]), max(row[column].width() for row in rows))
                                         ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: max() iterable argument is empty
```


### Additional context (optional)

_No response_
Issue with Comment Handling in `Describe` Output
Hi,

I encountered an issue with the [`Describe`](https://sqlglot.com/sqlglot/expressions.html#Describe) functionality in `sqlglot`. The comments in the SQL query are not being preserved as expected.

Here’s a minimal example to illustrate the problem:

```python
import os
# os.environ['SQLGLOTRS_TOKENIZER'] = '0'

import importlib.metadata
import sqlglot

package_name = 'sqlglot'
version = importlib.metadata.version(package_name)
print(f"{package_name} version: {version}\n")

sql = '/*left*/ EXPLAIN SELECT /*hint*/ col FROM t1 /*right*/'

ast = sqlglot.parse(sql, read='mysql')

print(ast)
print(list(map(lambda x: x.sql(dialect='mysql'), ast)))
```

### Output:
```
sqlglot version: 25.30.0

[Describe(
  this=Select(
    expressions=[
      Column(
        this=Identifier(this=col, quoted=False))],
    from=From(
      this=Table(
        this=Identifier(this=t1, quoted=False),
        _comments=[
          right])),
    _comments=[
      hint]))]
['DESCRIBE /* hint */ SELECT col FROM t1 /* right */']
```

### Expected Output:
```sql
/*left*/ DESCRIBE /* hint */ SELECT col FROM t1 /* right */
```

### Actual Output:
```sql
DESCRIBE /* hint */ SELECT col FROM t1 /* right */
```
ray/serve/generated file is missing after running setup-dev.py
### What happened + What you expected to happen

When running `python setup-dev.py`, it creates softlink for each python package. However, since the generated folder is not part of the repository, creating the symbolic link for the `serve` package inadvertently overwrites the folder and the generated folder can't be found anymore.

### Versions / Dependencies

Lastest

### Reproduction script

```

pip install -U "ray[serve] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp39-cp39-macosx_11_0_arm64.whl"
python python/ray/setup-dev.py

```

### Issue Severity

Medium: It is a significant difficulty but I can work around it.
CKV_AZURE_136: False-Positive For Read Replicas In Azure Database For PostgreSQL - Flexible Server
**Describe the issue**
CKV_AZURE_136 fails for read replicas in Azure Database for PostgreSQL - Flexible Server.
According to the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-read-replicas), geo-redundant backups are not supported for replicas.

> Unsupported features on read replicas
Certain functionalities are restricted to primary servers and can't be set up on read replicas. These include:
Backups, including geo-backups.
High availability (HA)

> ..replicas can't have geo-backup enabled. The feature can only be activated at the standard server's creation time (not a replica).

Checkov should not report this check as failed when evaluating replicas.

**Examples**
```
resource "azurerm_postgresql_flexible_server" "replica" {
  count               = var.replica_count
  name                = "${local.database_name}-replica-${count.index}"
  resource_group_name = var.resource_group.name
  location            = var.resource_group.location
  delegated_subnet_id = var.shared.subnet_id
  private_dns_zone_id = var.shared.dns_zone.id
  sku_name            = var.sku_name
  storage_mb          = var.storage_mb
  version             = var.postgresql_version

  [...]

  # replication
  create_mode      = "Replica"  # <-- This makes the server a replica.
  source_server_id = azurerm_postgresql_flexible_server.primary.id

  tags = local.standard_tags
  lifecycle {
    precondition {
      condition     = !startswith(var.sku_name, "B_")
      error_message = "Replicas are not supported for burstable SKUs."
    }
    ignore_changes = [
      zone,
      high_availability.0.standby_availability_zone,
      tags
    ]
  }
}
```

**Version (please complete the following information):**
 - Checkov Version 3.2.22
feat(snowflake): Adding support for DATE, TO_DATE, TRY_TO_DATE functions
Fixes #3152 

Add support for the mentioned functions by:

- Converting all of them to `exp.TsOrDsToDate` expressions if the format is present _or_ to date casts otherwise
- Adding a new argument `safe` in `exp.TsOrDsToDate` to preserve the roundtrip of `TRY_TO_DATE`, which has a different error handling behavior compared to `TO_DATE / DATE` 

For the transpilation issue #3152, no change was needed in DuckDB as it already transpiles the aforementioned expression to `strptime` (note that it produces a `TIMESTAMP`, thus the cast to `DATE`):

```
>>> import sqlglot
>>> sqlglot.transpile("SELECT try_to_date('01-01-1999', 'MM-DD-YYYY')", read="snowflake", write="duckdb")
SELECT CAST(STRPTIME('01-01-1999', '%m-%d-%Y') AS DATE)
```

Docs
-----------

1. [Snowflake TRY_TO_DATE](https://docs.snowflake.com/en/sql-reference/functions/try_to_date)
2. [Snowflake DATE / TO_DATE](https://docs.snowflake.com/en/sql-reference/functions/to_date)
3. [DuckDB strptime](https://duckdb.org/docs/sql/functions/dateformat.html#strptime-examples)
[Feature] Migrate from OpenSearch to Elasticsearch to improve Search Performance
## Work Details
- Current performance with AWS OpenSearch Free Tier shows query times averaging 1.5 seconds.
- Plan to migrate to a self-hosted Elasticsearch instance on EC2(free tier) using Docker.
- Expected outcome: Reduce query times significantly, targeting under 500ms based on preliminary tests showing potential improvements.

## Task List
- [x] Set up Docker environment on EC2 for Elasticsearch deployment.
- [x] Migrate existing data and queries from OpenSearch to Elasticsearch.
- [x] Monitor and benchmark performance post-migration to ensure query times meet expectations.

This plan aims to enhance the search experience by leveraging the capabilities of Elasticsearch, which has demonstrated superior performance compared to OpenSearch in our initial assessments. The goal is to achieve a more responsive system that can handle user queries efficiently.
Bigquery annotate_types incorrectly sets TIMESTAMP as DATETIME even when schema type is provided
```python
import sqlglot
import sqlglot.errors
from sqlglot.optimizer import optimize

dialect = "bigquery"
schema = {"test": {"products": {"created_at": "TIMESTAMP"}}}
expression = sqlglot.parse_one("SELECT * FROM test.products WHERE created_at > '2024-10-01'", read=dialect)
optimized_expression = optimize(
    expression,
    dialect=dialect,
    schema=schema,
)
formatted_sql = optimized_expression.sql(dialect=dialect, pretty=True)
print("@formatted_sql", formatted_sql)
```
formatted_sql is 
```SQL
SELECT
  `products`.`created_at` AS `created_at`
FROM `test`.`products` AS `products`
WHERE
  `products`.`created_at` > CAST('2024-10-01' AS DATETIME)
```

which cannot be ran due to 
No matching signature for operator < for argument types: TIMESTAMP, DATETIME.
Same order of training samples with `NoDuplicatesBatchSampler`
It seems that  `NoDuplicatesBatchSampler` produces the same set of batches in the same order regardless of the epoch index.
Indeed, in [this piece of code](https://github.com/UKPLab/sentence-transformers/blob/e156f38b1007e7860218c27223e1d577f3a021fe/sentence_transformers/sampler.py#L184C3-L187C102), the order of the indices in `remaining_indices` does not depend on the random permutation `torch.randperm(len(self.dataset), generator=self.generator)` as it is reset to the ordered range with `set`.

Moreover, the seed in line 185 does not change from one epoch to another (the [`set_epoch`](https://github.com/UKPLab/sentence-transformers/blob/e156f38b1007e7860218c27223e1d577f3a021fe/sentence_transformers/sampler.py#L32) method does not seem to be used...)
Several problems in RLOOTrainer
### System Info

main

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder
- [ ] My own task or dataset (give details below)

### Reproduction

1. metrics["loss/value_avg"] = self.accelerator.gather(vf_loss_stats).mean().item()
this metrics is useless since we don't use value func in RLOO

2. self.state.epoch = self.state.episode / self.train_dataset_len  # used by self.log 
This will cause problems on calculation since the true epoch count is (self.state.episode / (args.rloo_k * self.train_dataset_len) Since every instruction is repeated args.rloo_k times

3. Multiple self.state.global_step += 1
will cause the saving process to go wrong




### Expected behavior

this to be right