mteb/LocBenchRR huggingface dataset:
scipy.stats.rankdata causes constant folding warning for method='dense' but not method='ordinal'
### Description

[`scipy.stats.rankdata`](https://jax.readthedocs.io/en/latest/_autosummary/jax.scipy.stats.rankdata.html) causes a constant folding warning for `method='dense'` but not `method='ordinal'`:

```
$ py -c "import jax; jax.scipy.stats.rankdata(jax.numpy.zeros(10**7), 'ordinal')"
$ py -c "import jax; jax.scipy.stats.rankdata(jax.numpy.zeros(10**7), 'dense')"
2024-11-04 20:21:27.997499: E external/xla/xla/service/slow_operation_alarm.cc:65] Constant folding an instruction is taking > 1s:

  %reduce-window.6 = s32[625000,16]{0,1} reduce-window(s32[625000,16]{0,1} %constant.174, s32[] %constant.17), window={size=1x16 pad=0_0x15_0}, to_apply=%region_5.113

This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.

If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
2024-11-04 20:21:33.721446: E external/xla/xla/service/slow_operation_alarm.cc:133] The operation took 6.728977s
Constant folding an instruction is taking > 1s:

  %reduce-window.6 = s32[625000,16]{0,1} reduce-window(s32[625000,16]{0,1} %constant.174, s32[] %constant.17), window={size=1x16 pad=0_0x15_0}, to_apply=%region_5.113

This isn't necessarily a bug; constant-folding is inherently a trade-off between compilation time and speed at runtime. XLA has some guards that attempt to keep constant folding from taking too long, but fundamentally you'll always be able to come up with an input program that takes a long time.

If you'd like to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
```

Looking at the code for `rankdata`, the culprit might be one of the 3 lines of code starting [here](https://github.com/jax-ml/jax/blob/ab47d4687f647de3aa145a9a782fb7b4aaf92af4/jax/_src/scipy/stats/_core.py#L206).

XLA dump [here](https://www.dropbox.com/scl/fo/rruuywlngh1r03hj9c2r1/AM-ym1pWfIUhkHA2hOiQNko?rlkey=2xxwdrmssgfyk7yz61xrt1t7d&st=h6yp3a8x&dl=0).

### System info (python version, jaxlib version, accelerator, etc.)

jax:    0.4.35
jaxlib: 0.4.34
numpy:  1.26.4
python: 3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 15.0.0 (clang-1500.3.9.4)]
device info: cpu-1, 1 local devices"
process_count: 1
platform: uname_result(system='Darwin', node='Carloss-MacBook-Pro-2.local', release='23.6.0', version='Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6031', machine='arm64')
Allow configuring `shift=` for SD3 dynamically
**Is your feature request related to a problem? Please describe.**
Allow passing `shift=` per inference call (like timesteps) on the pipeline, for flow matching scheduler, or allow `set_shift()` etc. on the scheduler. This seems to be the key to getting good results with SD3 https://x.com/bfitzgerald242/status/1801018438120341911
ENH: allow get_dummies to accept dtype argument
- [x] closes #18330 (there's no issue for this one)
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Update in version 0.19.0 made `get_dummies` return uint8 values instead of floats (#8725). While I agree with the argument that `get_dummies` should output integers __by default__ (to save some memory), in many cases it would be beneficial for user to choose other dtype.

In my case there was serious performance degradation between versions 0.18 and 0.19. After investigation, reason behind it turned out to be the change to `get_dummies` output type. DataFrame with dummy values was used as an argument to np.dot in an optimization function (second argument was matrix of floats). Since there were lots of iterations involved, and on each iteration np.dot was converting all uint8 values to float64, conversion overhead took unreasonably long time. It is possible to work around this issue by converting dummy columns "manually" afterwards, but it adds unnecessary complexity to the code and is clearly less convenient than calling `get_dummies` with `dtype=float`.

Apart from performance considerations, I can imagine `dtype=bool` to be a common use case.

`get_dummies(data, dtype=None)` is allowed and will return uint8 values to match the [DataFrame](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) interface (where None allows inferring datatype, which is default behavior).

I've extended the test suite to run all the `get_dummies` tests (except for those that don't deal with internal dtypes, like `test_just_na`) twice, once with `uint8` and once with `float64`.
Implement safetensors checkpoint loading for Trainer
### Feature request

At the moment, Trainer loads models with `torch.load` method directly onto the cpu:

(`Trainer._load_from_checkpoint` method)
```python
                ... 
                # We load the model state dict on the CPU to avoid an OOM error.
                state_dict = torch.load(os.path.join(resume_from_checkpoint, WEIGHTS_NAME), map_location="cpu")
                # workaround for FSDP bug https://github.com/pytorch/pytorch/issues/82963
                # which takes *args instead of **kwargs
                load_result = model.load_state_dict(state_dict, False)
                # release memory
                del state_dict
                self._issue_warnings_after_load(load_result)
                ...
```

Loading on cpu with safetensors is a lot faster, so this method should (?) be preffered if safetensors library is installed.

### Motivation

I care to speed up checkpointing process since I use it to store the best model checkpoint every time the metric is improved (via callback).

### Your contribution

The change should be straightforward, but I am not sure if safetensors checkpointing should be defaulted to if safetensors are installed, or configured manually like in `PreTrainedModel.from_pretrained` method.
DOC: Improve documentation df.interpolate() for methods   ‘time’, ‘index’ and ‘values’
### Pandas version checks

- [X] I have checked that the issue still exists on the latest versions of the docs on `main` [here](https://pandas.pydata.org/docs/dev/)


### Location of the documentation

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.interpolate.html

### Documentation problem

It is not possible to understand what exactly the method `interpolate` does from reading the documentation. See e.g. this SE post for more details

https://stackoverflow.com/questions/65511992/pandas-interpolation-type-when-method-index

### Suggested fix for documentation

Rewrite doctstring and documentation page for the method
NanoBeirEvaluator takes in empty dataset
I noticed that NanoBeir takes in an empty dataset and returns no error during instantiation. But when the model gets passed, the error kinda seems confusing.
So -
1. is it fine to allow the evaluator to take in an empty dataset.
2. Should we change the error message saying "invalid datset or something"

Who can help?
@tomaarsen 

Snippet reproduction
```
from sentence_transformers import SentenceTransformer
from sentence_transformers.evaluation import NanoBEIREvaluator

datasets = []

#model = SentenceTransformer("sentence-transformers-testing/stsb-bert-tiny-safetensors")

evaluator = NanoBEIREvaluator( # this bit here returns no error
    dataset_names=datasets,
)

results = evaluator(model) #this raised an error
```

#####################################################################

Error log
```
{
	"name": "KeyError",
	"message": "'cosine_ndcg@10'",
	"stack": "---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[4], line 1
----> 1 evaluator(model)

sentence_transformers\\evaluation\\NanoBEIREvaluator.py:351, in NanoBEIREvaluator.__call__(self, model, output_path, epoch, steps, *args, **kwargs)
    348 if not self.primary_metric:
    349     if self.main_score_function is None:
    350         score_function = max(
--> 351             [(name, agg_results[f\"{name}_ndcg@{max(self.ndcg_at_k)}\"]) for name in self.score_function_names],
    352             key=lambda x: x[1],
    353         )[0]
    354         self.primary_metric = f\"{score_function}_ndcg@{max(self.ndcg_at_k)}\"
    355     else:

sentence-transformers\\sentence_transformers\\evaluation\\NanoBEIREvaluator.py:351, in <listcomp>(.0)
    348 if not self.primary_metric:
    349     if self.main_score_function is None:
    350         score_function = max(
--> 351             [(name, agg_results[f\"{name}_ndcg@{max(self.ndcg_at_k)}\"]) for name in self.score_function_names],
    352             key=lambda x: x[1],
    353         )[0]
    354         self.primary_metric = f\"{score_function}_ndcg@{max(self.ndcg_at_k)}\"
    355     else:

KeyError: 'cosine_ndcg@10'"
}
```
Whisper Prompting
### Feature request

Add prompting for the Whisper model to control the style/formatting of the generated text.

### Motivation

During training, Whisper can be fed a "previous context window" to condition on longer passages of text.

The original OpenAI Whisper implementation provides the user with the option of passing an [`initial_prompt`](https://github.com/openai/whisper/blob/6dea21fd7f7253bfe450f1e2512a0fe47ee2d258/whisper/transcribe.py#L96) to the model. This prompt is replaces the "previous context window" during inference.

By passing the prompt as the "previous context window", the Whisper model conditions its generation on whatever text is passed as the prompt. This allows the user to control aspects of the generation, such as spellings of named entities and punctuation formatting (see https://github.com/openai/whisper/discussions/963#discussioncomment-4987057).

This is possibly a cheaper way of adapting the Whisper model to specific decoding constraints than fine-tuning.

This notebook demonstrates prompting with the initial codebase, and explains how this can be achieved for HF's Whisper: https://colab.research.google.com/drive/14FSeaoRvgs5arOTfiMQBnQ5NaLyma7Tq?usp=sharing

The proposed API for prompting would look something as follows:
1. Encode prompt text to prompt token ids (`processor.get_prompt_ids`) - this method is a wrapper around `processor.tokenizer.__call__` that **doesn't** add the special token ids:
```python
prompt = "IR, Newswire"
prompt_ids = processor.get_prompt_ids(prompt)
```
2. Pass the input audio and prompt token ids to the `.generate` method to get the predicted ids:
```python
pred_ids = model.generate(input_features, prompt_ids=prompt_ids)
```
3. Decode the predicted ids and 'slice' off the prompt (we can do this by passing the `prompt_ids`):
```python
pred_str = processor.batch_decode(pred_ids, prompt_ids=prompt_ids)
```

=> We would need to wrap all of this `forced_decoder_ids` logic into the generate method and update the processor/tokenizer accordingly.

### Your contribution

Happy to guide the integration and review any PRs!
Privilege escalation in @sharing endpoint (PloneHotfix20200121)
- https://github.com/plone/Products.CMFPlone/issues/3021
- https://plone.org/security/hotfix/20200121/privilege-escalation-when-plone-restapi-is-installed

`plone.restapi.deserializer.local_roles.DeserializeFromJson` has a weakness.
This endpoint was introduced in plone.restapi 1.0a10 (2017-03-22).

Please apply the patch from `restapi_local_roles.py` from the hotfix. Should be this, but please check:

```
--- a/PloneHotfixNext/Products/PloneHotfixNext/restapi_local_roles.py
+++ b/PloneHotfixNext/Products/PloneHotfixNext/restapi_local_roles.py
@@ -36,11 +36,17 @@ if DeserializeFromJson is not None:
         # roles
         roles_reindex = False
         new_roles = data.get("entries", None)
+        managed_roles = frozenset([r['id'] for r in sharing_view.roles()])
+
         if new_roles is not None:
             # the roles are converted into a FrozenSet so we have to filter
             # the data structure we get.
             for user in new_roles:
                 roles_list = [key for key in user["roles"] if user["roles"][key]]
+
+                # Limit roles to ones the user is allowed to delegate
+                roles_list = set(roles_list).intersection(managed_roles)
+
                 user["roles"] = roles_list
             roles_reindex = sharing_view.update_role_settings(new_roles, reindex=False)
```

@lukasgraf since you and Niklaus discovered this, could you take care of this?
[Bug]:  Support Falcon Mamba 
### Your current environment

Does VLLM support  Falcon Mamba models? if not, when it will be supported 

### 🐛 Describe the bug

Does VLLM support  Falcon Mamba models? if not, when it will be supported
Support for `annotated_types.Not`
### Initial Checks

- [X] I have searched Google & GitHub for similar requests and couldn't find anything
- [X] I have read and followed [the docs](https://docs.pydantic.dev) and still think this feature is missing

### Description

I need to use a `PositiveOddInt` type. Such a type can be generalized to `PositiveInt` and `NotMultipleOf(2)`.
json-schema for `NotMultipleOf(2)` would look like `"not": {"multipleOf": 2}`

Adding support for `annotated_types.Not` would make possible something like this:
```python
class PositiveOddIntRootModel(pydantic.RootModel):
    root: Annotated[int, annotated_types.Not(annotated_types.MultipleOf(2)), annotated_types.Ge(0)]
```

I have a separate implementation of NotMultipleOf and PositiveOddInt [here](https://github.com/h2020charisma/ramanchada2/blob/main/src/ramanchada2/misc/types/positive_not_multiple.py) but it uses pydantic1. Now as i am trying to migrate to pydantic2, [here](https://docs.pydantic.dev/latest/concepts/types/#customizing_validation_with_get_pydantic_core_schema) i have seen that future changes are possible and that annotated-types and native types are preferred. Would it be possible to add support for `annotated_types.Not` or could you suggest an alternative?

### Affected Components

- [ ] [Compatibility between releases](https://docs.pydantic.dev/changelog/)
- [X] [Data validation/parsing](https://docs.pydantic.dev/concepts/models/#basic-model-usage)
- [X] [Data serialization](https://docs.pydantic.dev/concepts/serialization/) - `.model_dump()` and `.model_dump_json()`
- [X] [JSON Schema](https://docs.pydantic.dev/concepts/json_schema/)
- [ ] [Dataclasses](https://docs.pydantic.dev/concepts/dataclasses/)
- [ ] [Model Config](https://docs.pydantic.dev/concepts/config/)
- [X] [Field Types](https://docs.pydantic.dev/api/types/) - adding or changing a particular data type
- [ ] [Function validation decorator](https://docs.pydantic.dev/concepts/validation_decorator/)
- [ ] [Generic Models](https://docs.pydantic.dev/concepts/models/#generic-models)
- [ ] [Other Model behaviour](https://docs.pydantic.dev/concepts/models/) - `model_construct()`, pickling, private attributes, ORM mode
- [ ] [Plugins](https://docs.pydantic.dev/) and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.
Crash when filtering empty detections: xyxy shape (0, 0, 4).
Reproduction code:

```python
import supervision as sv
import numpy as np

CLASSES = [0, 1, 2]

prediction = sv.Detections.empty()
prediction = prediction[np.isin(prediction["class_name"], CLASSES)]
```

Error:

```
Traceback (most recent call last):
  File "/Users/linasko/.settler_workspace/pr/supervision-fresh/run_detections.py", line 7, in <module>
    prediction = prediction[np.isin(prediction["class_name"], CLASSES)]
  File "/Users/linasko/.settler_workspace/pr/supervision-fresh/supervision/detection/core.py", line 1206, in __getitem__
    return Detections(
  File "<string>", line 10, in __init__
  File "/Users/linasko/.settler_workspace/pr/supervision-fresh/supervision/detection/core.py", line 144, in __post_init__
    validate_detections_fields(
  File "/Users/linasko/.settler_workspace/pr/supervision-fresh/supervision/validators/__init__.py", line 120, in validate_detections_fields
    validate_xyxy(xyxy)
  File "/Users/linasko/.settler_workspace/pr/supervision-fresh/supervision/validators/__init__.py", line 11, in validate_xyxy
    raise ValueError(
ValueError: xyxy must be a 2D np.ndarray with shape (_, 4), but got shape (0, 0, 4)
```
replace deprecated appdirs dependency with platformdirs
### Search before asking

- [X] I searched the [issues](https://github.com/sqlfluff/sqlfluff/issues) and found no similar issues.


### Description

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068011

python3-appdirs is dead upstream[1] and its Debian maintainer has indicated
that it should not be included in trixie[2]. A recommended replacement is
python3-platformdirs[3], which is a fork of appdirs with a very similar API.

Please migrate from appdirs to platformdirs or some other replacement,
so that appdirs can be removed.

### Use case

n/a

### Dialect

none

### Are you willing to work on and submit a PR to address the issue?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md)
Weighted SSL model not unfreezable
### Describe the bug

In our HF Weighted SSL model implementation, we `detach()` the hidden states, meaning weights are not updated.

Relevant code:
https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/lobes/models/huggingface_transformers/weighted_ssl.py#L81

```
hidden_states = torch.stack(feats.hidden_states, dim=0).detach()
```

### Expected behaviour

If passing `freeze=False` I'd expect the weights to get updated.

### To Reproduce

_No response_

### Environment Details

_No response_

### Relevant Log Output

_No response_

### Additional Context

_No response_
[1.12 regression] New error on type[memoryView] | type[str]
**Bug Report**

new to 1.12.0
incorrect error message on union of two type one being memoryview

Complete file checked:

"""
a_type: type[memoryview] | type[str] = memoryview
"""

mypy 1.11.2 no error

mypy 1.12.0
Incompatible default for argument "a_type" (default has type "type[memoryview[Any]]", argument has type "type[memoryview[int]] | type[str]")

**Your Environment**
github action ubuntu various , python various

- Mypy command-line flags:
-no flags
- Mypy configuration options from `mypy.ini` (and other config files):
- None
- Python version used: via github actions
3.8, 3.9, 3.10, 3.11, 3.12 and 3.13 

while
"""
a_type: type[memoryview] = memoryview
""""
Works ok!
Local Alertmanager
**Description:**
As our monitoring capabilities enhance, we need a method to alert on logs, or metrics we are tracking. In this ticket we want to integrate [Alertmanager](https://prometheus.io/docs/alerting/latest/overview/) with our PLG stack to enable fine grained alerting capabilities. As an MVP implementation, we would like to integrate SendGrid with Alertmanager to notify the appropriate parties of some issue. The goal is to implement Prometheus [recording rules ](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and (or) [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) for some metric that we are currently tracking. When the said metrics breaches some threshold or value, Prometheus should fire an alert to Alertmanager who then routes it to the correct receiver. 


**Acceptance Criteria:** 
_Create a list of functional outcomes that must be achieved to complete this issue_

- [ ] Prometheus connects to Alertmanager
- [ ] Prometheus sends alerts to Alertmanager
- [ ] Alertmanager integrated with SendGrid
- [ ] Alert emails are received from Alertmanager/SendGrid
- [ ] Documentation updated indicating Alertmanager integration
- [ ] README is updated, if necessary

**Tasks:**
_Create a list of granular, specific work items that must be completed to deliver the desired outcomes of this issue_ 

- [x] Add or create Prometheus rules for a chosen metric or metrics alerting
- [x] Add Alertmanager to compose file 
- [x] [Configure](https://prometheus.io/docs/alerting/latest/configuration/#email_config) Alertmanager with Sendgrid
- [x] Generate a test to fire the chosen alert(s)
- [x] Run Testing Checklist and confirm all tests pass

**Notes:**
_Add additional useful information, such as related issues and functionality that isn't covered by this specific issue, and other considerations that will be helpful for anyone reading this_ 

N/A

**Supporting Documentation:**
_Please include any relevant log snippets/files/screen shots_

- [Alertmanager Docs](https://prometheus.io/docs/alerting/latest/configuration/)
- [Prometheus Docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/)

**Open Questions:**
_Please include any questions or decisions that must be made before beginning work or to confidently call this issue complete_ 

- _Open Question 1_
- _Open Question 2_
Add EKS 1.31 as a supported version
**Describe the issue**
EKS 1.31 has been released.  However `CKV_AWS_339` fails as this is not listed as a supported version.

**Examples**
```
resource "aws_eks_cluster" "eks_cluster" {
  ...
  version                   = "1.31"
```
**Version (please complete the following information):**
 - Checkov Version 3.2.256 (latest)

**Additional context**
https://github.com/bridgecrewio/checkov/blob/main/checkov/terraform/checks/resource/aws/EKSPlatformVersion.py seems to be where the fix is needed
DOC: Document merge_cells="columns" in to_excel
https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.to_excel.html

The `merge_cells` argument can also take `"columns"` due to #35384. This should be added to the docstring.
MAINT conversion old->new/new->old tags
Towards #30298 

This PR provides a way to convert old tags into new tags and new tags into old tags.
We should make sure that:

- An estimator implementing `_get_tags` or `_more_tags` get a warning but that the check_estimator should be working reasonably
- `_safe_tags` should raise a deprecation warning
- we should make sure that `get_tags` can get old tags and convert it to new tags. The tricky part is to detect whether `__sklearn_tags__` if present is implemented by the `BaseEstimator` from scikit-learn or the child class. If this is only the `BaseEstimator`, we should raise a warning to move to the new API and we should temporary use the `_get_tags`/`_safe_tags`, otherwise we are fine.
At some places we're updating just one key of an object or one element of an array but we send the whole object to MongoDB to update which is inefficient.