mteb/LocBenchRR huggingface dataset:
Improve how normalize_chunks selects chunk sizes if auto is given
Specifying auto for a dimension when new chunks are selected like in rechunking allows us to choose a chunk size that we think is a good fit and efficient. 

This is for now an issue to collect thoughts from others and feedback, I am pretty sure that I haven't found all suboptimal corner cases yet.


Assume we have the following array:

```
import dask.array as da

arr = da.random.random((5000, 300, 372), chunks=(600, 150, 172))
```

Triggering a Rechunk operation to -1 and auto for dimensions 2 and 3 results in the following chunks:

```
x = arr.rechunk((-1, "auto", "auto"))
```

<img width="747" alt="Screenshot 2024-08-20 at 15 34 19" src="https://github.com/user-attachments/assets/31e5c070-fc01-40be-af63-21fa06e457f5">

```normalize_chunks`` targets the 128MiB that is the default for auto pretty aggressively and creates a significantly more complex task graph than necessary if we would compromise a little bit on the in memory size of the chunks.

Choosing 50 and (58, 57, 57) per input chunk would allows us to split every chunk into 3 pieces along every axis and thus avoid including the neighbouring chunks alongside this axis.

I am pretty sure that there are better examples where this makes things very complex, but this should illustrate the general issue.

The proposal is to compromise a little bit on the chunk size side of things to create a simpler topology if possible.
Mark ES-MDA as recommended
The ES-MDA algorithm is the recommended algorithm for multi iteration use. We should mark it as such. This should not affect the logging and usage statistics of methods.
`dvc exp remove --queue -A`: possibly incomplete result returned
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

As discussed in https://github.com/iterative/dvc/pull/10633#discussion_r1857943029, 
when using `dvc exp remove` with both `--queue` and `-A `, an incomplete list of removed experiments might be returned: the queued ones would be missing from the returned list, even if they are effectively removed as expected.

This is just a minor inconvenience at the moment, but for the sake of correctness and code cleanliness, I'll put it here.

### Reproduce

1. create repo
2. run experiments 
3. queue experiments (`dvc exp run --queue`)
4. `dvc exp remove --queue -A`


### Expected
The list of all experiments, both queued and committed
**Additional Information (if any):**

https://github.com/iterative/dvc/blob/d38b2dbcb873b5112976c5ad40c5574b5d2a41f3/dvc/repo/experiments/remove.py#L69-L71

line 71 should use `removed.extend` just as it is done in the other if / elif code blocks

I can start working on it soon.
Support `datetime.(date|datetime)` in `Expression`(s)
### What is your suggestion?

- Target expressions [datetime](https://vega.github.io/vega/docs/expressions/#datetime), [utc](https://vega.github.io/vega/docs/expressions/#utc)

**See #3650 for detail**
No need to need benchmarks for different values of length for generate_rpacket_last_interaction_tracker_list.
@officialasishkumar Currently the said benchmark runs for different values of length, which is not required.
The related code:
https://github.com/tardis-sn/tardis/blob/e1aa88723a6836e8a25cc1afb24b578b1b78651f/benchmarks/transport_montecarlo_packet_trackers.py#L33-L38

My question, was this intended? if not can you please remove that since it can increase the runtime of benchmark workflow.
Jump performance
Resolves [JP-3697](https://jira.stsci.edu/browse/JP-3697)

Closes #337

A number of changes to `twopoint_difference.py` and `jump.py` reduce memory usage by a factor of about 2 and improve runtime by a factor of anywhere from 3 to 20.  Tested output files (noted below) are unchanged, except for MIRI shower detection in `jw01701012001_02105_00001_mirifulong`.  This is due to a difference in replacing infilled saturated pixels after performing a convolution; the new version identifies fewer showers.  An example comment on this is here: 
https://jira.stsci.edu/browse/JP-3697?focusedId=760740&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-760740
Overflagging of jumps should no longer cause the pipeline to grind to a halt due to `np.where` combined with `for` loops over pixels; it should now have virtually no impact on performance.

Files tested, along with memory usage and runtimes in the jump step (all files are public except for program 5263):

| File                                                                   |  Old Runtime | Old Memory  |   New Runtime | New Memory |
| -------------------------------------------| --------| ---------| ----| -- |
| jw01668002001_04101_00001_mirimage   | 68 s | 27 GB | 15 s | 11 GB |
| jw01283001001_03101_00001_mirimage    | 420 s | 70 GB | 120s  | 30 GB |
| jw05263001001_05101_00001_nrs1             | 500 s | 40 GB | 30 s | 23 GB |
| jw02079004001_03201_00001_nrca1          | 35 s | 2 GB | 6 s | 2 GB |
| jw01701012001_02105_00001_mirifulong     | 13 s | 0.4 GB | 4 s | 0.4 GB |
| jw01366001001_04101_00001-seg001_nis | 155 s | 40 GB | 35 s | 23 GB |

A significant fraction of the line changes are due to fixing a mixture of 8-space indents and 4-space indents in `twopoint_difference.py`.  Most of these lines were not actually changed.


<!-- if you can't perform these tasks due to permissions, please ask a maintainer to do them -->
## Tasks
- [ ] update or add relevant tests
- [ ] update relevant docstrings and / or `docs/` page
- [ ] Does this PR change any API used downstream? (if not, label with `no-changelog-entry-needed`)
  - [ ] write news fragment(s) in `changes/`: `echo "changed something" > changes/<PR#>.<changetype>.rst` (see below for change types)
  - [ ] run regression tests with this branch installed (`"git+https://github.com/<fork>/stcal@<branch>"`)
    - [ ] [`jwst` regression test](https://github.com/spacetelescope/RegressionTests/actions/workflows/jwst.yml) 
    - [ ] [`romancal` regression test](https://github.com/spacetelescope/RegressionTests/actions/workflows/romancal.yml)

<details><summary>news fragment change types...</summary>

- ``changes/<PR#>.apichange.rst``: change to public API
- ``changes/<PR#>.bugfix.rst``: fixes an issue
- ``changes/<PR#>.general.rst``: infrastructure or miscellaneous change
</details>
Deprecate axis in GroupBy cumulative transformers
Deprecate axis in

- cumsum
- cumprod
- cumcount

and please skip tests that test this behaviour for dask-expr
[AGE-164] [Improvement] Speed up the endpoint /get_config
The call to retrieve the configuration takes 1.20 seconds, indicating that 'get_config' is highly unoptimised.

Here's the current process:

* It fetches the base_id.
* It checks permissions.
* It lists the environment for the app_id, which is obtained from the base_id.
* It searches one by one until it finds the correct name.
* It gets the variant_revision_id from the found environment.
* It returns the configuration from that variant.

This process involves a significant number of queries!

<sub>From [SyncLinear.com](https://synclinear.com) | [AGE-164](https://linear.app/agenta/issue/AGE-164/[improvement]-speed-up-the-endpoint-get-config)</sub>
`requires_package` is very slow
              It seems that `requires_package` is very slow:

![image](https://github.com/user-attachments/assets/186c6084-3765-4349-8205-81bdddae9c8e)

_Originally posted by @germa89 in https://github.com/ansys/pymapdl/issues/3703#issuecomment-2615615323_
(🐞) Wrong column number for `TypedDict.setdefault`
```py
from typing import TypedDict

class A(TypedDict):
    x: int
a: A
a.setdefault('x', '')  # test.py:6:1: error: Argument 2 to "setdefault" of "TypedDict" has incompatible type "str"; expected "int"
```
column 1?

Internally an error with the correct column number is generated, but is filtered out as a duplicate.
JAX indexing should support zero-dimensional boolean masks
NumPy supports this:

```python
>>> import numpy as np
>>> x = np.arange(3)
>>> m = np.array([], dtype=bool)
>>> x[m]
array([], dtype=int64)

>>> x = np.arange(12).reshape(3, 4)
>>> m = np.empty((3, 0), dtype=bool)
>>> x[m]
array([], dtype=int64)
```

JAX does not:
```python

>>> import jax.numpy as jnp
>>> x = jnp.arange(3)
>>> m = jnp.array([], dtype=bool)
>>> x[m]
IndexError: boolean index did not match shape of indexed array in index 0: got (0,), expected (3,)

>>> x = jnp.arange(12).reshape(3, 4)
>>> m = jnp.empty((3, 0), dtype=bool)
>>> x[m]
IndexError: boolean index did not match shape of indexed array in index 0: got (3, 0), expected (3, 4)
```
This behavior also appears to be required by the [Array API specification: Boolean Array Indexing](https://data-apis.org/array-api/latest/API_specification/indexing.html#boolean-array-indexing):

> - The size of each dimension in `B` must equal the size of the corresponding dimension in `A` or be `0`, beginning with the first dimension in `A`. If a dimension size does not equal the size of the corresponding dimension in `A` and is not `0`, then an `IndexError` exception must be raised.
Fix regex in finish_release.py
```
(venv) certbot [3.0.0] » python3 tools/finish_release.py 
certbot/tools/finish_release.py:114: SyntaxWarning: invalid escape sequence '\s'
  pattern = f'^\s+{channel}\s+{version}\s+(\d+)\s*'
certbot/tools/finish_release.py:114: SyntaxWarning: invalid escape sequence '\s'
  pattern = f'^\s+{channel}\s+{version}\s+(\d+)\s*'
certbot/tools/finish_release.py:114: SyntaxWarning: invalid escape sequence '\s'
  pattern = f'^\s+{channel}\s+{version}\s+(\d+)\s*'
```
The requirement of superuser postgresql access is problematic
## Problem
Mathesar needs a Postgres superuser to function correctly, from the docs at https://docs.mathesar.org/installation/build-from-source/ 

## Proposed solution
The mathesar user should not require superuser access. 

## Additional context
The superuser is a global permission meaning that a user that has superuser permission will be able to access (and do stuff) not on the mathesar but *all* the databases of the RDBMS. Considering that many production systems have a single RDBMS hosting many application this is a major problem since the mathsar user won't have any access boundaries on the same RDBMS. The mathesar user access can be unlimited but *must* be bounded without the mathesar database.
[Feature]: MLPSpeculator Tensor Parallel support
### 🚀 The feature, motivation and pitch

`MLPSpeculator`-based speculative decoding was recently added in https://github.com/vllm-project/vllm/pull/4947, but the initial integration only covers single GPU usage.

There will soon be "speculator" models available for larger target models that require multiple GPUs so we would like to ensure that TP can be used.

The first part of this issue would be testing it out in conjunction with https://github.com/vllm-project/vllm/pull/5414 and making necessary adjustments so that it will work with TP=1 for the speculator and TP=N for the target model.

Following this we can look at having the speculator itself run with TP>1, but that may be more involved since it will require some distributed coordination of the sampling of each speculated token in the MLPSpeculator loop. It might be possible to avoid additional communication here by the having the sampler used by the speculator model use a dedicated `torch.Generator` for its sampling and doing this sampling in tandem across the ranks.

@JRosenkranz already used `VocabParallelEmbedding` in the implementation so the model layers themselves should work fine.

cc @cadedaniel @sirejdua @JRosenkranz @tdoublep
Time issue when running `--test` on large microscopy images
**Disclaimer**
This issue is based on preliminary observations and does not contain a lot of details/examples at the moment.

## Issue description

I tried running the `--test` command on large microscopy images on `rosenberg`, and it takes a very long time.

I'll reference the following screenshot:
![test_on_om](https://user-images.githubusercontent.com/54086142/138940644-3f9618f9-d75e-41fb-9b3a-f10e8dc7a782.png)

From the `test` function called in main [here](https://github.com/ivadomed/ivadomed/blob/5207e7a5f0c882ae66248322ffbc518c0d2d6ce7/ivadomed/main.py#L470), the following happen:

At 19h20m56s, the segmentation of the first image starts (8557x7115 pixels).
At 19h29m59s, the segmentation of the second image starts  (5895x5266 pixels).
At 19h32m25s, the segmentations are completed, those timings are about normal (maybe a bit long, as describe in #972)

At this point, the test function is at line 85  in `testing.py` [here](https://github.com/ivadomed/ivadomed/blob/5207e7a5f0c882ae66248322ffbc518c0d2d6ce7/ivadomed/testing.py#L85), so it seems that the `metric_mgr.get_results()` function takes 40 minutes to compute the evaluation metrics (until 20h12m09s), which seems excessive.

After that, the `evaluate` function called in main [here](https://github.com/ivadomed/ivadomed/blob/5207e7a5f0c882ae66248322ffbc518c0d2d6ce7/ivadomed/main.py#L480) runs on the first image (8557x7115 pixels).
I killed the process after 2.5 hours without progress.
In that case, I suspect that the bottleneck may be in [run_eval](https://github.com/ivadomed/ivadomed/blob/5207e7a5f0c882ae66248322ffbc518c0d2d6ce7/ivadomed/evaluation.py#L87). 

Further investigations are needed.
I would be happy to make my model and images available if someone is interested to look into this.
Thanks!
`sqlglot.optimizer.qualify.qualify` fails when using `CONNECT BY` + `level` from a CTE
## Fully reproducible code snippet
In Snowflake, the `SELECT FROM <data_source> START WITH ... CONNECT BY` construct allows a `<level_expression>` by basically allowing to select a `level` column in addition to the existing columns:

```sql
SELECT
  employee_ID,
  manager_ID,
  title,
  level --<-- This isn't a column from "employees", but a column allowed by Snowflake in `CONNECT BY` constructs
FROM employees
START WITH title = 'President'
CONNECT BY manager_ID = PRIOR employee_id
ORDER BY
  employee_ID NULLS LAST
```

`sqlglot.optimizer.qualify.qualify` handles this fine ✅

However, if the `SELECT` is `FROM` a CTE, it fails!

Taking the same example as before **but** putting all the data in a CTE (instead of a table) breaks `qualify`:
```py
from sqlglot import parse_one
from sqlglot.optimizer.qualify import qualify

SQL_DIALECT = "snowflake"

print(qualify(parse_one("""
WITH employees AS (
    SELECT *
    FROM (VALUES ('President', 1, NULL),
                 ('Vice President Engineering', 10, 1),
                 ('Programmer', 100, 10),
                 ('QA Engineer', 101, 10),
                 ('Vice President HR', 20, 1),
                 ('Health Insurance Analyst', 200, 20)
    ) AS t(title, employee_ID, manager_ID)
)
SELECT
  employee_ID,
  manager_ID,
  title,
  level
FROM employees
START WITH title = 'President'
CONNECT BY manager_ID = PRIOR employee_id
ORDER BY
  employee_ID NULLS LAST
""", dialect="snowflake"), schema=None, dialect="snowflake").sql(pretty=True, dialect=SQL_DIALECT))
```

Output:
```
Traceback (most recent call last):
  File "test_sqlglot.py", line 7, in <module>
    print(qualify(parse_one("""
          ^^^^^^^^^^^^^^^^^^^^^
  File "/sqlglot/optimizer/qualify.py", line 102, in qualify
    validate_qualify_columns_func(expression)
  File "sqlglot/optimizer/qualify_columns.py", line 109, in validate_qualify_columns
    raise OptimizeError(f"Column '{column}' could not be resolved{for_table}")
sqlglot.errors.OptimizeError: Column '"LEVEL"' could not be resolved
```


## Official Documentation
(see [documentation](https://docs.snowflake.com/en/sql-reference/constructs/connect-by#examples)).
Crawl4AI tool has error
I tweaked example code from here:
https://docs.phidata.com/tools/crawl4ai
and used this code:
```
from phi.agent import Agent
from phi.model.openai import OpenAIChat
from phi.tools.crawl4ai_tools import Crawl4aiTools
from dotenv import load_dotenv

load_dotenv()

agent = Agent(
    model=OpenAIChat(id="gpt-4o"),
    tools=[Crawl4aiTools(max_length=None)],
    show_tool_calls=True
)
agent.print_response("Summarize me the key points of this: https://blog.google/products/gemini/google-gemini-deep-research/")
```
but I've got error:
```
(phidata-venv) PS D:\Projects\AI_testing\phidata> python .\crawl4ai_example.py
▰▰▱▱▱▱▱ Thinking...INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
WARNING  Could not run function web_crawler(url=https://blog.google/products/gemini/google-gemini-deep-research, max_length=500)
ERROR    'NoneType' object is not callable
         Traceback (most recent call last):
           File "D:\Projects\AI_testing\phidata\phidata-venv\Lib\site-packages\phi\tools\function.py", line 313, in execute
             self.result = self.function.entrypoint(**entrypoint_args, **self.arguments)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           File "D:\Projects\AI_testing\phidata\phidata-venv\Lib\site-packages\pydantic\_internal\_validate_call.py", line 38, in wrapper_function
             return wrapper(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
           File "D:\Projects\AI_testing\phidata\phidata-venv\Lib\site-packages\pydantic\_internal\_validate_call.py", line 111, in __call__
             res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           File "D:\Projects\AI_testing\phidata\phidata-venv\Lib\site-packages\phi\tools\crawl4ai_tools.py", line 35, in web_crawler
             crawler = WebCrawler(verbose=True)
                       ^^^^^^^^^^^^^^^^^^^^^^^^
         TypeError: 'NoneType' object is not callable
```

**phidata v2.7.2**
and
**crawl4ai v0.4.1**
is used.
Reduce PAPERLESS_TASK_WORKERS and  PAPERLESS_WEBSERVER_WORKERS default values
### Description

# Problem with memory and CPU usage
In connection with #1084, reducing the `PAPERLESS_TASK_WORKERS` and  `PAPERLESS_WEBSERVER_WORKERS` default values to `1` helps not only with reducing the memory footprint (from approx. 400mb to approx. 200mb) but because the Django-Q's internals, the CPU usage goes downs as well (each worker checks in every second, which creates a Redis hit; the more workers, the more check-ins, the more Redis hits, the higher the idle CPU usage).

# Typical paperless use-case
I argue paperless is mostly used by hobbyists to store simple invoices, notes, etc. However, by and large, the instance is idling rather than running multicore, CPU heavy processes (see typical use-cases on [/r/selfhosted](https://www.reddit.com/r/selfhosted/search/?q=paperless))

# Change the defaults to match the typical use-case
If the instance is mostly idling and every once in a while processing a new document, it is more reasonable than not to limit the default number of workers to `1`. This is still configurable via env variables, so if anyone ever needs more workers, they can change it, but most users, simply starting the docker container, would benefit from the reduced memory and CPU usage.

Any thoughts, objections?

@stumpylog , I'm happy to create a PR to change the defaults and update the [documentation](https://paperless-ngx.readthedocs.io/en/latest/setup.html?highlight=PAPERLESS_TASK_WORKERS#considerations-for-less-powerful-devices) as well if we are happy with reducing the defaults to preserve memory and CPU.





# Affected code

On any modern CPU (including slow, Atom CPUs) paperless creates [unnecessary amount of task workers based on core-count](https://github.com/paperless-ngx/paperless-ngx/blob/9a1bd9637cbef0e7318a3c61cae14805c6510cea/src/paperless/settings.py#L438) and [exactly 2 web workers](https://github.com/paperless-ngx/paperless-ngx/blob/9a1bd9637cbef0e7318a3c61cae14805c6510cea/gunicorn.conf.py#L4) for historical (?) reasons

### Webserver logs

_No response_

### Paperless-ngx version

1.7.1

### Host OS

Ubuntu

### Installation method

Docker

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_
Auto-importing models feature for shell-command
Description
	 
This would be an update of the existing Django shell that auto-imports models for you from your app/project. Also, the goal would be to allow the user to subclass this shell to customize its behavior and import extra things.
wiki
​proposal
Auto-importing models feature for shell-command
Description
	 
This would be an update of the existing Django shell that auto-imports models for you from your app/project. Also, the goal would be to allow the user to subclass this shell to customize its behavior and import extra things.
wiki
​proposal
DynamoDB batch retrieval does slow sequential gets (online_read)
The current DynamoDB implementation does sequential gets (https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/online_stores/dynamodb.py#L163)

## Possible Solution
A better approach is to do some multi-get operation or at least run these queries in parallel and collect the results.
ControlNet broken from_single_file
### Describe the bug

controlnet loader from_single_file was originally added via #4084
and method `ControlNet.from_single_file()` works for non-converted controlnets.

but for controlnets in safetensors format that contain already converted state_dict, it errors out.

its not reasonable to expect from user to know what is the internal dict structure of the controlnet safetensors file  
before he can use it.

even worse, some of the newer controlnets are distributed as single-file-only and are already in diffusers format  
which makes them impossible to load in difufsers.
for example: <https://huggingface.co/Laxhar/noob_openpose/tree/main>

this issue was already mentioned several times, each time closed as "works as designed"  
when in reality its just a failure that should be addressed as an issue.  
see #8474 #9208 #8614 as examples of previous issues

### Reproduction

scenario-1: works with non-converted controlnet
```python
import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='Aptronym/SDNext', filename='ControlNet11/controlnet11Models_canny.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
```

scenario-1: fails for majority of controlnets available on huggingface
```python
import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='lllyasviel/sd_control_collection', filename='diffusers_xl_canny_small.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
```
initial failure is nonsense
> OSError: stable-diffusion-v1-5/stable-diffusion-v1-5 does not appear to have a file named config.json.

whats making this worse is that SD15 and SDXL share the same `ControlNet` class which causes some
confusion on the base repo where to lookup config.
e.g,, here we're loading SDXL controlnet and error referrs to SD15 repo.

anyhow, trying to force correct config:
```py
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16, config='diffusers/controlnet-canny-sdxl-1.0-small')
```

results in even worse nonsense failure during loading of state_dict:
> TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not NoneType

### System Info

diffusers=0.32.0.dev0
python==3.12.3
torch==2.5.1+cu124

### Who can help?

@yiyixuxu @sayakpaul @DN6 @asomoza