Talip7/scikit-learn-issues-multilabel huggingface dataset:
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
@NamanAgarwal0905 thank you so much for your test. The negative offset still happens. This is your result.

<img width="1604" height="723" alt="Image" src="https://github.com/user-attachments/assets/7033543d-b4e4-44aa-bcc6-5f9186d10bcd" />

But I'm a fool. I forgot to remove "Close", "High", "Low", "Open", "Volume" from the df_combined. That means that the scikit has this data!

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
@NamanAgarwal0905 I've removed the original columns "Close", "High", "Low", "Open", "Volume" (in the engineer_features()) but the issue still exists. Here is your modified version. I had to comment some traces to make it work.
[test_3.py](https://github.com/user-attachments/files/22116702/test_3.py)

<img width="1585" height="830" alt="Image" src="https://github.com/user-attachments/assets/29b721cc-6067-4047-92bd-31696e22e83d" />

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
df['close_return'] didnt have shift(1). Here is a fixed version but the issue is still there:
[test4.py](https://github.com/user-attachments/files/22116896/test4.py)

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
Thanks for reporting the issue. Could you please also include a [minimal and reproducible example](http://sscce.org/) for us to better diagnose the problem?

Ideally we should be able to copy/paste your code to reproduce the issue.

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
As a result you will see this picture:

<img width="1642" height="696" alt="Image" src="https://github.com/user-attachments/assets/6c8e8a04-bb7b-4e80-9812-1ef33ca7333d" />

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
Hi, just another finding. If I set estimators=1 I'll get flat values. [Line 79](https://github.com/scikit-learn/scikit-learn/issues/32062#issuecomment-3333078724). Maybe it can spot some light. Thank you for the further investigation.

<img width="1649" height="869" alt="Image" src="https://github.com/user-attachments/assets/0ed77840-fc70-4435-9aa5-c1d2b709c581" />

<img width="1395" height="811" alt="Image" src="https://github.com/user-attachments/assets/512fa9b9-6b3f-49c8-8fc7-239ddec06b87" />

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
And another one finding. If I set `criterion='absolute_error'` there will be no Y offset. [Line 79](https://github.com/scikit-learn/scikit-learn/issues/32062#issuecomment-3333078724).
@adrinjalali for vis.

<img width="1580" height="739" alt="Image" src="https://github.com/user-attachments/assets/bae83b79-eea8-4f8d-ab86-07f736b631f1" />

<img width="1404" height="718" alt="Image" src="https://github.com/user-attachments/assets/a379a5f2-c35f-4970-8240-9acfff3674f2" />

Labels: Bug, Needs Investigation
Regressor Prediction Makes a Negative Y Offset

### Describe the bug

Hi, I've found a strange situation where regressor prediction makes a negative Y offset. See an orange line on my picture below.
Here is my py file and json data:
[test_scikit.zip](https://github.com/user-attachments/files/22069020/test_scikit.zip)

You will need to change JS_PATH  to your path:
JS_PATH = "D:/Projects/Crpt/CryptoMaiden/Bot/Base/Test/btc_data.json"

You will also need to install a poltly lib.

<img width="1625" height="921" alt="Image" src="https://github.com/user-attachments/assets/e8ac2c73-c556-4570-be28-6bfaedd7ba82" />

I'm new to the Scikit so I decided to report the issue.

I tried RandomForestRegressor, AdaBoostRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor and all of them have this issue. 
To change a regressor in my code you can just comment/uncomment it on lines 89-95.

### Steps/Code to Reproduce

# See my attached ZIP file!

### Expected Results

No negative Y offset.

### Actual Results

Just run my py file!

### Versions

```shell
1.7.1
```

COMMENT:
It seems this offset happens when a price goes higher then all trained prices.
@adrinjalali for vis

<img width="1614" height="765" alt="Image" src="https://github.com/user-attachments/assets/9ec222b7-6354-4036-b5e3-b48a643da31b" />

Labels: Bug, Needs Investigation
The dcg_score and ndcg_score documentation are hard to understand

### Describe the issue linked to the documentation

The documentation for the `dcg_score` and `ndcg_score` leave much to be desired.

I believe this is also a by-product of competing definitions of the discount cumulative gains (DCG) and normalised DCG (nDCG) in literature. Namely

$$\mathrm{DCG_{p}} = \sum_{i=1}^{p} \frac{rel_{i}}{\log_{2}(i+1)}$$

and

$$\mathrm{DCG_{p}} = \sum_{i=1}^{p} \frac{ 2^{rel_{i}} - 1 }{ \log_{2}(i+1)}.$$


The `dcg_score` uses the former definition, I do not think this is very clear.

The description for the DCG score (`dcg_score`) says "Sum the true scores ranked in the order induced by the predicted scores, after applying a logarithmic discount". While this is technically correct to what `dcg_score` does, like many maths equations, it is hard to understand without using maths notation.

If a user wants to clarify the exact equation of the DCG used past the description they might go to the references, however,

- the first reference is the Wikipedia which offers both definitions;

- the third reference, "Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th Annual Conference on Learning Theory (COLT 2013)", defines the discount function an a much more general way. While interesting does not give the user any insight into how the `dcg_score` is actually implemented.

My criticism of the `ndcg_score` is the same.

### Suggest a potential alternative/fix

I would propose giving an explicit definition of the DCG along the lines of

$$\mathrm{DCG_{k}} = \sum_{i=1}^{k} \frac{rel_{i}}{\log_{2}(i+1)}$$

where each $rel_i$ is the true score ranked in the order induced by the predicted scores.

Also, to do something similar for the `ndcg_score`.

COMMENT:
/take

Labels: Documentation, module:metrics
Leiden Clustering

### Describe the workflow you want to enable

The "Leiden" Clustering algorithm is considered one of the most powerful clustering algorithms, often outperforming competitors by a wide margin. 
The algorithm fulfils the inclusion criteria: its now 6 years old, has some 5200 citations. 

Currently, it is implemented in scanpy and cugraph where the latter includes a fast, gpu-enabled implementation. Due to its empirical performance, inclusion in scikit-learn would be a welcome addition for practitioners as it is vastly superior to most clustering algorithms currently included in scikit learn (on non-trivial datasets).

### Describe your proposed solution

I propose to include the Leiden algorithm as a clustering algorithm in scikit-learn.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

[From Louvain to Leiden: guaranteeing well-connected communities](https://www.nature.com/articles/s41598-019-41695-z)

COMMENT:
Hello,

I'm very interested in taking on the implementation of the Leiden algorithm for scikit-learn. I have a good understanding of the algorithm and am confident I can implement it to fit the scikit-learn API.

Before I begin coding, I'd like to ask the maintainers for their feedback and approval on this feature proposal. Does this align with the project's current roadmap? Are there any specific design considerations or potential dependency issues I should keep in mind?

I'm looking forward to contributing!

Labels: New Feature, Needs Decision - Include Feature
Leiden Clustering

### Describe the workflow you want to enable

The "Leiden" Clustering algorithm is considered one of the most powerful clustering algorithms, often outperforming competitors by a wide margin. 
The algorithm fulfils the inclusion criteria: its now 6 years old, has some 5200 citations. 

Currently, it is implemented in scanpy and cugraph where the latter includes a fast, gpu-enabled implementation. Due to its empirical performance, inclusion in scikit-learn would be a welcome addition for practitioners as it is vastly superior to most clustering algorithms currently included in scikit learn (on non-trivial datasets).

### Describe your proposed solution

I propose to include the Leiden algorithm as a clustering algorithm in scikit-learn.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

[From Louvain to Leiden: guaranteeing well-connected communities](https://www.nature.com/articles/s41598-019-41695-z)

COMMENT:
Thanks for opening this issue. From a quick look at the paper, it seems that this clustering algorithm is primarily meant for graphs where each node is connected to a few other nodes.

In the context of scikit-learn clustering, the graph structure is not already known a priori: nodes are typically described by feature vectors (rows in `X`) on which it is possible to define distances and then clustering algorithm typically perform distance-based decisions to tell if two records/nodes below to the same cluster or not.

Some clustering algorithms in scikit-learn (e.g. `sklearn.cluster.AgglomerativeClustering`) do take an optional graph connectivity structure as an extra constraint, but it's rather an exception than a rule. Other algorithms accept a (potentially truncated) precomputed affinity matrix, but they also accept working on raw feature vectors.

My questions are:

- how would you fit the Louvain clustering algorithm to the scikit-learn API?
- do you consider that each row in `X` is a node connected to all other nodes with a strength that relates to the inverse of the distances?
- does the Leiden clustering paper propose a standard way to do that? (I ctrl-f'ed the world "distance" and could not find any occurence).
- wouldn't a graph processing library such as networkx better suited for this algorithm>

>  it is vastly superior to most clustering algorithms currently included in scikit learn (on non-trivial datasets)

How do you compare clustering results for the applications you have in mind? In particular, do you have references that compare the results of clustering algorithm implemented in scikit-learn to Leiden clustering?

Labels: New Feature, Needs Decision - Include Feature
Leiden Clustering

### Describe the workflow you want to enable

The "Leiden" Clustering algorithm is considered one of the most powerful clustering algorithms, often outperforming competitors by a wide margin. 
The algorithm fulfils the inclusion criteria: its now 6 years old, has some 5200 citations. 

Currently, it is implemented in scanpy and cugraph where the latter includes a fast, gpu-enabled implementation. Due to its empirical performance, inclusion in scikit-learn would be a welcome addition for practitioners as it is vastly superior to most clustering algorithms currently included in scikit learn (on non-trivial datasets).

### Describe your proposed solution

I propose to include the Leiden algorithm as a clustering algorithm in scikit-learn.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

[From Louvain to Leiden: guaranteeing well-connected communities](https://www.nature.com/articles/s41598-019-41695-z)

COMMENT:
Thanks @martinalex000 for the follow-up.

> From my experience, the combination of the umap-neighborhood-graph and leiden-modularity-clustering really works well across datasets and would be a welcome addition to the set of clustering algorithms.

I am not familiar with the details of the umap-neighborhood-graph method. Would the pipeline you propose be similar to what we do with precomputed kNN distance graphs done in this example:

https://scikit-learn.org/stable/auto_examples/neighbors/approximate_nearest_neighbors.html

> In our experiments, we used ARI to compare clustering quality, see top-left corner.

Out of curiosity, do you have public Python code to reproduce the results of that figure?

Labels: New Feature, Needs Decision - Include Feature
Leiden Clustering

### Describe the workflow you want to enable

The "Leiden" Clustering algorithm is considered one of the most powerful clustering algorithms, often outperforming competitors by a wide margin. 
The algorithm fulfils the inclusion criteria: its now 6 years old, has some 5200 citations. 

Currently, it is implemented in scanpy and cugraph where the latter includes a fast, gpu-enabled implementation. Due to its empirical performance, inclusion in scikit-learn would be a welcome addition for practitioners as it is vastly superior to most clustering algorithms currently included in scikit learn (on non-trivial datasets).

### Describe your proposed solution

I propose to include the Leiden algorithm as a clustering algorithm in scikit-learn.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

[From Louvain to Leiden: guaranteeing well-connected communities](https://www.nature.com/articles/s41598-019-41695-z)

COMMENT:
Follow-up: I found the `Leiden` class in https://anonymous.4open.science/r/tneb_clustering-4056/src/corc/our_algorithms.py that preprocesses data using [scanpy.pp.neighbors](https://scanpy.readthedocs.io/en/stable/api/generated/scanpy.pp.neighbors.html) and as a result extract a k=15 neighbors graph with UMAP connectivities.

However, it's not clear to me what the input data passed to `Leiden.fit` looks like: it's called `counts` (suggesting integer data) and converted to a `scipy.sparse` matrix. It's doubly weird because I would have expected dense floating point features at this point.

Furthermore, I have no ideas what UMAP connectivies are from a mathematical point but they do seem to be automatically picked up by the subsequent [scanpy.tl.leiden](https://scanpy.readthedocs.io/en/stable/generated/scanpy.tl.leiden.html) step.

Labels: New Feature, Needs Decision - Include Feature
Leiden Clustering

### Describe the workflow you want to enable

The "Leiden" Clustering algorithm is considered one of the most powerful clustering algorithms, often outperforming competitors by a wide margin. 
The algorithm fulfils the inclusion criteria: its now 6 years old, has some 5200 citations. 

Currently, it is implemented in scanpy and cugraph where the latter includes a fast, gpu-enabled implementation. Due to its empirical performance, inclusion in scikit-learn would be a welcome addition for practitioners as it is vastly superior to most clustering algorithms currently included in scikit learn (on non-trivial datasets).

### Describe your proposed solution

I propose to include the Leiden algorithm as a clustering algorithm in scikit-learn.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

[From Louvain to Leiden: guaranteeing well-connected communities](https://www.nature.com/articles/s41598-019-41695-z)

COMMENT:
Just as a quick follow-up: is there anything missing that would help with the decision? As I said earlier, I consider Leiden the best well-established clustering algorithms out there and would like to see it being included in scikit-learn.

Labels: New Feature, Needs Decision - Include Feature