Talip7/scikit-learn-issues-multilabel huggingface dataset:
HDBSCAN Ongoing Work

## Introduction

This is a (hopefully) exhaustive list of ongoing/future work for HDBSCAN. These have all been discussed and are considered wanted, but some still require thorough investigation (especially heuristic evaluations).

## Priority List

The higher priority items appear earlier in this list.

- [ ] https://github.com/scikit-learn/scikit-learn/pull/26888
    - [x] https://github.com/scikit-learn/scikit-learn/pull/25914
    - [ ] Finalize `{KD, Ball}Tree` API to avoid writing custom dispatcher
- [ ] Support `np.nan` in Cython implementation for sparse matrices
    - [x] https://github.com/scikit-learn/scikit-learn/pull/26889
- [ ] [Reintroduce `Boruvka` algorithm](https://github.com/scikit-learn/scikit-learn/pull/27572) (removed in https://github.com/scikit-learn/scikit-learn/commit/b7736ef6db1650ba4c3d8d830348aacfe5589015)
- [ ] Implement PWD backend for weighted `argkmin` in medoid calculation
- [ ] Investigate PWD backend for `mst_from_*` functions in `_linkage.pyx`
- [ ] Investigate PWD backend for `_reachability.pyx`
- [ ] Benchmark KD vs Ball Tree efficiency
- [ ] Add consistent threading semantics to enable `prange`, e.g. in `_reachability.pyx`
- [x] Improve partition strategy in `_reachability.pyx` (cf. https://github.com/scikit-learn/scikit-learn/pull/24701#discussion_r1035775220)
- [ ] Add support for `np.inf` values when `metric=='precomputed'` and `X` is sparse.

COMMENT:
@lmcinnes Thanks. That’s good to hear. It would then be nice if the hdbscan package pointed to scikit-learn‘s hdbscan.

Labels: New Feature, Moderate, module:cluster, cython
C++ back-end library for scikit-learn ?

### Describe the workflow you want to enable

Hi,

This question is related to the design of a clear separation of back-end computational code of algorithms and the front-end code of plotting/data access/doc/tests/benchmarks. 

The two environments are clearly not related. The back-end needs a lot of technicality/math/scientific knowledge while the front-end needs  a smart understanding of user activities (Culture, GUIs, UX, doc, support) etc.

Scikit-learn is, IMHO, one of the best implementations of machine learning software. It relies internally of best state of the art algorithms and includes some of their best implementations in C++ (libsvm etc) and has one the most fluent while quite simple APIs (fit/predict).

My question is about the existence of plans to implement a well defined C++ library implementing the back-end including the algorithms and the common API in a Domain Specific Language (DSL). 

This library can directly be used in all front-end activities, easily imported in other languages (think a version of scikit-learn for R, a blasphem ?), or other languages like Java, Rust, Perl, SQL, etc.

For python users, this separation makes it also possible to be sure, for example,  that no GIL is to be cared of in computational codes.

I know this is a wishful thinking, a dream. It can be technically complex, take months  for a team to put in place, but is also a common practice for software professionals where usually the back-end and front-end separation is not only natural but also sometimes, geographical, cultural and even legal.

### Describe your proposed solution

Implement a C++ back-end library for scikit-learn.


### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I don't think we're going to implement a C++ backend the way you explain. There are efforts to have a plugin API for alternate implementations of some algorithms though, e.g.: https://github.com/scikit-learn/scikit-learn/pull/25535

Our algorithms are not all in Cython / C / C++, and a significant amount is in pure python, and with recent efforts with dataframe API and array API, there will be even more focus on pure python implementations to use those different backends through the two APIs.

Labels: New Feature, Needs Triage
Backward compatibility issue for loading models with joblib in 1.3.0

### Describe the bug

Version 1.3.0 has broken backward compatibility for loading older models with `joblib`. 

### Steps/Code to Reproduce

## Step 1 : Save a model in 1.2.X
```python
import sklearn as sk
from sklearn.ensemble import RandomForestRegressor
from joblib import dump

print(sk.__version__) # e.g. 1.2.2

rf = RandomForestRegressor()
rf.fit([[1, 2], [3, 4]], [1, 2])

dump(rf, "rf-1.2.2.joblib", compress=3)
```
## Step 2 : Upgrade to 1.3.0 and try loading older model
```python
import sklearn as sk
from sklearn.ensemble import RandomForestRegressor
from joblib import load

print(sk.__version__) # 1.3.0

rf = load("rf-1.2.2.joblib")
```

### Expected Results

The model loads correctly.

### Actual Results

```python
  File "sklearn/tree/_tree.pyx", line 714, in sklearn.tree._tree.Tree.__setstate__
  File "sklearn/tree/_tree.pyx", line 1418, in sklearn.tree._tree._check_node_ndarray
ValueError: node array from the pickle has an incompatible dtype:
- expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}
- got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]
```

### Versions

```shell
sklearn : 1.2.2 & 1.3.0
joblib : 1.3.1
```

COMMENT:
This is expected behavior. We do not support loading models across versions.

Labels: Bug
Handle np.nan / missing values in KBinsDiscretizer

Especially when the output is one-hot encoded, it would be quite natural to encode missing values as zeros or add an extra missingness indicator feature.

Related to:

- https://github.com/scikit-learn/scikit-learn/issues/26793

COMMENT:
@ogrisel Do we want to handle this for one-hot encoding only or for ordinal as well? Since it is not totally clear how the missing values should be replaced for the ordinal case.

Labels: Enhancement
Handle np.nan / missing values in KBinsDiscretizer

Especially when the output is one-hot encoded, it would be quite natural to encode missing values as zeros or add an extra missingness indicator feature.

Related to:

- https://github.com/scikit-learn/scikit-learn/issues/26793

COMMENT:
Also just for reference I think there is an open PR which might be related to this: #17179

Labels: Enhancement
Handle np.nan / missing values in KBinsDiscretizer

Especially when the output is one-hot encoded, it would be quite natural to encode missing values as zeros or add an extra missingness indicator feature.

Related to:

- https://github.com/scikit-learn/scikit-learn/issues/26793

COMMENT:
I think `OrdinalEncoder` can handle `nan` values, maybe it could be used here

Labels: Enhancement
Handle np.nan / missing values in KBinsDiscretizer

Especially when the output is one-hot encoded, it would be quite natural to encode missing values as zeros or add an extra missingness indicator feature.

Related to:

- https://github.com/scikit-learn/scikit-learn/issues/26793

COMMENT:
I had a look at this issue times ago. I found that the issue could be somewhat tricky, it requires more specs for decision on the limit cases. You can have:

- no missing values in train data
- some missing values in train data
- all values are missing in train data

and then when transforming new data with the fitted transformer, you have to plan for the case where the data is missing, independently of how looked the training data. When to raise an error ? etc

I couldn't convince myself that there's a reasonable default and the most reasonable takes might not be the same depending on if you have ordinal output or one-hot encoding output. 

Ultimately I thought it would require a couple additional keywords to the kbinsdiscretizer api (and maybe deprecation cycle ?)

Labels: Enhancement
Handle np.nan / missing values in KBinsDiscretizer

Especially when the output is one-hot encoded, it would be quite natural to encode missing values as zeros or add an extra missingness indicator feature.

Related to:

- https://github.com/scikit-learn/scikit-learn/issues/26793

COMMENT:
It seems those questions have been answered already when the same work was done on [OrdinalEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html).

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
Note: we could also add `handle_missing="passthrough"` for consistency with other preprocessors that allow this. I am not sure it would be that useful though. Maybe `handle_missing="passthrough"` would be a YAGNI.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
I will take this issue.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
@lorentzenchr do you have any opinion on which options are best for this estimator?

I think we might want to start with `handle_missing="error"` (explicit parameter to preserve the current behavior) + `handle_missing="indicator"` and keep the other options in mind for possible follow-up PR if someone expresses a need for it.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
My opinion is that our `Pipeline` concept seems to be broken as hinted by the fact that we implement or discuss support for missing values in many estimators and transformers instead of promoting a pipeline with classes from the `impute` module.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
I agree the usability of the pipeline is a problem but for this specific case I do not really see how improving the pipeline would allow to compose a missing value indicator transformer or a `SimpleImputer` imputer in a user-friendly way that would match what I describe in the description of this issue. I think the missing data handling logic for a spline transformer would much more naturally be handled by the spline transformer itself in this case.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
> Note that handle_missing="indicator" would be different and statistically more meaningful than  SimpleImputer(strategy="mean", add_indicator=True) with SplineTransformer and furthermore would make for leaner ML pipelines (better UX).

@ogrisel Can you elaborate?

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
If the missingness pattern in `X[:, i]` alone is predictive of `y` by itself (for a given feature `i`), it would be sensible for a fitted linear model to have a strong `coef` only on the missingness indicator feature and near zero `coef` values to all the feature spline knots. This is what should happen if `SplineTransformer` itself is updated to implement the `handle_missing="indicator"` I suggest in the description of this issue.

If instead you rely mean-imputation + missingness indicator as a preprocessing step before `SplineTransformer`, at least 2 `coef` values will be significantly non-zero: the one for the missingness indicator feature, and the one for the spline knots that happen to also match the `np.nanmean(X[:, i])` (which is arbitrary).

It makes model inspection more challenging.

Labels: Enhancement
Handle np.nan / missing values in SplineTransformer

I think it would be quite natural to add an option to `SplineTransformer` to accept inputs with missing values as follows:

- `handle_missing="error"` the default (keep current behavior)
- `handle_missing="zero"`/"constant": encode missing values by setting all output features for that input column to 0 (or some other constant, see discussion below),
- `handle_missing="indicator"`: append an extra binary feature as missingness indicator and encode missing values as 0 on the remaining output features.

Note that `handle_missing="indicator"` would be different and statistically more meaningful than  `SimpleImputer(strategy="mean", add_indicator=True)` with `SplineTransformer` and furthermore would make for leaner ML pipelines (better UX).

I am not sure if we need to add the `handle_missing="zero"` option. It would break the property the sum of output values of a given `SplineTransformer` encoding always sum to 1 while `handle_missing="indicator"` would preserve this property (in addition to make missingness more explicit to the downstream model in case missingness is informative one way or another).

If we want to preserve the sum to 1 property while not adding an explicit missingness indicator feature, maybe we could instead provide `handle_missing="constant"` (not sure about the name) that would encode missing values as `1 / n_outputs` to preserve the "sum to 1" property. Not entirely sure if this would result in a more interesting prior than the zero encoding.

COMMENT:
The coefficients multiply basis splines not knots, therefore I don’t follow your conclusion. An example demonstrating your claims would be convincing (just with manual feature preprocessing).

Labels: Enhancement