Talip7/scikit-learn-issues-multilabel huggingface dataset:
Regression in `ColumnTransformer` due to internal `FunctionTransformer`

In https://github.com/scikit-learn/scikit-learn/pull/27801, we make sure that the output of `func` and the `get_feature_names_out` are consistent.

However, it seems that we have a side effect when the `FunctionTransformer` is created inside a `ColumnTransformer` in some case. The example below will provide a dataframe and the identity function will return as-is and the column name will not be consistent with the `get_feature_names_out`.

```python
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler

X = pd.DataFrame(np.random.randn(10, 4))

preprocessor = ColumnTransformer(
    transformers=[("scaler", StandardScaler(), [0, 1])],
    remainder="passthrough",
)
preprocessor.fit_transform(X)
```

```
E               ValueError: The output generated by `func` have different column names than the one generated by the method `get_feature_names_out`. Got output with columns names: [0] and `get_feature_names_out` returned: ['x0']. This can be fixed in different manners depending on your use case:
E               (i) If `func` returns a container with column names, make sure they are consistent with the output of `get_feature_names_out`.
E               (ii) If `func` is a NumPy `ufunc`, then forcing `validate=True` could be considered to internally convert the input container to a NumPy array before calling the `ufunc`.
E               (iii) The column names can be overriden by setting `set_output(transform='pandas')` such that the column names are set to the names provided by `get_feature_names_out`.
```

COMMENT:
So this kind of a tricky one. I am not sure exactly how to solve it.

We kind of improve the expected behaviour of the `FunctionTransformer` when used standalone. However, in the `ColumnTransformer`, we always relied on the `get_feature_names_out` generation even if it was not coherent with the `fit_transform` of the transformer.

Maybe we should reconsider the fix in #27801, and always overwrite the column names from `fit_transform` using `get_feature_names_out`. We might want to raise a warning such that people are aware of it.

Labels: Bug
Regression in `ColumnTransformer` due to internal `FunctionTransformer`

In https://github.com/scikit-learn/scikit-learn/pull/27801, we make sure that the output of `func` and the `get_feature_names_out` are consistent.

However, it seems that we have a side effect when the `FunctionTransformer` is created inside a `ColumnTransformer` in some case. The example below will provide a dataframe and the identity function will return as-is and the column name will not be consistent with the `get_feature_names_out`.

```python
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler

X = pd.DataFrame(np.random.randn(10, 4))

preprocessor = ColumnTransformer(
    transformers=[("scaler", StandardScaler(), [0, 1])],
    remainder="passthrough",
)
preprocessor.fit_transform(X)
```

```
E               ValueError: The output generated by `func` have different column names than the one generated by the method `get_feature_names_out`. Got output with columns names: [0] and `get_feature_names_out` returned: ['x0']. This can be fixed in different manners depending on your use case:
E               (i) If `func` returns a container with column names, make sure they are consistent with the output of `get_feature_names_out`.
E               (ii) If `func` is a NumPy `ufunc`, then forcing `validate=True` could be considered to internally convert the input container to a NumPy array before calling the `ufunc`.
E               (iii) The column names can be overriden by setting `set_output(transform='pandas')` such that the column names are set to the names provided by `get_feature_names_out`.
```

COMMENT:
I wondering if you have any thoughts @thomasjpfan

Labels: Bug
Trees fitted in 1.3.2 produce different outcome when evaluated in 1.4

### Describe the bug

We have a number of ensemble tree models in production fitted using 1.3.2 or older. Many of these models produce different outcomes when evaluated on the same data in sklearn 1.3.2 and 1.4.

Analysis led me to the change in `DecisionTreeClassifier.predict_proba()` from this PR: https://github.com/scikit-learn/scikit-learn/pull/27639

Based on this conversation I understand that in order to support monotonicity constraint, probabilities were allowed to be outside of [0, 1] bounds as see in this diff:
![image](https://github.com/scikit-learn/scikit-learn/assets/55986945/e0d980cf-7263-48af-85a5-d4c7efe6aed4)

Tree splitting criterion was modified to ensure that probabilities are within [0, 1] bounds. This works only if the user fits the tree in 1.4 and evaluates it in the same version. I was not able to find an example where a tree fitted in 1.4 produces probabilities outside of [0,1]. However, trees fitted in older versions do violate this constraint.

Would you consider rolling back probability normalization for trees that don't have monotonicity constraint so this method produces correctly normalized probabilities on trees fitted in prior versions? I don't think scikit-learn makes any guarantees about compatibility of estimators created in older versions. However, it is quite disruptive for anyone who maintains older tree models essentially forcing people to re-fit them if they switch to 1.4.

Note: I have not provided code to reproduce because it requires 2 different versions of sklearn.

### Steps/Code to Reproduce

N/A

### Expected Results

probabilities add up to 1

### Actual Results

Probabilities do not add up to 1

### Versions

```shell
System:
    python: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:27:59) [MSC v.1937 64 bit (AMD64)]
executable: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\python.exe
   machine: Windows-10-10.0.22631-SP0

Python dependencies:
      sklearn: 1.4.0
          pip: 23.3.2
   setuptools: 69.0.3
        numpy: 1.26.3
        scipy: 1.12.0
       Cython: 3.0.7
       pandas: 2.2.0
   matplotlib: 3.8.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libomp
       filepath: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\Library\bin\libomp.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libblas
       filepath: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\Library\bin\libblas.dll
        version: 0.3.25
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\vcomp140.dll
```

COMMENT:
That's expected. We only support loading models with the same scikit-learn version as they were persisted: https://scikit-learn.org/stable/model_persistence.html

Labels: Bug
Trees fitted in 1.3.2 produce different outcome when evaluated in 1.4

### Describe the bug

We have a number of ensemble tree models in production fitted using 1.3.2 or older. Many of these models produce different outcomes when evaluated on the same data in sklearn 1.3.2 and 1.4.

Analysis led me to the change in `DecisionTreeClassifier.predict_proba()` from this PR: https://github.com/scikit-learn/scikit-learn/pull/27639

Based on this conversation I understand that in order to support monotonicity constraint, probabilities were allowed to be outside of [0, 1] bounds as see in this diff:
![image](https://github.com/scikit-learn/scikit-learn/assets/55986945/e0d980cf-7263-48af-85a5-d4c7efe6aed4)

Tree splitting criterion was modified to ensure that probabilities are within [0, 1] bounds. This works only if the user fits the tree in 1.4 and evaluates it in the same version. I was not able to find an example where a tree fitted in 1.4 produces probabilities outside of [0,1]. However, trees fitted in older versions do violate this constraint.

Would you consider rolling back probability normalization for trees that don't have monotonicity constraint so this method produces correctly normalized probabilities on trees fitted in prior versions? I don't think scikit-learn makes any guarantees about compatibility of estimators created in older versions. However, it is quite disruptive for anyone who maintains older tree models essentially forcing people to re-fit them if they switch to 1.4.

Note: I have not provided code to reproduce because it requires 2 different versions of sklearn.

### Steps/Code to Reproduce

N/A

### Expected Results

probabilities add up to 1

### Actual Results

Probabilities do not add up to 1

### Versions

```shell
System:
    python: 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:27:59) [MSC v.1937 64 bit (AMD64)]
executable: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\python.exe
   machine: Windows-10-10.0.22631-SP0

Python dependencies:
      sklearn: 1.4.0
          pip: 23.3.2
   setuptools: 69.0.3
        numpy: 1.26.3
        scipy: 1.12.0
       Cython: 3.0.7
       pandas: 2.2.0
   matplotlib: 3.8.2
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libomp
       filepath: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\Library\bin\libomp.dll
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libblas
       filepath: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\Library\bin\libblas.dll
        version: 0.3.25
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\smozharov\AppData\Local\miniconda3\envs\env311_3a\vcomp140.dll
```

COMMENT:
We can reopen the issue if you provide a minimal reproducible with the latest version.

Labels: Bug
⚠️ CI failed on Wheel builder ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/7598773815)** (Jan 21, 2024)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/7606191759) on Jan 22, 2024

Labels: Needs Triage
ENH: use the sparse-sparse backend for computing pairwise distance

First reported in: https://github.com/scikit-learn-contrib/imbalanced-learn/issues/1056

We have a regression in `kneighbors` with sparse matrix from 1.1.X to 1.3.X.
A code sample to reproduce:

```python
# %%
import sklearn
sklearn.__version__

# %%
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, 10_000
X = sparse.random(n_samples, n_features, density=0.01, format="csr", random_state=0)
rng = np.random.default_rng(0)
y = rng.integers(0, 2, size=n_samples)
knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)

# %%
%%timeit
knn.kneighbors(X, return_distance=False)
```

#### 1.1.X

```
21.5 ms ± 217 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

#### `main`

```
1.16 s ± 9.87 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

#### Small benchmark

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/5abe644f-5c05-4124-ab43-dd0a1cdd3e58)

<details>

```python
# %%
import sklearn
sklearn.__version__

# %%
import time
from collections import defaultdict
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, [500, 1_000, 2_500, 5_000, 7_500, 10_000, 25_000, 50_000]

results = defaultdict(list)
for nf in n_features:
    X = sparse.random(n_samples, nf, density=0.01, format="csr", random_state=0)
    rng = np.random.default_rng(0)
    y = rng.integers(0, 2, size=n_samples)
    knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)
    start = time.time()
    knn.kneighbors(X, return_distance=False)
    elapsed_time = time.time() - start
    results["version"].append(sklearn.__version__)
    results["n_features"].append(nf)
    results["elapsed_time"].append(elapsed_time)

# %%
import pandas as pd

results = pd.DataFrame(results)
results.to_csv(f"bench_{sklearn.__version__}.csv", index=False)
```

```python
# %%
import pandas as pd

results_main = pd.read_csv(f"bench_1.5.dev0.csv")
results_1_1 = pd.read_csv(f"bench_1.1.3.csv")
results = pd.concat([results_main, results_1_1], axis=0)
results

# %%
import seaborn as sns

sns.set_context("talk")

# %%
sns.relplot(
    data=results,
    x="n_features",
    y="elapsed_time",
    hue="version",
    kind="line",
    height=6,
    aspect=1.5,
    legend=True,
)
sns.despine(offset=10, trim=True)
```

</details>

#### Debug profiling

My first investigation look like we are spending all our time in the following function:

https://github.com/scikit-learn/scikit-learn/blob/5c7e831306e0a087c2b6af6913fa5b3c402f6d67/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.tp#L39-L69

But I'm a bit rusty with profiling native code. I need a bit more time to investigate.

COMMENT:
I'm under the impression that having a large number of features is something that was not taken into account when benchmarking in https://github.com/scikit-learn/scikit-learn/pull/24556

Labels: Enhancement
ENH: use the sparse-sparse backend for computing pairwise distance

First reported in: https://github.com/scikit-learn-contrib/imbalanced-learn/issues/1056

We have a regression in `kneighbors` with sparse matrix from 1.1.X to 1.3.X.
A code sample to reproduce:

```python
# %%
import sklearn
sklearn.__version__

# %%
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, 10_000
X = sparse.random(n_samples, n_features, density=0.01, format="csr", random_state=0)
rng = np.random.default_rng(0)
y = rng.integers(0, 2, size=n_samples)
knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)

# %%
%%timeit
knn.kneighbors(X, return_distance=False)
```

#### 1.1.X

```
21.5 ms ± 217 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

#### `main`

```
1.16 s ± 9.87 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

#### Small benchmark

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/5abe644f-5c05-4124-ab43-dd0a1cdd3e58)

<details>

```python
# %%
import sklearn
sklearn.__version__

# %%
import time
from collections import defaultdict
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, [500, 1_000, 2_500, 5_000, 7_500, 10_000, 25_000, 50_000]

results = defaultdict(list)
for nf in n_features:
    X = sparse.random(n_samples, nf, density=0.01, format="csr", random_state=0)
    rng = np.random.default_rng(0)
    y = rng.integers(0, 2, size=n_samples)
    knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)
    start = time.time()
    knn.kneighbors(X, return_distance=False)
    elapsed_time = time.time() - start
    results["version"].append(sklearn.__version__)
    results["n_features"].append(nf)
    results["elapsed_time"].append(elapsed_time)

# %%
import pandas as pd

results = pd.DataFrame(results)
results.to_csv(f"bench_{sklearn.__version__}.csv", index=False)
```

```python
# %%
import pandas as pd

results_main = pd.read_csv(f"bench_1.5.dev0.csv")
results_1_1 = pd.read_csv(f"bench_1.1.3.csv")
results = pd.concat([results_main, results_1_1], axis=0)
results

# %%
import seaborn as sns

sns.set_context("talk")

# %%
sns.relplot(
    data=results,
    x="n_features",
    y="elapsed_time",
    hue="version",
    kind="line",
    height=6,
    aspect=1.5,
    legend=True,
)
sns.despine(offset=10, trim=True)
```

</details>

#### Debug profiling

My first investigation look like we are spending all our time in the following function:

https://github.com/scikit-learn/scikit-learn/blob/5c7e831306e0a087c2b6af6913fa5b3c402f6d67/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.tp#L39-L69

But I'm a bit rusty with profiling native code. I need a bit more time to investigate.

COMMENT:
I confirm that nearly all the time (>99%) is spent in `_middle_term_sparse_sparse_*`.

Labels: Enhancement
ENH: use the sparse-sparse backend for computing pairwise distance

First reported in: https://github.com/scikit-learn-contrib/imbalanced-learn/issues/1056

We have a regression in `kneighbors` with sparse matrix from 1.1.X to 1.3.X.
A code sample to reproduce:

```python
# %%
import sklearn
sklearn.__version__

# %%
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, 10_000
X = sparse.random(n_samples, n_features, density=0.01, format="csr", random_state=0)
rng = np.random.default_rng(0)
y = rng.integers(0, 2, size=n_samples)
knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)

# %%
%%timeit
knn.kneighbors(X, return_distance=False)
```

#### 1.1.X

```
21.5 ms ± 217 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

#### `main`

```
1.16 s ± 9.87 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

#### Small benchmark

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/5abe644f-5c05-4124-ab43-dd0a1cdd3e58)

<details>

```python
# %%
import sklearn
sklearn.__version__

# %%
import time
from collections import defaultdict
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, [500, 1_000, 2_500, 5_000, 7_500, 10_000, 25_000, 50_000]

results = defaultdict(list)
for nf in n_features:
    X = sparse.random(n_samples, nf, density=0.01, format="csr", random_state=0)
    rng = np.random.default_rng(0)
    y = rng.integers(0, 2, size=n_samples)
    knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)
    start = time.time()
    knn.kneighbors(X, return_distance=False)
    elapsed_time = time.time() - start
    results["version"].append(sklearn.__version__)
    results["n_features"].append(nf)
    results["elapsed_time"].append(elapsed_time)

# %%
import pandas as pd

results = pd.DataFrame(results)
results.to_csv(f"bench_{sklearn.__version__}.csv", index=False)
```

```python
# %%
import pandas as pd

results_main = pd.read_csv(f"bench_1.5.dev0.csv")
results_1_1 = pd.read_csv(f"bench_1.1.3.csv")
results = pd.concat([results_main, results_1_1], axis=0)
results

# %%
import seaborn as sns

sns.set_context("talk")

# %%
sns.relplot(
    data=results,
    x="n_features",
    y="elapsed_time",
    hue="version",
    kind="line",
    height=6,
    aspect=1.5,
    legend=True,
)
sns.despine(offset=10, trim=True)
```

</details>

#### Debug profiling

My first investigation look like we are spending all our time in the following function:

https://github.com/scikit-learn/scikit-learn/blob/5c7e831306e0a087c2b6af6913fa5b3c402f6d67/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.tp#L39-L69

But I'm a bit rusty with profiling native code. I need a bit more time to investigate.

COMMENT:
FYI, I'm running a larger benchmark for different `n_features`, `n_samples`, `density` of the sparse matrix to see if we should keep a single backend or switch in a smart manner to the previous dispatching mechanism.

Labels: Enhancement
ENH: use the sparse-sparse backend for computing pairwise distance

First reported in: https://github.com/scikit-learn-contrib/imbalanced-learn/issues/1056

We have a regression in `kneighbors` with sparse matrix from 1.1.X to 1.3.X.
A code sample to reproduce:

```python
# %%
import sklearn
sklearn.__version__

# %%
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, 10_000
X = sparse.random(n_samples, n_features, density=0.01, format="csr", random_state=0)
rng = np.random.default_rng(0)
y = rng.integers(0, 2, size=n_samples)
knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)

# %%
%%timeit
knn.kneighbors(X, return_distance=False)
```

#### 1.1.X

```
21.5 ms ± 217 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

#### `main`

```
1.16 s ± 9.87 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

#### Small benchmark

![image](https://github.com/scikit-learn/scikit-learn/assets/7454015/5abe644f-5c05-4124-ab43-dd0a1cdd3e58)

<details>

```python
# %%
import sklearn
sklearn.__version__

# %%
import time
from collections import defaultdict
import numpy as np
from scipy import sparse
from sklearn.neighbors import KNeighborsRegressor

n_samples, n_features = 1_000, [500, 1_000, 2_500, 5_000, 7_500, 10_000, 25_000, 50_000]

results = defaultdict(list)
for nf in n_features:
    X = sparse.random(n_samples, nf, density=0.01, format="csr", random_state=0)
    rng = np.random.default_rng(0)
    y = rng.integers(0, 2, size=n_samples)
    knn = KNeighborsRegressor(n_neighbors=5).fit(X, y)
    start = time.time()
    knn.kneighbors(X, return_distance=False)
    elapsed_time = time.time() - start
    results["version"].append(sklearn.__version__)
    results["n_features"].append(nf)
    results["elapsed_time"].append(elapsed_time)

# %%
import pandas as pd

results = pd.DataFrame(results)
results.to_csv(f"bench_{sklearn.__version__}.csv", index=False)
```

```python
# %%
import pandas as pd

results_main = pd.read_csv(f"bench_1.5.dev0.csv")
results_1_1 = pd.read_csv(f"bench_1.1.3.csv")
results = pd.concat([results_main, results_1_1], axis=0)
results

# %%
import seaborn as sns

sns.set_context("talk")

# %%
sns.relplot(
    data=results,
    x="n_features",
    y="elapsed_time",
    hue="version",
    kind="line",
    height=6,
    aspect=1.5,
    legend=True,
)
sns.despine(offset=10, trim=True)
```

</details>

#### Debug profiling

My first investigation look like we are spending all our time in the following function:

https://github.com/scikit-learn/scikit-learn/blob/5c7e831306e0a087c2b6af6913fa5b3c402f6d67/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.tp#L39-L69

But I'm a bit rusty with profiling native code. I need a bit more time to investigate.

COMMENT:
How likely is it to have this fixed pretty quickly? I wonder how realistic it is to include it in 1.4.1

Labels: Enhancement