Talip7/scikit-learn-issues-multilabel huggingface dataset:
Dummy estimators don't have the `feature_names_in_` nor `n_features_in_` attributes

### Describe the bug

`DummyClassifier` and `DummyRegressor` estimators don't have the `feature_names_in_` nor `n_features_in_` attributes. The reason is that they don't call `self._validate_data` during `fit` like other estimators do.

### Steps/Code to Reproduce

```python
from sklearn.datasets import load_breast_cancer
from sklearn.dummy import DummyClassifier

X, y = load_breast_cancer(return_X_y=True, as_frame=True)

dummy = DummyClassifier().fit(X, y)
print(dummy.feature_names_in_)  # Fails
```

### Expected Results

No errors.

### Actual Results

```
Traceback (most recent call last):
  File "C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\IPython\core\interactiveshell.py", line 3548, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-bf0e7a849755>", line 7, in <module>
    print(dummy.feature_names_in_)  # Fails
          ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DummyClassifier' object has no attribute 'feature_names_in_'
```

### Versions

```shell
System:
    python: 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)]
executable: C:\Users\Mavs\Documents\Python\ATOM\venv311\Scripts\python.exe
   machine: Windows-10-10.0.19045-SP0
Python dependencies:
      sklearn: 1.3.2
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.24.4
        scipy: 1.11.3
       Cython: 3.0.5
       pandas: 2.1.2
   matplotlib: 3.8.0
       joblib: 1.3.2
threadpoolctl: 3.2.0
Built with OpenMP: True
threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
        version: 0.3.21
threading_layer: pthreads
   architecture: Zen
       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: vcomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\sklearn\.libs\vcomp140.dll
        version: None
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\scipy.libs\libopenblas_v0.3.20-571-g3dec11c6-gcc_10_3_0-c2315440d6b6cef5037bad648efc8c59.dll
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Zen
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libiomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\torch\lib\libiomp5md.dll
        version: None
       user_api: openmp
   internal_api: openmp
    num_threads: 1
         prefix: libiomp
       filepath: C:\Users\Mavs\Documents\Python\ATOM\venv311\Lib\site-packages\torch\lib\libiompstubs5md.dll
        version: None
```

COMMENT:
WDYT @thomasjpfan ?

Labels: Bug
Ensure predictions sparse before `sp.hstack` in `ClassifierChain`

We use `sp.hstack` in a number of places in `ClassifierChain` where we may be stacking sparse with dense, e.g.,:

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L948

and

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L693

AFAICT it seems stacking a sparse with dense via `sp.hstack` gives you a sparse array (even though `sp.hstack` is not documented to support dense):

```bash
In [34]: from scipy.sparse import coo_matrix, hstack
    ...: 
    ...: A = coo_matrix([[1, 2], [3, 4]])

In [35]: B = np.zeros((2,2))

In [36]: hstack([A,B])
Out[36]: 
<2x4 sparse matrix of type '<class 'numpy.float64'>'
        with 4 stored elements in COOrdinate format>
```

Maybe due to: https://github.com/scipy/scipy/blob/f990b1d2471748c79bc4260baf8923db0a5248af/scipy/sparse/_construct.py#L654 ?

Should we ensure y is sparse before using `sp.hstack` ?

I had quick look at our code, I could not find any other cases where it would be possible to be stacking dense + sparse. I think `ClassifierChain` is unique in that we do not usually combine `X` with `y`

Discussed here: https://github.com/scikit-learn/scikit-learn/pull/27700#discussion_r1378691272

cc @glemaitre

COMMENT:
So I looked closer at the issue. This is indeed supported. If the behaviour change, then I expect our test suite to fail so this is not a big deal.

I also did not spot any improvement in terms of performance (speed and memory) so I think that we can let this code as-is.

Labels: Needs Triage
Ensure predictions sparse before `sp.hstack` in `ClassifierChain`

We use `sp.hstack` in a number of places in `ClassifierChain` where we may be stacking sparse with dense, e.g.,:

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L948

and

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L693

AFAICT it seems stacking a sparse with dense via `sp.hstack` gives you a sparse array (even though `sp.hstack` is not documented to support dense):

```bash
In [34]: from scipy.sparse import coo_matrix, hstack
    ...: 
    ...: A = coo_matrix([[1, 2], [3, 4]])

In [35]: B = np.zeros((2,2))

In [36]: hstack([A,B])
Out[36]: 
<2x4 sparse matrix of type '<class 'numpy.float64'>'
        with 4 stored elements in COOrdinate format>
```

Maybe due to: https://github.com/scipy/scipy/blob/f990b1d2471748c79bc4260baf8923db0a5248af/scipy/sparse/_construct.py#L654 ?

Should we ensure y is sparse before using `sp.hstack` ?

I had quick look at our code, I could not find any other cases where it would be possible to be stacking dense + sparse. I think `ClassifierChain` is unique in that we do not usually combine `X` with `y`

Discussed here: https://github.com/scikit-learn/scikit-learn/pull/27700#discussion_r1378691272

cc @glemaitre

COMMENT:
Interesting! Is there documentation about this or just in the code? Thanks

Labels: Needs Triage
Ensure predictions sparse before `sp.hstack` in `ClassifierChain`

We use `sp.hstack` in a number of places in `ClassifierChain` where we may be stacking sparse with dense, e.g.,:

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L948

and

https://github.com/scikit-learn/scikit-learn/blob/36f6734789fc7e4940792c1cfb6a6e90dfcae484/sklearn/multioutput.py#L693

AFAICT it seems stacking a sparse with dense via `sp.hstack` gives you a sparse array (even though `sp.hstack` is not documented to support dense):

```bash
In [34]: from scipy.sparse import coo_matrix, hstack
    ...: 
    ...: A = coo_matrix([[1, 2], [3, 4]])

In [35]: B = np.zeros((2,2))

In [36]: hstack([A,B])
Out[36]: 
<2x4 sparse matrix of type '<class 'numpy.float64'>'
        with 4 stored elements in COOrdinate format>
```

Maybe due to: https://github.com/scipy/scipy/blob/f990b1d2471748c79bc4260baf8923db0a5248af/scipy/sparse/_construct.py#L654 ?

Should we ensure y is sparse before using `sp.hstack` ?

I had quick look at our code, I could not find any other cases where it would be possible to be stacking dense + sparse. I think `ClassifierChain` is unique in that we do not usually combine `X` with `y`

Discussed here: https://github.com/scikit-learn/scikit-learn/pull/27700#discussion_r1378691272

cc @glemaitre

COMMENT:
It is only in the code because the documentation state of scipy state that the array should be sparse. So here there is an implicit conversion happening.

Labels: Needs Triage
allow_nan tag in Pipelines

Unfortunately, our tag system for allowing nans does not work with pipelines. Lets say we have a pipeline with two steps and the final step does not accept nans:

1. If the first step is an Imputer, then the pipeline accept nans. For example: `make_pipeline(SimpleImputer(), LogisticRegression())`
2. If the first step is a StandardScalar (which accept nans and leaves them along), then the pipeline does not accept nans. For example: `make_pipeline(StandardScalar(), LogisticRegression())`

We likely need a "output_nan" tag to reliability give a pipeline a "allow_nan" tag.

COMMENT:
If the first step is a `StandardScaler` and every other step doesn't break given `nan`, then the pipeline supports it, right? I mean, a pipeline only doesn't support nan if something breaks given nan as input, otherwise it technically supports nans, I think.

But I think `outputs_nan` would be more clear in semantics and given `allow_nan` + `outputs_nan` we can clearly say how the pipeline feels and what it does given nans.

Labels: API, Needs Decision, RFC
allow_nan tag in Pipelines

Unfortunately, our tag system for allowing nans does not work with pipelines. Lets say we have a pipeline with two steps and the final step does not accept nans:

1. If the first step is an Imputer, then the pipeline accept nans. For example: `make_pipeline(SimpleImputer(), LogisticRegression())`
2. If the first step is a StandardScalar (which accept nans and leaves them along), then the pipeline does not accept nans. For example: `make_pipeline(StandardScalar(), LogisticRegression())`

We likely need a "output_nan" tag to reliability give a pipeline a "allow_nan" tag.

COMMENT:
> If the first step is a StandardScaler and every other step doesn't break given nan, then the pipeline supports it, right?

Yup, this is the only configuration where we can be sure that the pipeline accepts nans. There is not enough information to tag `make_pipeline(SimpleImputer(), LogisticRegression()`.

Labels: API, Needs Decision, RFC
allow_nan tag in Pipelines

Unfortunately, our tag system for allowing nans does not work with pipelines. Lets say we have a pipeline with two steps and the final step does not accept nans:

1. If the first step is an Imputer, then the pipeline accept nans. For example: `make_pipeline(SimpleImputer(), LogisticRegression())`
2. If the first step is a StandardScalar (which accept nans and leaves them along), then the pipeline does not accept nans. For example: `make_pipeline(StandardScalar(), LogisticRegression())`

We likely need a "output_nan" tag to reliability give a pipeline a "allow_nan" tag.

COMMENT:
> outputs_nan

Actually, we could even have a step that create `nan` and in this case the label `output_nan` is not trivial anymore, isn't it?

Labels: API, Needs Decision, RFC
allow_nan tag in Pipelines

Unfortunately, our tag system for allowing nans does not work with pipelines. Lets say we have a pipeline with two steps and the final step does not accept nans:

1. If the first step is an Imputer, then the pipeline accept nans. For example: `make_pipeline(SimpleImputer(), LogisticRegression())`
2. If the first step is a StandardScalar (which accept nans and leaves them along), then the pipeline does not accept nans. For example: `make_pipeline(StandardScalar(), LogisticRegression())`

We likely need a "output_nan" tag to reliability give a pipeline a "allow_nan" tag.

COMMENT:
> Actually, we could even have a step that create nan and in this case the label output_nan is not trivial anymore, isn't it?

Maybe the name should be "can_output_nan". If a step can generate nans, then `can_output_nan=True`.

I'm still unsure about the tag's name. In our case, "output" is for transformers only. It does not mean the output from `predict`, etc.

Labels: API, Needs Decision, RFC
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
Pinging people that might know the answer straight away: @jjerphan @ogrisel @fcharras @Micky774?

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
I was looking at the following that could be interested as well: https://www.osti.gov/servlets/purl/1557469

But it seems that SYRK should skip some multiplication that GEMM is performing.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
`SYRK` must in theory be more appropriate than `GEMM` in this case (i.e. when $A = B$), but only comparing implementations will tell us.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
@glemaitre thanks for adding more folks, also I will take a look at paper : https://www.osti.gov/servlets/purl/1557469.

@jjerphan  yes in theory it is more appropriate, i don't have comparison. I will try see by adding it.
do you have any reference , how can we add blas functions.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
You can easily add them from SciPy.

See:

https://github.com/scikit-learn/scikit-learn/blob/2d9f4c442030450cd6246765df2a546e6440b8ac/sklearn/utils/_cython_blas.pxd#L38

https://github.com/scikit-learn/scikit-learn/blob/2d9f4c442030450cd6246765df2a546e6440b8ac/sklearn/utils/_cython_blas.pyx#L184-L186

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
thank you!

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
Looking forward to a PR with some quick benchmarks.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
For information, numpy uses syrk under the hood if it detects `X.T @ X`.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
Also, optimizations will work for full matrices, for pattern relying on chunks (_à la_ `PairwiseDistancesReduction`) we can only benefit from `SYRK` on the diagonal chunks.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
@jjerphan so for  `PairwiseDistancesReduction `, if it is already using the chunks of 256 size, how we can identify the diagonal chunks?
`X_start == Y_start and X_end == Y_end`
Any thoughts on how we can make it work for full matrices!

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
Also can you help with understanding why at scikit-learn we are relying on chunks, as in backend OpenBLAS is also doing it!

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
`X is Y and X_start == Y_start and X_end == Y_end` should be sufficient to identify when to use `SYRK`, but this might add too much complexity and it might cause regression due to branching and branch mis-prediction for every `n_chunks + 1` iterations here:

https://github.com/scikit-learn/scikit-learn/blob/3b06962d280f8776e3e94c7aed862081a82a9cc6/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.tp#L445-L446

You can try and perform benchmarks with https://github.com/scikit-learn/pairwise-distances-reductions-asv-suite, but _a priori_ using `SYRK` in `PairwiseDistancesReduction` would not help significantly.

Chunks are used for `PairwiseDistancesReduction` as explained in the private submodule documentation, here:

https://github.com/scikit-learn/scikit-learn/blob/3b06962d280f8776e3e94c7aed862081a82a9cc6/sklearn/metrics/_pairwise_distances_reduction/__init__.py#L8-L29

Let us know if something is unclear.

Labels: Performance, Needs Benchmarks
Use SYRK instead of GEMM in pairwise distance

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/27877

<div type='discussions-op-text'>

<sup>Originally posted by **darshanp4** November 30, 2023</sup>
Hello 

I was checking the DBSCAN algo , where mostly computing pairwise distance it use -2X*X**T, so for this operation currently sklearn uses the blas _gemm. where it consuming much time. If we can use the syrk which also level3 blas (https://pyclblas.readthedocs.io/en/latest/SYRK.html). Which is more optimized for this type of operations.

Is there any similar approach any one seen or tried.

Thank you.  </div>

COMMENT:
To keep intermediate data-structures in CPU cache, cache size matters for performance. which will be different for every ISA. But here it is hardcoded to 256. In OpenBLAS it is handled for each ISA. So shouldn't we leave it for OpenBLAS to handle it more optimize way.

Labels: Performance, Needs Benchmarks